Upgrade of NetScaler breaks NetScaler - Post upgrade the NetScaler doesn't want to work anymore

[J.R. van Doornik]

Our current NetScalers are version 13.1 24.38, which has a known vunerability. 

 

In order to correct this, we've downloaded the 13.1 42.47 version, and tried installing it on two separate occasions. 

 

Since the NetScalers are on a VM environment, we snapshotted the 24.38 versions, and then initiated the upgrade on the system to get it to version 42.47. 

 

Our first try ended up with the NetScalers presenting a blank screen instead of the login page. At this point we brought down the upgraded systems, and reverted to the snapshotted to reinstate the functionality, as it's demanded pretty much 24/7 so longer periods of downtime are not really desired.. 

 

I managed to find https://support.citrix.com/article/CTX277650/blankwhite-page-after-upgrading-citrix-adc-or-citrix-gateway which sounded exactly like the scenario we encountered. 

 

The httpd files were pulled off the working 24.38 system, and today we had our second go. Shut down the snapshotted machines, booted the original (upgraded machines) again, replaced the httpd files as indicated and then ran into the fact that the webpage was no longer blank, but informing us of the fact that the server disconnected our session. 

 

After toying with that for 15 minutes and not getting that to work either, we again shut down the upgraded machines, and again reverted back to the snapshot.

 

So... we know we're working with a vunerable version of NetScaler, but seem to be unable to upgrade to the last version. And at present we're unsure what the options are to get this to work. I'd therefore like to ask for any potential ideas of what might be going on here and why the 1-on-1 upgrade isn't working where during past upgrades this didn't give us any issues. Manually re-entering the entire set of information and certificates in hopes that that might work isn't something I would deem feasable.

 

Thanks in advance for any and all insights.

[Rhonda Rowland1709152125]

This is absolutely an issue you should open a case with support about.  Either it is build bug that needs to be fixed (and a new build is needed) or you need some additional info for a workaround to patch.

 

If you were to spin up a NEW vpx instance from the build to test, you wouldn't have to manually enter the files.

You restore a copy of the /nsconfig/ns.conf, /nsconfig/ssl/<certs>

Depending on gateway features: key directories under /var/ for portal theme and some other details.

There may be some other directories.

You can use the create system backup full comamdn on the original.

Extract the zip archavie, and then identify what additional directories to restore manually if settings are needed. while you evaluate this.

 

You can do this as a controlled OFFLINE test, just to see if a) it will work and then b) evaluate whether you are missing more settings than that. Just to see if it gives you an option to proceed,  while waiting on info from tech support.

 

 

 

[J.R. van Doornik]

While we do have the NetScalers up and running, our support knowledge on Linux is rudimentary. We're way more focused on Windows. 

 

That said, we'll probably have a go at either retrying the upgrade from scratch again (since I did notice the behavior changed from the blank screen to a notice the server disconnected the page), or possibly rebuilding from base up and running said restore.

 

I'm unsure on the support options open to us at present with Citrix, so I would have to look into that aswell. 

[Marcelo Oguma de Souza1709152865]

I would also try disabling the cache in NetScaler Gateway vserver (it is enabled by default even if you have the Integrated Caching feature disabled).

You can do this by adding a NOCACHE policy with highest priority (lower number, priority=1 for example) and rule=true, to your Gateway vservers.

Do this a few days before trying the upgrade, to make sure cache on clients is cleared. Then after a successful upgrade you can remove this policy again after some days to re-enable caching.

Note: with caching disabled, per user traffic might increase a bit during the login, if your links are close to saturation, I recommend to watch it closely.

 

This article has a step by step guide (although it was for a much older firmware, the process to disable cache is still valid)

https://www.jasonsamuel.com/2016/09/13/fixing-the-citrix-netscaler-gateway-blank-page-issue-when-upgrading-from-11-0-to-11-1/

[J.R. van Doornik]

Thanks for updating the thread. 

 

I opened an incident with Citrix support, and they indicated they'd be interested in screenshots and logs. 

 

I'm waiting for them to confirm which logs, so we can schedule downtime for the NetScalers and grab the appropriate data to get to a working solution asap. 

 

I've also offered to allow a remote session, so they can actually see what happens. 

 

I have yet to hear back further, but I do plan to update this thread with any further details that come to light. (If there is one thing I hate it's finding a thread describing your issue, and finding it abandoned from comments and thus without a solution available). 

[J.R. van Doornik]

Took some time today (originally 45 minutes were alotted, but it dragged on to 90 minutes) with Citrix support. 

 

For some unknown reason the upgrade seems to 'corrupt' the license file, making the NetScaler think it has no license for the Citrix Gateway service, which is what it needs for the functionality of allowing home-users to access the portal. 

 

Redeploying the old license file did nothing, and we grabbed the new license files from the Citrix site. Removed all license files, uploaded the last one downloaded from Citrix, and found the Citrix Gateway popped up in the licenses again. 

 

At that point there however was a yellow exclamation mark with the System\Gateway and System\Virtual Server bit. Right clicking that showed the feature was disabled, and we could enable it.

 

At that point however the Virtual Servers should contain atleast two servers shown... but the screen showed NO virtual server configurations present. 

 

My guess at this point is that either removing the old licenses or replacing the license with the newly downloaded one wiped the configuration that was suddenly unsupported by the license. 

 

We reverted back to the old version for now, and probably need to take stock of the configuration as it sits on the current machines, in order to duplicate that to the updated machines, certificates and configuration alike. 

[J.R. van Doornik]

Okay... issue has been resolved by my colleague when working with CItrix Support. 

 

Apparently the license files active on the Netscalers were outdated or something, despite them still working as expected. So no issues using the NetScalers and licenses, which might have hinted at the problem. 

 

Step 1 was to redownload the license files from the Citrix website (which seemed to be different ones than the one installed), and replace the existing license files with the new ones. 

Reboot the NetScalers, and verify their working as expected with the new license files. 

 

Then the upgrade was performed on the secondary node, which came back normally post-upgrade. 

Primary node was then upgraded, which somehow again killed the license file, but after re-adding the file and rebooting that sorted itself aswell. 

 

Multiple users seem to have reported this issue  with this specific upgrade and at Citrix that is under internal investigation. 

 

Hope this atleast helps someone down the road running into the same issue.