Jump to content

Netscaler VPX won't boot with Dell R740 and XenServer


Recommended Posts

I have been running Netscaler VPX instances on XenServer for years with no issue.  I recently bought new hardware - Dell R740 servers with dual Xeon Gold 6150's, 768GB of RAM and Tesla M10 GPUs to replace older Dell R620 servers.  I moved all of our VMs, including the Netscalers, off of the old XenServer implementation on the R620's to the new pool on the R740's.  XenServer is the latest (7.3) and all of my other VM's work just fine but upon booting the Netscalers I get "fatal trap 9" and dumped to a db> prompt.  I even tried downloading fresh virtual appliances from citrix.com and get the same result right after importing then.  However, when I import them onto one of the old R620 servers with the older Xeon E5 series chips, they boot right up.  Anyone have any ideas on what could cause this?  The Netscaler is my only guest running FreeBSD.  Everything else is either Windows, CentOS or Red Hat and they all work fine.  This leads me to believe this is something to do with the newer Xeon scalable chips or perhaps something else about the newer Dell hardware that the older FreeBSD that is being used for the Netscalers (8.4) cannot address.  To confirm this I downloaded an ISO of FreeBSD 8.4 (same version that Netscaler VPX uses) and attempted to boot into it and got the same error.  I tried booting it both within XenServer and directly on the R740 hardware.  Newer versions of FreeBSD ISOs (11.1 is the other one I tried) boot up just fine but 8.4 will not boot on this newer hardware, whether in XenServer or directly on the bare metal.  

 

Netscaler VPX is v12-56 (latest).  I did try importing older versions (10 and 11) with the same result.  I've attached a screenshot of the error - it's the same error whether it's the original VMs or a newly imported appliance.  Again, the error only occurs on the new R740's - the old R620's still boot these just fine.

 

Thanks to anyone who has any thoughts!

Capture.PNG.e7ec2ded3c4dd8d89cdf8e7518974baa.PNG

Link to comment
Share on other sites

9 hours ago, Siddhartha Sarmah said:

 

Good to know.  I wiped a host and loaded 7.0 on it and patched it to 7.0.0-125380c and got the same result; the VPX VM will boot on an R620 but not on an R740.  "fatal trap 9" again.  I've been running Netscaler VPX 12 on XenServer 7.3 since it came out on R620's with no trouble.

 

I have a case open with both Netscaler support and XenServer support.  So far, neither seems to have any ideas other than they need to lab it up.  

Link to comment
Share on other sites

2 minutes ago, Siddhartha Sarmah said:

That's good info. Hope you have passed this on to Support.

 

This seems like a  H/W (perhaps processor) compatibility issue

I did.  So far all I have gotten is a note that it is not currently possible for them to go beyond FreeBSD 8.4.  That is not comforting considering that is a 2013 release.  They are up to 11.1 on FreeBSD.   

 

I agree - I think this is an incompatibility with the Xeon Scalable processors.  Perhaps related to the requirement to run X2APIC in order to run XenServer at all on these processors.

Link to comment
Share on other sites

On 2/13/2018 at 11:29 AM, Daniel Challinor1709152724 said:

I have exactly the same issue as well using Xenserver 7.3 on Dell PowerEdge R640 Servers. Won't work with old or new NetScalers.

 

Do you have a support case number with Citrix that I can reference when I log a support case?

 

Regards,

 

Dan

Netscaler case #75864945 and XenServer case #75864993

Link to comment
Share on other sites

The only way I could resolve this problem on the Dell PowerEdge R640 with Intel Scalable Processors (Silver) was by downgrading to XenServer 7.0 as per the NetScaler 12 supported  hyper visor limitations document as mentioned by siddharthas https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html

 

Everything is working fine now with the NetScaler.

 

 

Link to comment
Share on other sites

34 minutes ago, Daniel Challinor1709152724 said:

The only way I could resolve this problem on the Dell PowerEdge R640 with Intel Scalable Processors (Silver) was by downgrading to XenServer 7.0 as per the NetScaler 12 supported  hyper visor limitations document as mentioned by siddharthas https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html

 

Everything is working fine now with the NetScaler.

 

 

Interesting - for me it still didn’t work.

 

The problem there is that the 14 gen hardware is not officially supported by XenServer 7.0 - http://hcl.xenserver.org/servers/841/Dell_EMC_PowerEdge_R640

 

It’s sort of a catch-22.  Do you want XenServer to be unsupported or do you want Netscaler to be unsupported?  If you have the 14gen hardware, that seems to be your current choice.

Link to comment
Share on other sites

It really is a catch 22 situation. The response I received back from Citrix is below:

 

"I have verified the issue with our Engineering team and as per Engineering team's confirmation we don’t support VPX on XS7.3 at present, currently supported versions are XS 6.5 and 7.0.

 

Request you to please try with XS 7.0/6.5 and let us know if you have any issues with the same.

 

Also I will be closing this case as of now, as this is out of support criteria as VPX on XS version 7.3 is not supported. Please refer the support matrix for the same as below:

https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html"

 

I have asked Citrix to re-open the case as I do not see this as being satisfactory that the customer I am implementing this for will be having to run an unsupported product. They are having this built so they can stop using XenApp 6.5 which will soon be EOL.

Link to comment
Share on other sites

  • 3 months later...
5 hours ago, Jonathon Wiggins1709152709 said:

We are having this issue as well. 

  • XenServer v7.1 LTSR CU1
  • NetScaler v11.1 or v12.1

Since XenServer v7.0 isn't supported any more, we need to get this working with LTSR. I have a ticket opened with Citrix Support.

That’s good - I get the impression that the more people who open cases on this the faster it will get resolved.  They have a fix so I’m not sure why they are delaying deploying it.

Link to comment
Share on other sites

  • 1 month later...
3 hours ago, Tommy Gallagher1709159681 said:

Hi Charles,

 

 

Do you know what the fix is?

My understanding is that the NetScaler team had to make some pretty major modifications to the kernel of the NetScaler in order to get around a bug in the newer Intel CPU's.  The fix is already out for the 11.1 code train (11.1-59.10) but 12.0 and 12.1 are still coming, due out later this quarter I'm told.

Link to comment
Share on other sites

1 hour ago, Charles Richardson1709154264 said:

My understanding is that the NetScaler team had to make some pretty major modifications to the kernel of the NetScaler in order to get around a bug in the newer Intel CPU's.  The fix is already out for the 11.1 code train (11.1-59.10) but 12.0 and 12.1 are still coming, due out later this quarter I'm told.

Thanks for the reply. I have logged a call with Citrix

Link to comment
Share on other sites

49 minutes ago, Joel Donaldson1709152440 said:

Is there some kind of CPU masking that can be done to get the VM to boot and then upgrade/downgrade the VM?

That is a fantastic question and something I think has promise - if I knew more about possible ways to apply CPU masking to a specific VM I would certainly be trying it in this case.  I say this because a trick I learned (and this is how I've kept NetScaler running in my environment of 14 Gen hardware) is to start NetScaler on 13 gen hardware (Dell R730 in my case), let it boot fine.  Then live migrate it from the R730 to my main pool of R740 servers.  The NetScaler will run on the R740 fine until it is shut down.  It will even reboot so long as it is a warm reboot (reboot the machine without it ever entering the "OFF" state) on the R740.  But any attempt to cold boot it on the R740 results in the crash.  So, yes, I'm sure if you could somehow mask whatever part of the new CPU's it doesn't like, it would probably boot and work just fine.  That said, I have no idea how one would do that or if it's even possible.  

Link to comment
Share on other sites

Hi to all, I have this morning same problem on new ESX 6.5.0, 676564

Dell PowerEdge R740xD

Intel Xeon Gold 6136

 

After un reboot of the VM the compatibility mode of the Netscaler had been upgraded to ESXI 6.5 (VM version 13)

--> The VM won't boot anymore with the TRAP message.

 

--> Resolution :

Hopefully I have old backup of that VM in VMWARE compatibility 9.

-->  And with this compatibility mode the Freebsd Boot without TRAP and my Nestscaler is UP again.

 

Link to comment
Share on other sites

  • 1 year later...

I am also stuck on not be able to run my NetScaler VPX on the new Dell R740 hardware.  Currently I am on 11.1 62.8nc but when I shut the VPX down and moved it over to my new pool, it would not boot.  I don't recall the exact message but it normally boots in about 20 seconds and in this case it did nothing.  I powered it down and moved it back to my old Dells and it worked just fine again.

 

I'm assuming I need to upgrade from 11.1, but to what version and how many jumps do I need to make?

Link to comment
Share on other sites

You need to make sure that the version of the NetScaler/ADC is supported on the version of XenServer you are using as I found out from the original post.

 

As you are using NetScaler/ADC 11.1 this is only supported on XenServer 6.2 and 6.5. From the XenServer HCL a minimum version of XenServer 7.1.0 CU2 is supported on the R740.

 

Therefore, if you are using XenServer 7.1 you should be able to use NetScaler/ADC versions 12.0, 12.1, and 13.0, XenServer 7.6 Supports only version 13.0, the supported versions can be found in the Support matrix and usage guidelines below:

 

https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html

https://docs.citrix.com/en-us/citrix-adc/12-1/deploying-vpx/supported-hypervisors-features-limitations.html

https://docs.citrix.com/en-us/citrix-adc/13/deploying-vpx/supported-hypervisors-features-limitations.html

 

If you have installed XenServer 8.0 there does not appear to be any versions of NetScaler/ADC that are supported on this version.

 

Hope this helps,

 

Dan

 

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...