Charles Richardson1709154264 Posted February 11, 2018 Posted February 11, 2018 I have been running Netscaler VPX instances on XenServer for years with no issue. I recently bought new hardware - Dell R740 servers with dual Xeon Gold 6150's, 768GB of RAM and Tesla M10 GPUs to replace older Dell R620 servers. I moved all of our VMs, including the Netscalers, off of the old XenServer implementation on the R620's to the new pool on the R740's. XenServer is the latest (7.3) and all of my other VM's work just fine but upon booting the Netscalers I get "fatal trap 9" and dumped to a db> prompt. I even tried downloading fresh virtual appliances from citrix.com and get the same result right after importing then. However, when I import them onto one of the old R620 servers with the older Xeon E5 series chips, they boot right up. Anyone have any ideas on what could cause this? The Netscaler is my only guest running FreeBSD. Everything else is either Windows, CentOS or Red Hat and they all work fine. This leads me to believe this is something to do with the newer Xeon scalable chips or perhaps something else about the newer Dell hardware that the older FreeBSD that is being used for the Netscalers (8.4) cannot address. To confirm this I downloaded an ISO of FreeBSD 8.4 (same version that Netscaler VPX uses) and attempted to boot into it and got the same error. I tried booting it both within XenServer and directly on the R740 hardware. Newer versions of FreeBSD ISOs (11.1 is the other one I tried) boot up just fine but 8.4 will not boot on this newer hardware, whether in XenServer or directly on the bare metal. Netscaler VPX is v12-56 (latest). I did try importing older versions (10 and 11) with the same result. I've attached a screenshot of the error - it's the same error whether it's the original VMs or a newly imported appliance. Again, the error only occurs on the new R740's - the old R620's still boot these just fine. Thanks to anyone who has any thoughts!
Siddhartha Sarmah Posted February 11, 2018 Posted February 11, 2018 Doesn't look like XS 7.3 is supported https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html
Charles Richardson1709154264 Posted February 11, 2018 Author Posted February 11, 2018 9 hours ago, Siddhartha Sarmah said: Doesn't look like XS 7.3 is supported https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html Good to know. I wiped a host and loaded 7.0 on it and patched it to 7.0.0-125380c and got the same result; the VPX VM will boot on an R620 but not on an R740. "fatal trap 9" again. I've been running Netscaler VPX 12 on XenServer 7.3 since it came out on R620's with no trouble. I have a case open with both Netscaler support and XenServer support. So far, neither seems to have any ideas other than they need to lab it up.
Siddhartha Sarmah Posted February 12, 2018 Posted February 12, 2018 That's good info. Hope you have passed this on to Support. This seems like a H/W (perhaps processor) compatibility issue
Charles Richardson1709154264 Posted February 12, 2018 Author Posted February 12, 2018 2 minutes ago, Siddhartha Sarmah said: That's good info. Hope you have passed this on to Support. This seems like a H/W (perhaps processor) compatibility issue I did. So far all I have gotten is a note that it is not currently possible for them to go beyond FreeBSD 8.4. That is not comforting considering that is a 2013 release. They are up to 11.1 on FreeBSD. I agree - I think this is an incompatibility with the Xeon Scalable processors. Perhaps related to the requirement to run X2APIC in order to run XenServer at all on these processors.
Daniel Challinor1709152724 Posted February 13, 2018 Posted February 13, 2018 I have exactly the same issue as well using Xenserver 7.3 on Dell PowerEdge R640 Servers. Won't work with old or new NetScalers. Do you have a support case number with Citrix that I can reference when I log a support case? Regards, Dan
Charles Richardson1709154264 Posted February 15, 2018 Author Posted February 15, 2018 On 2/13/2018 at 11:29 AM, Daniel Challinor1709152724 said: I have exactly the same issue as well using Xenserver 7.3 on Dell PowerEdge R640 Servers. Won't work with old or new NetScalers. Do you have a support case number with Citrix that I can reference when I log a support case? Regards, Dan Netscaler case #75864945 and XenServer case #75864993
Daniel Challinor1709152724 Posted February 21, 2018 Posted February 21, 2018 The only way I could resolve this problem on the Dell PowerEdge R640 with Intel Scalable Processors (Silver) was by downgrading to XenServer 7.0 as per the NetScaler 12 supported hyper visor limitations document as mentioned by siddharthas https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html Everything is working fine now with the NetScaler.
Charles Richardson1709154264 Posted February 21, 2018 Author Posted February 21, 2018 34 minutes ago, Daniel Challinor1709152724 said: The only way I could resolve this problem on the Dell PowerEdge R640 with Intel Scalable Processors (Silver) was by downgrading to XenServer 7.0 as per the NetScaler 12 supported hyper visor limitations document as mentioned by siddharthas https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html Everything is working fine now with the NetScaler. Interesting - for me it still didn’t work. The problem there is that the 14 gen hardware is not officially supported by XenServer 7.0 - http://hcl.xenserver.org/servers/841/Dell_EMC_PowerEdge_R640 It’s sort of a catch-22. Do you want XenServer to be unsupported or do you want Netscaler to be unsupported? If you have the 14gen hardware, that seems to be your current choice.
Daniel Challinor1709152724 Posted February 21, 2018 Posted February 21, 2018 It really is a catch 22 situation. The response I received back from Citrix is below: "I have verified the issue with our Engineering team and as per Engineering team's confirmation we don’t support VPX on XS7.3 at present, currently supported versions are XS 6.5 and 7.0. Request you to please try with XS 7.0/6.5 and let us know if you have any issues with the same. Also I will be closing this case as of now, as this is out of support criteria as VPX on XS version 7.3 is not supported. Please refer the support matrix for the same as below: https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html" I have asked Citrix to re-open the case as I do not see this as being satisfactory that the customer I am implementing this for will be having to run an unsupported product. They are having this built so they can stop using XenApp 6.5 which will soon be EOL.
Jonathon Wiggins1709152709 Posted June 18, 2018 Posted June 18, 2018 We are having this issue as well. XenServer v7.1 LTSR CU1 NetScaler v11.1 or v12.1 Since XenServer v7.0 isn't supported any more, we need to get this working with LTSR. I have a ticket opened with Citrix Support.
Charles Richardson1709154264 Posted June 18, 2018 Author Posted June 18, 2018 5 hours ago, Jonathon Wiggins1709152709 said: We are having this issue as well. XenServer v7.1 LTSR CU1 NetScaler v11.1 or v12.1 Since XenServer v7.0 isn't supported any more, we need to get this working with LTSR. I have a ticket opened with Citrix Support. That’s good - I get the impression that the more people who open cases on this the faster it will get resolved. They have a fix so I’m not sure why they are delaying deploying it.
Tommy Gallagher1709159681 Posted August 14, 2018 Posted August 14, 2018 Hi Charles, On 6/18/2018 at 11:40 PM, Charles Richardson1709154264 said: That’s good - I get the impression that the more people who open cases on this the faster it will get resolved. They have a fix so I’m not sure why they are delaying deploying it. Do you know what the fix is?
Charles Richardson1709154264 Posted August 14, 2018 Author Posted August 14, 2018 3 hours ago, Tommy Gallagher1709159681 said: Hi Charles, Do you know what the fix is? My understanding is that the NetScaler team had to make some pretty major modifications to the kernel of the NetScaler in order to get around a bug in the newer Intel CPU's. The fix is already out for the 11.1 code train (11.1-59.10) but 12.0 and 12.1 are still coming, due out later this quarter I'm told.
Tommy Gallagher1709159681 Posted August 14, 2018 Posted August 14, 2018 1 hour ago, Charles Richardson1709154264 said: My understanding is that the NetScaler team had to make some pretty major modifications to the kernel of the NetScaler in order to get around a bug in the newer Intel CPU's. The fix is already out for the 11.1 code train (11.1-59.10) but 12.0 and 12.1 are still coming, due out later this quarter I'm told. Thanks for the reply. I have logged a call with Citrix
Joel Donaldson1709152440 Posted August 20, 2018 Posted August 20, 2018 Is there some kind of CPU masking that can be done to get the VM to boot and then upgrade/downgrade the VM?
Charles Richardson1709154264 Posted August 20, 2018 Author Posted August 20, 2018 49 minutes ago, Joel Donaldson1709152440 said: Is there some kind of CPU masking that can be done to get the VM to boot and then upgrade/downgrade the VM? That is a fantastic question and something I think has promise - if I knew more about possible ways to apply CPU masking to a specific VM I would certainly be trying it in this case. I say this because a trick I learned (and this is how I've kept NetScaler running in my environment of 14 Gen hardware) is to start NetScaler on 13 gen hardware (Dell R730 in my case), let it boot fine. Then live migrate it from the R730 to my main pool of R740 servers. The NetScaler will run on the R740 fine until it is shut down. It will even reboot so long as it is a warm reboot (reboot the machine without it ever entering the "OFF" state) on the R740. But any attempt to cold boot it on the R740 results in the crash. So, yes, I'm sure if you could somehow mask whatever part of the new CPU's it doesn't like, it would probably boot and work just fine. That said, I have no idea how one would do that or if it's even possible.
Jean-marc massou Posted August 22, 2018 Posted August 22, 2018 Hi to all, I have this morning same problem on new ESX 6.5.0, 676564 Dell PowerEdge R740xD Intel Xeon Gold 6136 After un reboot of the VM the compatibility mode of the Netscaler had been upgraded to ESXI 6.5 (VM version 13) --> The VM won't boot anymore with the TRAP message. --> Resolution : Hopefully I have old backup of that VM in VMWARE compatibility 9. --> And with this compatibility mode the Freebsd Boot without TRAP and my Nestscaler is UP again.
Travis MAIN NUMBER Rabe Posted September 16, 2019 Posted September 16, 2019 I am also stuck on not be able to run my NetScaler VPX on the new Dell R740 hardware. Currently I am on 11.1 62.8nc but when I shut the VPX down and moved it over to my new pool, it would not boot. I don't recall the exact message but it normally boots in about 20 seconds and in this case it did nothing. I powered it down and moved it back to my old Dells and it worked just fine again. I'm assuming I need to upgrade from 11.1, but to what version and how many jumps do I need to make?
Daniel Challinor1709152724 Posted September 16, 2019 Posted September 16, 2019 You need to make sure that the version of the NetScaler/ADC is supported on the version of XenServer you are using as I found out from the original post. As you are using NetScaler/ADC 11.1 this is only supported on XenServer 6.2 and 6.5. From the XenServer HCL a minimum version of XenServer 7.1.0 CU2 is supported on the R740. Therefore, if you are using XenServer 7.1 you should be able to use NetScaler/ADC versions 12.0, 12.1, and 13.0, XenServer 7.6 Supports only version 13.0, the supported versions can be found in the Support matrix and usage guidelines below: https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html https://docs.citrix.com/en-us/citrix-adc/12-1/deploying-vpx/supported-hypervisors-features-limitations.html https://docs.citrix.com/en-us/citrix-adc/13/deploying-vpx/supported-hypervisors-features-limitations.html If you have installed XenServer 8.0 there does not appear to be any versions of NetScaler/ADC that are supported on this version. Hope this helps, Dan
Travis MAIN NUMBER Rabe Posted September 16, 2019 Posted September 16, 2019 Ok - we got it figured out. I had to upgrade to 12.1 and then it worked on the new 14thGEN hardware.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.