Jump to content
Welcome to our new Citrix community!

Firmware 13.0 58.30


Recommended Posts

Just a warning that I installed build 58.30 on Friday (was given an advanced release due to the major issues I've been experiencing with the previous 2 releases of firmware) and it fixed a memory leak issue I was experiencing but when I failed over to this node the node crashed from some authentication module (says the crash dump). I've had a ticket open with Citrix since the end of February and not a happy camper.  Not sure if others are experiencing similar issues.

 

Cheers

Sandy

Link to comment
Share on other sites

What action causes the crash? The act of failing over? Or someone performing authentication? Or other?

 

We have a few critical issues we are waiting on a new release for, so I would like to upgrade to this build; but I would like more detail on what you are experiencing. Otherwise I may need to revert to 12.1.

 

Thanks for sharing the detail!

Edited by Ross Bender
add thanks
Link to comment
Share on other sites

The failover itself is successful and then shortly after it fails back and it took us a bit to realize it had crashed.  This is what I was told exactly "I am creating a new ticket with Engineering for a deeper analysis. As a result of the analysis by development we got that this crash is occurring in function “ns_aaa_process_webview_for_authv3” as a result of an authentication request.
The packet engine was attempting a memory copy but it appears as if the length of the copy is unusually high. This leads an access violation."

 

We are on 47.24 and soon after that upgrade I noticed my passive node would run out of memory after about week.  It took about 6 weeks for Citrix support to acknowledge I had a memory leak.  I upgraded to 52.24 and my memory leak was fixed however nobody could authenticate.  I have a VIP on my VPX that load balances multiple LDAP servers and LDAP just simply stopped working.  I had to go back to the bad memory leak version but since then the memory leak got so bad that I maybe had 15 min of uptime on a failover if I was lucky so I have been basically without HA for about a month.

 

Link to comment
Share on other sites

@Sandy Williams Have you had any update on the crash being experienced in 13.0 build 58.30? I upgraded one of our Netscalers to this version tonight and when doing some testing saw a crash on both HA nodes, so I'm afraid I'm seeing the same issue. I'm in the process of opening a support case.

 

Were you provided any workaround or a new build that resolves the issue? We had to update to build 58.30 to resolve other issues, so I'm afraid downgrading isn't feasible.

Link to comment
Share on other sites

Hi,

 

our ADCs are crashing on the new firmware too. After upgrading the first node of the HA Cluster, everything seems normal until the failover command. I saved the core and crash files, and reverted to backup snapshot. Firmware 47.24 runs stable, but has issues with native OTP Management Page, which i was hoping to resolve with the update.

How can i read core and crash files? I will open a case as well.

 

Greetings, Andre

 

There is a hint in release notes, did anyone try? I will need a new downtime to try.

 

Caching

In a cluster setup, a Citrix ADC appliance might crash, when:
- upgrading the setup from Citrix ADC 13.0 47.x or 13.0 52.x build to a later build
- upgrading the setup to Citrix ADC 13.0 47.x or 13.0 52.x build

Workaround:
During the upgrade process, perform the following steps:
- Disable all cluster nodes and then upgrade each cluster node
- Enable all cluster nodes after all the nodes are upgraded

[ NSHELP-21754 ]

Link to comment
Share on other sites

  • 3 weeks later...

I see 13.0-58.32 was released which mentions fixes for NSAUTH-8617 and NSHELP-21754. Unfortunately the release notes do not specify a fix for the issue we are facing (NSHELP-23418).

 

I'm curious to hear if the new build is stable or introduces more issues. :S

Link to comment
Share on other sites

Just upgraded a HA pair from 13.0 47.24 to 13.0 58.32. One node reboots every 5 minutes and the client can only log on to their virtual desktop through the Browser. When they try to connect through the Workspace App directly, it is stuck in a login loop.

 

Customer opened a Support case.

Link to comment
Share on other sites

On 6/10/2020 at 12:37 AM, Sandy Williams said:

Just a warning that I installed build 58.30 on Friday (was given an advanced release due to the major issues I've been experiencing with the previous 2 releases of firmware) and it fixed a memory leak issue I was experiencing but when I failed over to this node the node crashed from some authentication module (says the crash dump). I've had a ticket open with Citrix since the end of February and not a happy camper.  Not sure if others are experiencing similar issues.

 

Cheers

Sandy

 

We have also experienced the same nsaaad module crash after upgrading our ADCs to 58.30 on 12/06. Support ticket is still open with Citrix.

Our journey to 58.30 has not been without issue for the last couple of firmware upgrades.
We originally upgraded from 47.24 after a bug was found that caused the ADC to stop processing a users AD groups if the user was a member of duplicate groups, resulting in apps not being enumerated.
This was fixed in 52.24, so we upgraded our ADCs accordingly, after about a week, our LDAP authentication seemed to crash randomly.
After troubleshooting with support, escalation advised that there was a bug in 52.24 that after rebooting an appliance it resulted in "the internal transport layer service might get unregistered. As a result, any transport protocol service request on the appliance fails." Which was supposed to be fixed in 58.30 under [ NSNET-15252 ].
So we upgraded again, and are now facing daily ADC reboots, with the logs advising of nsaaad module crashes.

Still waiting on an answer from support around what has caused this, but my intuition tells me another firmware upgrade is imminent.

Link to comment
Share on other sites

  • 2 weeks later...
  • 1 month later...
On 7/29/2020 at 12:45 AM, Ross Bender said:

@Alex Henstra - Did you get any update on your issue where users are stuck in a login loop when connecting through the Workspace App?

 

@Matt Zaro1709155061 - Did you have any update on what was causing your issues? What version are you currently running, and are you experiencing any other issues with it?

I think the customer reverted back to 47.24 and later upgraded to Build 64.35 without issues.

Link to comment
Share on other sites

On 7/29/2020 at 8:45 AM, Ross Bender said:

@Alex Henstra - Did you get any update on your issue where users are stuck in a login loop when connecting through the Workspace App?

 

@Matt Zaro1709155061 - Did you have any update on what was causing your issues? What version are you currently running, and are you experiencing any other issues with it?

 

9 hours ago, Alex Henstra said:

I think the customer reverted back to 47.24 and later upgraded to Build 64.35 without issues.

 

 

The issue was found to be caused by a a buffer overflow in the nsaaad module caused by a miscalculation in group string length i.e the size of the total number of groups a user was a member of.
We had some users with a large number of AD groups (upwards of 600).
Whilst Citrix found that this was indeed a bug in the firmware code, we also found the use of a particular AD group was exacerbating the issue.

Citrix provided us a private fix, which alleviated the nsaaad module crashing and rebooting the ADC.
Their comment was that the fix would be included in an upcoming 64.x build as per the comment from @Alex Henstra 
However, in the interim we are still running with 58.30

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...