Jump to content
Welcome to our new Citrix community!

Access Gateway broken after 13.0 84.10 or 84.11


Joost Sannen

Recommended Posts

Hi,

 

After upgrading from 13.0 83.29 to 84.10 or 84.11 all our Access Gateways are broken. After a successfull logon and before going to Storefront I see

 

image.thumb.png.9e0ffea83c808197016203bae46cd93e.png

 

Dutch for "Please select one of the following:" without any visible options. The URL is /logon/LogonPoint/index.html . The same happens with a default theme. Using Firefox it appears that cookies are refused because they are not valid or expired. We use LDAP and RADIUS without nFactor and a reqrite to hide the second password field. Very simple. 

 

Might be related  to some 'fixes' with SameSite cookie attribute? Is anyone else experiencing the same problem?

 

UPDATE: Problem is also present in all current versions of 13.0 and 13.1 

Edited by Joost Sannen
typo
Link to comment
Share on other sites

Suggestions Citrix did was to use advanced session policies https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/configure-gateway-session-policies-for-storefront.html#create-a-session-policy-for-web-browser-based-access and to Citrix Gateway -> Global Settings -> Under Authentication settings -> Change AAA Authentication settings disable static caching. 

 

Both produce the same result. Waiting for a GTM now.

Link to comment
Share on other sites

Found in debug logging on the ADC with the Access Gateway

 

-ServerPort 443 - ProtocolVersion TLSv1.1 - CipherSuite "NA" - Session New - Reason "Handshake failure-Internal Error"

Event: SSL_HANDSHAKE_FAILURE

 

[EDIT]

In packet tracing you see

  • ADC to Storefront 'Client Hello'
  • Storefront to ADC 'Fatal, Handshake Failure (40)'
  • Storefront connection reset (RST, ACK).
  • ADC to Storefront connection reset (RST, ACK).

 

Storefront is load balanced on another HA pair VPX. Strange is the setup has been working for ages and now

  • Frontend ADC 13.0 83.29 - Backend ADC 13.0.84.10 = OK
  • Frontend ADC 13.0 84.10 - Backend ADC 13.0.84.10 = ERROR
  • Frontend ADC 13.0 84.11 - Backend ADC 13.0.84.10 = ERROR
  • Frontend ADC 13.1 12.51 - Backend ADC 13.0.84.10 = ERROR

 

Diving into ciphers now.

 

Edited by Joost Sannen
more info
Link to comment
Share on other sites

Ok, never mind the ciphers. It does work when I change the Web Interface Address in the Session profile from https to http. Changing it from http to https was one of steps to resolve the problem in an earlier stage. 

 

So I wanted to reproduce the original problem of "Please select one of the following:" without any visible options again. I restored the config of our test ADC to the time of the initial problem and ... it works!

 

Only change we are aware of is that for another problem with Citrix ADM we upgraded the ADC with the same build 13.0 84.11 again. Maybe one or more files only got overwritten at the second attempt. 

Link to comment
Share on other sites

5 hours ago, Joost Sannen said:

Ok, never mind the ciphers. It does work when I change the Web Interface Address in the Session profile from https to http. Changing it from http to https was one of steps to resolve the problem in an earlier stage. 

 

So I wanted to reproduce the original problem of "Please select one of the following:" without any visible options again. I restored the config of our test ADC to the time of the initial problem and ... it works!

 

Only change we are aware of is that for another problem with Citrix ADM we upgraded the ADC with the same build 13.0 84.11 again. Maybe one or more files only got overwritten at the second attempt. 

Hi Joost, just to make sure we understand. In the first upgrade you ran into this problem. Then you installed the same version again, and the second time you dont have this issue? So just install it a second time and all is good? Did anyone have the same experience? I dont have a lab to test this without messing with our production MPX.

Link to comment
Share on other sites

1 minute ago, Sjoerd Dijsselbloem said:

Hi Joost, just to make sure we understand. In the first upgrade you ran into this problem. Then you installed the same version again, and the second time you dont have this issue? So just install it a second time and all is good? Did anyone have the same experience? I dont have a lab to test this without messing with our production MPX.

 

Hi Sjoerd, yes that's the only explanation I have. Would be great when someone can test this.

Link to comment
Share on other sites

After consulting Citrix I did another attempt and tested both ADC's of the HA pair afterwards. I can confirm the Access Gateways are working this time. Running version 13.0 84.11 now.

 

Due to lack of any other changes in the chain from internet to Storefront this must be some file(s) on the ADC didn't got overwritten at my first attempt. 

Link to comment
Share on other sites

9 hours ago, Joost Sannen said:

After consulting Citrix I did another attempt and tested both ADC's of the HA pair afterwards. I can confirm the Access Gateways are working this time. Running version 13.0 84.11 now.

 

Due to lack of any other changes in the chain from internet to Storefront this must be some file(s) on the ADC didn't got overwritten at my first attempt. 

 

Argghh..... Unfortunately the problem is still present for some users. It's not a client problem. Windows is fully patched, browser is up-to-date as is the Workspace app. Went back to 13.0 83.29 to let all users use the published resources.

Link to comment
Share on other sites

Hej guys who are experiencing the same problem: Can you confirm you're using “set ssl vserver xxxx -HSTS ENABLED -maxage 157680000” too?

 

This time it seems I can't reproduce the issue after disabling this setting and doing the same with a rewrite policy like CTX205221. NStrace files uploaded to Citrix for investigation.

Link to comment
Share on other sites

  • 3 weeks later...
On 2/14/2022 at 4:54 AM, Joost Sannen said:

Hej guys who are experiencing the same problem: Can you confirm you're using “set ssl vserver xxxx -HSTS ENABLED -maxage 157680000” too?

 

This time it seems I can't reproduce the issue after disabling this setting and doing the same with a rewrite policy like CTX205221. NStrace files uploaded to Citrix for investigation.

 

I ran into this same issue when using that setting.

Link to comment
Share on other sites

  • 3 weeks later...

In firmware 13.0 85.15 the problem still exists. Still waiting for Citrix. I did

 

set ssl vserver xxxx -HSTS DISABLED

add rewrite action REW_ACT_STS insert_http_header Strict-Transport-Security "\"max-age=157680000\""
add rewrite policy REW_POL_STS true REW_ACT_STS

bind lb vserver xxxx -policyName REW_POL_STS -priority 100 -gotoPriorityExpression END -type RESPONSE

 

for now and that works.

 

  • Like 2
Link to comment
Share on other sites

On 3/25/2022 at 2:35 AM, Ben Brayshaw said:

Has anyone tried to see if the new firmware build for 13.0 85.15 has fixed this issue? It's in the release notes as a fixed issue.

 

I am not able to find this as fixed issue in the release notes https://docs.citrix.com/en-us/citrix-adc/downloads/release-notes-13-0-85-15.html Are you able to copy the NSHELP ID? Thanks!

Link to comment
Share on other sites

  • 4 weeks later...

We've started having the same problem with the newer firmware versions.  The temporary workaround has been to disable HSTS.  We deploy a lot of Citrix Netscaler (ADC)'s and have had it all nicely scripted for years now so we get the SSL Labs A+ rating.  Disabling HSTS drops us down to an A rating, but for the time being it's all we can do.

 

Bummer, built a new Netscaler using 13.1 build 21.50 this week and the issue still exists.

> show ver

NetScaler NS13.1: Build 21.50.nc, Date: Apr  7 2022, 00:18:35   (64-bit)

Done

 

#VirtualServer is name of Virtual Server used for Citrix Gateway

set ssl vserver VirtualServer -ssl3 DISABLED -tls1 DISABLED -tls11 DISABLED -tls12 ENABLED -tls13 ENABLED -HSTS DISABLED -maxage 157680000
set ssl parameter -denySSLReneg FRONTEND_CLIENT
add ssl cipher SecureCiphers
bind ssl cipher SecureCiphers -cipherName TLS1.3-AES256-GCM-SHA384 -cipherPriority 1
bind ssl cipher SecureCiphers -cipherName TLS1.3-AES128-GCM-SHA256 -cipherPriority 2
bind ssl cipher SecureCiphers -cipherName TLS1.3-CHACHA20-POLY1305-SHA256 -cipherPriority 3
bind ssl cipher SecureCiphers -cipherName TLS1.2-ECDHE-ECDSA-AES128-GCM-SHA256 -cipherPriority 4
bind ssl cipher SecureCiphers -cipherName TLS1.2-ECDHE-ECDSA-AES256-GCM-SHA384 -cipherPriority 5
bind ssl cipher SecureCiphers -cipherName TLS1.2-ECDHE-RSA-AES128-GCM-SHA256 -cipherPriority 6
bind ssl cipher SecureCiphers -cipherName TLS1.2-ECDHE-RSA-AES256-GCM-SHA384 -cipherPriority 7
bind ssl vserver VirtualServer -certkeyName SSL

 

Glad I found others with the same problem.  Like I said, we've been doing this for years and have had it nicely scripted and it only broke recently.  Too bad a mature product is running in to unexpected bugs.  

 

Previously used:

set ssl vserver VirtualServer -ssl3 DISABLED -tls1 DISABLED -tls11 DISABLED -tls12 ENABLED -tls13 ENABLED -HSTS ENABLED -maxage 157680000

 

 

Link to comment
Share on other sites

  • 3 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...