Gateway Callback fails after upgrade to Storefront 2203

[Chris McKenzie1709161237]

Today I tried upgrading two separate Storefront server groups from 1912 LTSR CU4 to 2203 LTSR, the installation was successful however afterwards users trying to login to the web portal were getting a "Cannot complete your request" message

When users try logging in we see this error logged on the Storefront server:

The AG Web Service at: https://my.gateway.com/CitrixAuthService/AuthService.asmx failed with the following error. This endpoint will be ignored until: 3/24/2022 2:41:50 AM
Citrix.DeliveryServices.Authentication.CitrixAGBasic.Exceptions.AGCommunicationException, Citrix.DeliveryServices.Authentication.CitrixAGBasic, Version=3.23.0.0, Culture=neutral, PublicKeyToken=null
A communication error occurred while attempting to contact the Citrix Gateway authentication service at https://my.gateway.com/CitrixAuthService/AuthService.asmx. Check that the authentication service is running.
   at Citrix.DeliveryServices.Authentication.CitrixAGBasic.Client.AGClient.GetAccessInfo(String sessionId, String username, String domain)
   at Citrix.DeliveryServices.Authentication.CitrixAGBasic.Client.CitrixAGBasicWebService.GetAccessInfo(String sessionId, String username, String domain)

System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The request was aborted: Could not create SSL/TLS secure channel.
   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Citrix.DeliveryServices.Authentication.CitrixAGBasic.AGAuthService.AuthenticationServiceSoap.GetAccessInformation(String sessionId, String username, String domain)
   at Citrix.DeliveryServices.Authentication.CitrixAGBasic.Client.AGClient.GetAccessInfo(String sessionId, String username, String domain)

 

Our users login via a Citrix Gateway with Storefront configured to fully delegate authentication to the gateway (smart card login type / Federated Authentication Service)

The updated server can definitely hit the Gateway and was working fine before the upgrade, users connecting to the other Storefront server that's still running v1912 can login successfully

 

We have separate regional deployments with their own Storefront servers and Gateway appliances, both environments have the same issue

 

Has anyone else had this issue after installing Storefront version 2203?

 

[Thomas Klein]

Hello,

 

same pain in my case. Update from CVAD 2112 (Storefront 1912 CU4) -> 2203, the CAG callback also fails with "Could not create SSL/TLS secure channel".

SSL Labs A+ Rating on ADC existing. There is no FAS present, the configuration is more simple with CAG and clientless Access.

 

I'm curious where the solution is hidden.

 

Best regards

Thomas

[Andy Vanderbeken]

Same boat. Just finished my upgrade from Storefront 1912 LTSR CU4 to 2203 and Windows Event viewer (on the Storefront + DDC server) lists:

 

An SSL connection could not be established: None of the SSL cipher suites offered  were accepted by the server.. This message was reported from the Citrix XML Service at address https://(storefrontURL)/scripts/wpnbr.dll. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.

 

So as the message says the XML service which is configured to use https (in my case and most others cases I assume) just shuts itself down since after the upgrade which leads to an empty page (no apps nor desktops visible) when logging in directly to Storefront URL over https.

 

Furthermore I want to add that this clearly is a pure Storefront problem, not Netscaler since after the 'successful' upgrade I notice immediately that logins to the internal Storefront url over https also don't work anymore and in that most basic scenario Netscaler gateway isn't even in the picture yet.

 

Tip for others: As a temporary workaround you can disable https on the delivery controller communication in storefront and fall back to http which makes logins immediately possible in scenario's when logging in directly (=through the internal storefront server url over LAN or 3d party VPN) but not indirectly (through Netscaler Gateway):

 

 

 

 

 

Through Netscaler you'll still get "Cannot complete your request"

 

 

So yeah I as well would like to know what went wrong here with https communication to DDC in the 2203 build. Clearly this can't have been QA tested properly by Citrix as what @Thomas Klein and myself just did here is the most cookie-cutter upgrade one can expect: a clean working QA installation proven to be working and then 'successfully' upgraded to 2203.

 

Citrix WTH are you guys doing lately ??

 

[Thomas Klein]

Yes, what (if any?) QA did here is questionable ?

 

Had a look at a wireshark trace regarding Storefront callback looking at TLS Handshake from Storefront to ADC:

 

 

I'm not sure.... but TLS v1.2 Record Layer using TLS 1.0 seems to be strange....

Incoming SSL Traffic to Storefront works, but maybe SSL Library used for SSL Outgoing Traffic is broken?!

 

[James Kindon]

https://docs.citrix.com/en-us/storefront/current-release/whats-new/known-issues.html
 

ciphers….?

[Chris McKenzie1709161237]

1 hour ago, James Kindon said:

https://docs.citrix.com/en-us/storefront/current-release/whats-new/known-issues.html
 

ciphers….?

 

I did see that however the conditions in the description state

1. It happens if you use the Citrix ADC load balancing feature to distribute the load to the delivery controller servers.
2. And StoreFront is using HTTPS to communicate with the load balancing delivery controller services. 

Neither of these are true in my case

It also looks like that known issue has been around since Storefront 1912 CU2 and not something new with 2203

[Thomas Klein]

3 hours ago, James Kindon said:

https://docs.citrix.com/en-us/storefront/current-release/whats-new/known-issues.html
 

ciphers….?

Hello,

 

some news:

 

CAG Callback works successful, if CAG vServer offers any TLS v1.0 Ciphers / TLS v1.0 enabled in the SSL Profile. If not (in case of SSL Labs A+ Rating / only TLS v1.2 / TLS v1.3 enable), Callback throws the known error.

 

So should that REALLY be the solution to re-enable TLS v1.0 on the CAG?

Hopefully not.

 

Best regards

Thomas

[Chris McKenzie1709161237]

17 hours ago, Thomas Klein said:

Hello,

 

some news:

 

CAG Callback works successful, if CAG vServer offers any TLS v1.0 Ciphers / TLS v1.0 enabled in the SSL Profile. If not (in case of SSL Labs A+ Rating / only TLS v1.2 / TLS v1.3 enable), Callback throws the known error.

 

So should that REALLY be the solution to re-enable TLS v1.0 on the CAG?

Hopefully not.

 

Best regards

Thomas

 

Thanks Thomas, you're right, I enabled TLS 1.0 on the Gateway and I can now successfully login to Storefront 2203

 

Considering that Storefront 1912 supported TLS 1.2 I can't imagine that this behaviour is intentional... at least I hope so

[Kristof Vertongen]

Same issue here, can be fixed by enabling TLS 1.0 on the Gateway Virtual Server again.

 

Strangely enough, the issue only happened with customers who are using the Citrix Gateway Advanced VPX license.

Our internal NFR ADC VPX Platinum license Gateway and one customer who is using the old free Gateway license didn't have the issue.

 

All devices were configured for Qualys SSL A+ in the same way.

[Andy Vanderbeken]

1. For specifically logins via CAG I can confirm that (re)enabling TLS v1.0 on the Netscaler made logins possible once more.  However I still got an additional error that no ticket could be obtained (when trying to launch a published app/desktop) until I disabled https on the STA config Storefront as well:

 

 

2. Aside from logins coming from CAG (passed through to the Storefront server) being broken, can anyone else please confirm that they also see the basic internal https Storefront functionality broken as in that after a successful login to https://StorefrontURL you don't see any published apps/desktops until you disable https in the Storefront "Manage delivery controllers" setting like this:

 

 

I tried disabling all Chrome browser procotols except TLS 1.0 but => no difference

I tried creating a new receiver website in Storefront => no difference

 

 

 

So far the real source of this problem looks to me like the Storefront XML service in this 2203 LTSR CU0 build no longer being able to accept or respond to those SSL cipher suites correctly.

 

In conclusion so far the solution for my case in order to restore all existing functionality (logins through CAG & directly to Storefront) is 3 things: reenabling TLS v1.0 and disabling https for STA and delivery controllers as shown in the 2 screenshots above.

 

PS: There was never loadbalancing active in my scenario so the known issues description doesn't seem to fit my case either

[Andreas Koch]

Jup, enabling TLS1 did the job.
Oh oh oh !

Spending some money for quality testing would be a good idea.

[Miguel Contreras]

Thanks for bringing this up!

 

That is strange...

 

Seems like it could be an issue with the cipher suites being negotiated.

 

Are any of you able to try the following?

1. Change the cipher suite order on the Gateway vserver so that the following are at the top of the list:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
    
2. Change the cipher suite order in the Storefront servers so that these are at the top:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
    
3. Change the cipher suite order in the DDCs so that these are at the top:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

To adjust the cipher suite order in the Windows servers, you can use GPO or local policy:

Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order

 

You can refer to Microsoft's doc for the default order supported in the different Windows versions: https://docs.microsoft.com/en-us/windows/win32/secauthn/cipher-suites-in-schannel

 

NOTE: do not delete any of the cipher suites in the Windows servers' list. Instead, move the ones I mentioned to the top of the list in that order. Deleting cipher suites from that list could potentially cause issues with other apps/services on those machines.

[Andreas Koch]

I tried this, did not work for me.
My soloution was revering storefront to 1912cu4

[Miguel Contreras]

1 hour ago, Andreas Koch said:

I tried this, did not work for me.
My soloution was revering storefront to 1912cu4

 

Thanks for the feedback, Andreas. What version of ADC are you running?

[Andreas Koch]

@Miguel Contreras

 

I'm running on ADC 13 and 13.1 all latest firmware

[Andy Vanderbeken]

15 hours ago, Miguel Contreras said:

Thanks for bringing this up!

 

That is strange...

 

Seems like it could be an issue with the cipher suites being negotiated.

 

Are any of you able to try the following?

1. Change the cipher suite order on the Gateway vserver so that the following are at the top of the list:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
    
2. Change the cipher suite order in the Storefront servers so that these are at the top:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
    
3. Change the cipher suite order in the DDCs so that these are at the top:
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

To adjust the cipher suite order in the Windows servers, you can use GPO or local policy:

Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order

 

You can refer to Microsoft's doc for the default order supported in the different Windows versions: https://docs.microsoft.com/en-us/windows/win32/secauthn/cipher-suites-in-schannel

 

NOTE: do not delete any of the cipher suites in the Windows servers' list. Instead, move the ones I mentioned to the top of the list in that order. Deleting cipher suites from that list could potentially cause issues with other apps/services on those machines.

 

I tried this to see if it would affect the internal Storefront https login. In addition to changing the order I did a full IISreset after each gpupdate and then a new login to the Storefront url directly: No effect or noticeable change.

 

Then I took all other common win10 combinations of cipher suites from the Microsoft site and tried those instead of the default suites=> no difference. I even added the ones that are not enabled by default.

 

I truly believe that this is not a cipher priority problem at all. In fact I believe it's not even a matter of which ciphers -regardless of order-  are being offered since the client side (netscaler or windows) was not changed at all when this problem was introduced. Also when I test a login from the very DDC+Storefront server to itself (client and server are the same in this scenario) on the storefront https URL it doesn't work either. The facts and all logic around it leads me to believe that even though the ciphers at windows are perfectly fine, the XML component is broken as in: can no longer use the ciphers correctly to process incoming offers. 

 

After all, regardless of the cipher order, as long as some compatible offer is found there should be a match, yet in my most basic example of the problem here it's just a windows client OS logging on to the internal storefront URL. There is no way that the following default cipher list in the OS (shown below) doesn't allow for at least 1 match:

 

TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256
 

And upon my every attempt with all ciper suite combinations the Delivery Controller logs seem to confirm this theory as far as I can see:

 

 

All this leads me to conclude that XML component is broken in this build and we simply need a hotfixed version of it.

 

 

 

[Miguel Contreras]

8 minutes ago, Andy Vanderbeken said:

 

I tried this to see if it would affect the internal Storefront https login. In addition to changing the order I did a full IISreset after each gpupdate and then a new login to the Storefront url directly: No effect or noticeable change.

 

Then I took all other common win10 combinations of cipher suites from the Microsoft site and tried those instead of the default suites=> no difference. I even added the ones that are not enabled by default.

 

I truly believe that this is not a cipher priority problem at all. In fact I believe it's not even a matter of which ciphers -regardless of order-  are being offered since the client side (netscaler or windows) was not changed at all when this problem was introduced. Also when I test a login from the very DDC+Storefront server to itself (client and server are the same in this scenario) on the storefront https URL it doesn't work either. The facts and all logic around it leads me to believe that even though the ciphers at windows are perfectly fine, the XML component is broken as in: can no longer use the ciphers correctly to process incoming offers. 

 

After all, regardless of the cipher order, as long as some compatible offer is found there should be a match, yet in my most basic example of the problem here it's just a windows client OS logging on to the internal storefront URL. There is no way that the following default cipher list in the OS (shown below) doesn't allow for at least 1 match:

 

TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256
 

And upon my every attempt with all ciper suite combinations the Delivery Controller logs seem to confirm this theory as far as I can see:

 

 

All this leads me to conclude that XML component is broken in this build and we simply need a hotfixed version of it.

 

 

 

 

 

Hey Andy,

 

Just in case... When changing the cipher suite order, a reboot is required for the changes to take effect. Did you reboot after making the changes? Or did you just run gpupdate without rebooting?

[Andy Vanderbeken]

I did a gupdate + iisreset instead of full reboot. Should that not be enough ?

[Miguel Contreras]

23 minutes ago, Andy Vanderbeken said:

I did a gupdate + iisreset instead of full reboot. Should that not be enough ?

 

Unfortunately not. Full OS reboot is required for cipher suite order changes to take effect.

[Andy Vanderbeken]

1 hour ago, Miguel Contreras said:

 

 

Hey Andy,

 

Just in case... When changing the cipher suite order, a reboot is required for the changes to take effect. Did you reboot after making the changes? Or did you just run gpupdate without rebooting?

 

just to have all bases covered, I repeated the tests with the one you suggested:

 

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256

 

as well as an extended version of the default Win10:

 

TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_RC4_128_MD5,TLS_RSA_WITH_DES_CBC_SHA,TLS_DHE_DSS_WITH_DES_CBC_SHA,TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA,TLS_RSA_WITH_NULL_MD5
 

No changes.

[Andy Vanderbeken]

1 hour ago, Miguel Contreras said:

 

 

Hey Andy,

 

Just in case... When changing the cipher suite order, a reboot is required for the changes to take effect. Did you reboot after making the changes? Or did you just run gpupdate without rebooting?

 

just to have all bases covered, I repeated the tests (but this time with a full reboot after each change) with the one you suggested:

 

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256

 

as well as an extended default win10 cipher suite:

 

TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_RC4_128_MD5,TLS_RSA_WITH_DES_CBC_SHA,TLS_DHE_DSS_WITH_DES_CBC_SHA,TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA,TLS_RSA_WITH_NULL_MD5
 

No change, the problem stays.

[Miguel Contreras]

Thanks for following up, Andy. I'll submit this feedback to the SF team.

[Timo Hamalainen1709153432]

This workaround worked in our test StoreFronts:

The issue can be workarounded to update the targetFramework properties in the web.config files of store service and authentication service folder, by following steps:

1. browse to C:\inetpub\wwwroot\Citrix\%store name% folder(or the store service folder of your custom drive) and locate the web.config file

2. locate the line of '<httpRuntime targetFramework="4.5" executionTimeout="300" appRequestQueueLimit="100"......' and update the targetFramework value to be "4.7", where it's currently 4.5

3. locate the line of '<compilation debug="false" targetFramework="4.5" />' and update the targetFramework value to be "4.7", where it's currently 4.5

4. browser to C:\inetpub\wwwroot\Citrix\"store name%Auth folder(or the authentication service folder of your custom drive) and locate the web.config file, and do step#2 and step#3 for this file as well.

5. step#1 - step#4 needs to be performed for every store/authentication service that had been created

[Aaron OReilly]

I have successfully tested a fix on Server 2022 with upgraded StoreFront 2203 based on how the .net framework handles the default TLS versions used on a system. This post pointed me in this direction  https://docs.microsoft.com/en-us/dotnet/framework/network-programming/tls#systemdefaulttlsversions 

 

Applying the following reg (and a reboot) tells .Net to use the system TLS for the default which appears to tell .net 4.5 that StoreFront is hard coded with to use the secure TLS versiosn available to the OS rather than its own older TLS defaults. This forces the 4.5 behaviour to be the same as what it is by default with newer .Net versions.

 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]

"SystemDefaultTlsVersions" = dword:00000001

"SchUseStrongCrypto" = dword:00000001

 

After this StoreFront can communicate correctly with the Delivery Controller XML when using SSL (my StoreFront and XML are on the same box in my lab), and with the Gateway callback  if using it with a Gateway. So apps/desktops now enumerate and launch correctly using Gateway or StoreFront.

 

[Edit] It looks like a new StoreFront deployment sets .Net to 4.7 in the web.config file as per the fix in the post above, but a StoreFront upgrade fails to make the change to the existing files and leave it as 4.5 causing the issue so the web.config edit is probably the correct workaround.

[dmohrmann24]

Hi,

we also had this issue with one of our customers and I also was able to reproduce it in my testlab. Aaron O'Reilly found a workaround:
Do the following on your Storefront servers:
 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]

"SystemDefaultTlsVersions" = dword:00000001

"SchUseStrongCrypto" = dword:00000001

You don't need a reboot. After that SF can talk to XML and STA via SSL again!