SSL31 error when starting an application using Citrix Workspace 22.3

[J.R. van Doornik]

Unsure why this is happening, but I figure I might ask around to see if anyone might have suggestions on what to try next. 

 

We have thin-clients connecting to our Citrix environment with no issues. Citrix Production environment is setup as a vDisk via Provisioning Services. Then there are some stand-alone Citrix servers offering specific software packages we don't want for all users. 

 

Some of our users connect to a remote Citix environment offering some apps from a supplier. We offer the links to that website within our Citrix environment, so users login to the Citrix environment, get presented with their Citrix desktop, click the URL which opens the browser and accesses the website of the supplier. They then login with the account the supplier provided for them, and end up in a StoreFront NetScaler environment of the supplier. Normally they click any of the offered icons to start the application being on offer. To facilitate this chain of usage the Workspace App was also installed on the Citrix desktop.

 

To reiterate - The user connects to OUR Citrix environment (which starts a desktop), and then connects to a REMOTE Citrix environment from a supplier from that desktop (using the installed webbrowser and Workspace App from the vDisk), making this a Citrix-on-Citrix connection. The supplier starts a Citrix application which the user can then use.

 

Quite recently this started failing with an SSL31 error. The user is able to connect to the website from the vDisk offered webbrowser, login and gets the icons presented. Clicking an icon does download te ICA file, but it seems that as that ICA file is passed on to the Workspace App the error occurs. 

 

The funny thing is that it is NOT failing if users try to connect directly from their endpoints to the suppliers site, nor does it fail on the stand-alone Citrix servers (which also use the Citrix-on-Citrix approach). The only place it fails is using the prmary vDisk based Citrix environment. Even worse, even on the vDisk environment the failure isn't consistent... Usually it fails, but there are known cases where it does work. 

 

Initially the issue seemed to stem from the usage of Google Chrome, since if the user reverted to IE there were no problems connecting. More recently it was found that Edge also seemed to work, but that's also been debunked by now as Edge has shown that error a couple of times aswell. And with IE being an obsolete browser being phased out by Microsoft, that solution also isn't feasable. Plus it doesn't make sense that the error occurs when the Workspace App opens the ICA file, which the browser at that point has nothing to do with anymore. 

 

The next thing we checked were the versions of the Workspace App but atleast between the vDisk and stand-alone Citrix servers those are identical (22.3.101.4). The former has issues, the latter does not. So that's been ruled out aswell. We then rebuilt the whole vDisk from scratch... problem did not abate. Certificates then? Seems to be fine in the whole certificate chain presented from within the browser when accessing the Storefront page of the supplier.

 

I then had a go at looking at the logging of the Citrix Workspace App in hopes it would provide me a readable log of some sort indicating where it was having a problem. Instead I get about 70 MB of data from a lot of sources but from the looks of things nothing to really indicate where the issue stems from (provided I can open the files and actually read their contents). So I'm considering that a bust too... Unless someone has a more pinpoint idea of what files to pick apart.

 

I might be down to actually installing a packet-sniffer on the endpoint and see if that picks something up, but I'm fairly sure the packetsniffer will only see the network-packets being sent out from the system, so that would be post-certificate encryption. And thus unreadable data. So I'm somewhat ruling that out as a possible solution avenue aswell.

 

I'm mostly out of ideas on what to try further, and would love to hear some input on what we might be able to try to get this annoying problem out of our environment. 

[J.R. van Doornik]

Nobody having any idea what might be wrong, or how we might go about further diagnosing this? ?

[CarlStalhood]

When you click an icon, Workspace app takes over. So the SSL connection is between the Workspace app and the VDA and the browser should have nothing to do with it.

 

Do you have any firewalls or proxy servers inspecting SSL traffic?

[J.R. van Doornik]

As stated, the icon does look like it's downloading the ICA file (right-click - Save As provides the ICA file for download). So any error is (I assume) part of the workings of the Workspace App as it processes the provided ICA file.  

 

If SSL inspection was an issue, I'd have also expected the browsing to the website to be an issue, but beside that, I'm willing to entertain the notion. 

 

Depending on where the client comes from there can be 1 or even 3 firewalls involved (WAN site-to-site interconnectivity and all that). I'm not counting any firewalls that are beyond my control (which the vendor may have implemented), nor the Windows software firewalls. 

 

The vendor has a dedicated connection which does NOT terminate in our datacenter, and to separate that traffic from our network, there is a firewall there that does certificate inspection. That firewall ONLY controls traffic to and from the vendor, and does nothing else. 

 

So basically the three variations are 

 

10% successrate - Citrix desktop vDisk -> DC firewall -> Private WAN -> Mainsite firewall -> Vendor line firewall -> Citrix environment of vendor

vs 

60% successrate - Citrix desktop dedicated disk -> DC firewall -> Private WAN -> Mainsite firewall -> Vendor line firewall -> Citrix environment of vendor

vs

100% successrate - Local desktop -> Mainsite firewall -> Vendor line firewall -> Citrix environment of vendor

 

It has been found that local started connection (atleast on the main site) that ONLY pass through that firewall (the Vendor line firewall above) always work. So while it does do certificate inspection on the SSL, it does not seem to be the root of the problem. 

 

When the connection is initiated from the Citrix desktop the users use, that Citrix desktop physically lives in our datacenter. It then is passed to the DC firewall in the datacenter, forwarded to the Mainsite firewall, which then passes the traffic to the Vendor line firewall which I already described above to get to the vendor. 

 

Neither the DC firewall nor the Mainsite firewall do certificate inspection, so that can't really be a cause then if it's indeed inspection taking place. 

 

Besides that, if the inspection was the cause, I'd have expected this problem to occur from October 2022 if anything since that was the last time we actually dug into the routing in the datacenter and reconfigured that. The problem however did not crop up until atleast January. Note that we did NOT do any routing reconfigurations on the main site where the breakout to the vendor lives.

 

Also, the certificate inspection would be active for all sessions, all the time. Which does not explain the fact that sometimes it does work, while other times it fails. Also the difference between the vDisk operated Citrix servers and the ones that have a single disk (where the former seems to have a 10% success rate, and the latter comes closed to 60% success when connecting to the vendor) wouldn't make sense if it was a firewall in between.

 

While I can try and disable certficate inspection on the firewall to the vendor that has it switched on, I'm not sure if that will do much, based on the local connections that always work. 

[J.R. van Doornik]

Addendum... I should also refer to the traffic flowing back to the client... 

 

The Vendor line firewall has NO rules governing that, and will thus allow only the traffic passed through it to the remote site to return. So that does have the certificate inspection on it. 

The mainsite firewall does NOT do certificate inspection to the WAN, and neither does the DC firewall.

[J.R. van Doornik]

At this point there is a workaround (which sidesteps the whole Citrix desktop environment), and I'm not getting any further feedback from the vendor, or their intermidiate within our company. 

 

Since this thread also doesn't seem to yield much further insight, I'm considering the issue unresolved but closed anyway. If someone still wants to hazard a guess or provide some insight I'm all ears tho. 

[J.R. van Doornik]

Said workaround is popping up problems here and there (mainly with drive mappings not being present on the thin clients being used), but there are other workarounds for the issue. 

 

At this point it seems as if there is an acceptance to wait for the deployment of our new desktop environment moving mostly away from the Citrix desktop solution that we currently have. That would cut out the Citrix-on-Citrix scenario, and based on our expirience seems to mostly bypass the issue.

 

Still... Anyone wanting to chip in: please do. 

[Aldric]

When you get this problem, did you maybe try to reset citrix workspace app?

Right click on the workspace app --> Advanced Preferences --> Reset Citrix Workspace

 

I had exactly the same today and solved in this way.

[J.R. van Doornik]

I'll forward that to the two colleagues who are more readily faced with the issue, but already thanks in advance for the suggestion. If I get feedback on that I'll be sure todrop it to the thread. 

 

The application owner has stated that 'waiting' for the new deployments may not be an achievable scenario, so he is still pushing for solving the matter. Since I do not have (nor will get) an account for the remote Citrix environment, I'm still very much dependant on someone who does have an account to test/try things. And the diagnosing seems very limted in the means I have to obtain and sort through all that. But from the looks of things it does fall back to us still to sort out what the heck is happening, and how to resolve it.

 

Any further suggestions are appreciated. 

[J.R. van Doornik]

Collegue tried this today, and actually got some results... Which may or may not be something more long-term... Testing is still ongoing, so I'm witholding any optimism until being informed otherwise. 

 

His response in testing this:

 

Started Chrome, logged on at the vendor portal, got the apps from the vendor StoreFront environment and kicked off the app. 

SSL error 31 appeared

Closed the error notification, and ran the reset  of the Citrix Workspace as suggested, which apparently took a bit of time. 

Then tried the app again (so never logged off from the portal), and was faced again with the SSL error 31. 

Closed the application, portal and Chrome. 

 

He then reset the Citrix Workspace app again, and then (for some reason I can't really explain) started up Firefox to try the same in that browser. 

Logged on to the portal, downloaded the ICA file, and started that, and the app started without issue. 

He then returned to Chrome to test it there, and can confirm that both Chrome and Edge now work as well. 

 

He is planning to run further testing tomorrow, so I hope to be able to add some further details tomorrow... 

[J.R. van Doornik]

I apologize for not getting back sooner. It's been kind of hectic here. 

 

That said: 

 

My colleague spoke to 4 users on the 16th of May, of which 3 reported that the vDisk environment was functional. So they could work on the primary work environment and make a connection to the vendor applications. 

 

The last user found that his connection from home to the vDisk environment was NON-functional, and when connecting at the office it WAS functional. (Do note, we do NOT pull ports, printers or disks along in the Citrix session to our Citrix environment. The only thing going across the wire is the KVM information).

 

My IT contact himself tried this on the 16th, and got the SSL31 failure once again, despite it working from his home-location the day before. Resetting the client (he tried 3 times!) did NOT resolve his problem in the office. 

 

Since a download-share (P: drive) is needed for certain tasks (so the Citrix session of the vendor DOES allow for disks to be visible within their sessions), my colleague made that mapping, thn tried again, and found NO SSL31 error. 

 

Then on the 17th he tried again from home, and found that the applications would start without the SSL31 error. 

 

The results are all over the place and I'm at a loss as to where the heck I can still look to get this sorted. It's not our own environment seeing everything else works without any issue. It's just starting applications on the vendors remote Citrix environment that causes issue, and even that isn't consistent (often it doesn't work, and sometimes it does). 

 

Grabbing the logs from the client might work, but I get 60 MB of data files of various sorts encompassing the whole system configuration, and not just what is happening in regards to the actual SSL connection. I'm unsure what folder and file I'd need to dig into to get further insight in what the client may be complaining about. 

 

I'm hoping the vendor might see something on his end in the logs, but despite multiple requests I have yet to hear anything back on that. 

[Andy Barendregt]

We had a similar problem with connecting to a remote Citrix Environment. Workaround for us was to import the generated cr file.
After a bit more troubleshooting we saw that importing the cr file lead to an addition of a missing root certificate.
Normally when connecting to the site, the certificate is auto imported in the store but with the changes in Edge, this is no longer the case.
The Vdisk Image where everything worked normally also had a version of Edge that still supported auto-import of the certificate.

You can test it by disabling the specific feature flag in Edge.
edge://flags/#edge-microsoft-root-store-enabled

Let me know if this works for you.

[J.R. van Doornik]

I'll pass this on to the team, and might actually get some time somewhere to have a test with this aswell... 

 

I'll keep this thread updated on (possible lack of) progress ? Thanks! Suggestions are always appreciated ?

[Robert Paul1709161642]

On 5/24/2023 at 8:34 AM, Andy Barendregt said:

We had a similar problem with connecting to a remote Citrix Environment. Workaround for us was to import the generated cr file.
After a bit more troubleshooting we saw that importing the cr file lead to an addition of a missing root certificate.
Normally when connecting to the site, the certificate is auto imported in the store but with the changes in Edge, this is no longer the case.
The Vdisk Image where everything worked normally also had a version of Edge that still supported auto-import of the certificate.

You can test it by disabling the specific feature flag in Edge.
edge://flags/#edge-microsoft-root-store-enabled

Let me know if this works for you.

I am having this issue only on corporate-owned computers and while using RSA SecurID authentication from the external Netscalers into the internal StoreFront environment. Like J.R., it downloads the .ica file, but when I attempt to open the file > it gives the SSL Error 31. Same computer/same browser, but using PKI Cert authentication, the .ica file opens.

 

I tried the disabling of the edge-microsoft-root-store-enabled; however, that did not result in a change.

 

If I am on a personal laptop logging in through the external Netscalers, then I can authenticate with either RSA SecurID or PKI certificate, and I can successfully open the .ica file and subsequent VDI.

 

[J.R. van Doornik]

Just a heads-up. Since we have a bank holiday in the Netherlands on Monday I (and anyone able to test) won't be in the office Monday. That being said, a lot of people like long weekends, so most anyone able to test this has snuck in a day off today too... 

 

Tuesday I work from home, so I do hope to get around to this on Wednesday the 31st. 

 

In regards to the update Robert made:

 

We found that NOT using the Citrix VDI environment and going directly to the vendors Citrix environment does work in 100% (I'm being cnservative and would call it 90%) of the time. It's the issue where our Citrix VDI environment is used to open the Citrix environment of the vendor that fails. 

 

Not sure what you mean wth 'PKI Cert authentication' and the ica file opening with that method tho. 

[Andy Barendregt]

On 5/25/2023 at 4:20 PM, Robert Paul1709161642 said:

I am having this issue only on corporate-owned computers and while using RSA SecurID authentication from the external Netscalers into the internal StoreFront environment. Like J.R., it downloads the .ica file, but when I attempt to open the file > it gives the SSL Error 31. Same computer/same browser, but using PKI Cert authentication, the .ica file opens.

 

I tried the disabling of the edge-microsoft-root-store-enabled; however, that did not result in a change.

 

If I am on a personal laptop logging in through the external Netscalers, then I can authenticate with either RSA SecurID or PKI certificate, and I can successfully open the .ica file and subsequent VDI.

 

Did the import of the .CR file work for you?

[J.R. van Doornik]

I apologize. Wanted to test today, but dependant on users to be present since I don't have any credentials to utilize the affected systems. 

I also contacted the vendor who's (once again) informed that the problem is still around. So that might also yield some extra results. 

[J.R. van Doornik]

One of the users is in, and just completed a test of he flag-setting. 

 

Setting the 'Microsoft Root Store' flag in Microsoft Edge from 'default' to 'Enabled' (which I surmise is the default state), resulted in an SSL31 notification.

 

Changing that setting to 'Disabled' then resulted in the user logging in normally and starting an app successfully using Microsoft Edge.

Since once is still actually none, I let the user logoff, dlose Edge, and then restart Edge and retry the login and application. 

She started a different application (she has multiple on that environment), which also started with no issues... 

 

So that setting seems to do something, tho I'm unsure what exactly. 

 

Also, since the 4 other test users are not in today, I can't verify this outside of the 1 user that I already tested. Asked the remainder to also apply this fix, and let me know, so hopefully that'll get back positive. 

 

I do have an appointment for a meeting with some technical engineers of the vendor on the 8th of June, so I hope that will result in something more substantial. But the disabling of the root-certificate setting seems to be a workaround for the issue so far. 

[J.R. van Doornik]

Progress, and a solution was found...

 

When diagnosing with the vendor, ofcourse the problem didn't rear it's ugly head. Despite that we dug into the backend and it was found that while the ROOT certificate for their website was installed, the INTERMEDIATE certificate was not. 

 

Once that was determined, I gathered both the root and intermediate, added those to a GPO to release on the Citrix environment and since then there have been NO problems anymore with the use of the vendor site. 

 

All I can guess at, is that the certificate was sometimes able to verify it's validity with the root CA certificate on the client, and much more often failed to verify that validity as the client went looking for the intermediate certificate which was not present. That's about all I can guess at for a possible root-cause of it sometimes working and most of the times failing. 

 

Anyway, I figured to provide the solution for any unfortunate souls that run into this problem somewhere in the future. ? Thanks to all that contributed with comments and suggestions. It's been highly appreciated ?