2203 LTSR issues

[Franc van de Westelaken]

After upgrading from 2106 to 2203 LTSR we have two major issues:

 

- disconnected session sometimes aren't being reconnected when a user logs on again. Instead a new session is created. But, since we are using FSLogix, this seconds session fails due to the fact that the VHDX containers are still being used in the other disconnected session. This issue is not related to the issue below, since the disconnected that don't reconnect are shown in Director.

- at random, some sessions, especially disconnected sessions, don't show up in director. They are only visible when logging on to the VDA directly and opening taskmanager -> users. Clients are Windows 10 clients using Citrix Workspace app 2112 or newer.

 

Anyone else experiencing this also?

[Andy Vanderbeken]

2 minutes ago, Martin Garber said:

Hi,

 

try to set this reg-keys

 

Local machine - software - Citrix created a new folder called "reconnect"

added dword key "DisableGPCalculation" with 1

added dword key "FastReconnect" with a 0

 

This fixed our problem

 

on the VDA server ?

[Martin Garber]

yes, on every VDA server

[Andy Vanderbeken]

4 hours ago, Martin Garber said:

Hi,

 

try to set this reg-keys

 

Local machine - software - Citrix created a new folder called "reconnect"

added dword key "DisableGPCalculation" with 1

added dword key "FastReconnect" with a 0

 

This fixed our problem

 

Holy cows it works ! Just tested it successfully on a test & QA delivery group with 1 VDA server. How did you come by this solution ?

 

Note: the "FastReconnect" Dword with value 0 was already present in my case so the "DisableGPCalculation" with value 1 (= disabling policy calculation during reconnection) made the difference for me.

[Darryl Sakach]

Other than resolving this issue, what is the impact of these settings?

[Andy Vanderbeken]

1 hour ago, Darryl Sakach said:

Other than resolving this issue, what is the impact of these settings?

 

as far as I can see "DisableGPCalculation" disables the checking of the WMI filters for gpo calculation upon reconnection and the "FastReconnect" feature, introduced with Windows Server 2019, is a performance optimization to speed up reconnecting to a disconnected remote session. Useless and multiple times broken now since release.

[Estefano Ramos]

Hello, here I have the same problem ...
after upgrading the VDA to version 2203 on the multisession servers, they began to lose registration randomly. My OS is WinServer 2019...
After rebooting the machine registers again; Did not have this problem with winserver 2019 with vda 1912CU3000. Does anyone have a solution? We've opened case on citrix, but we're still waiting for response.

[Stefan Wielers]

Same on our site with VDA2203/W2K19 and VDA2203/W2K22 (all delivery controller 2203). We've setup a task to restart Broker every hour. 

[Andy Vanderbeken]

Update: My reconnection problem is now completely solved and our Mac users are running problem free once more. I solved it by restoring all previous PVS image builds from backup tape and testing them one by one until I found the exact one where the problem starts occurring. Then I dug up my exact change notes for that build and edited a copy of that build where I tried to reverse every change one by one until I found the exact change that triggers this problem. Eventually I found the culprit:

 

“Microsoft Debug Diagnostic tool v2 Update 3.2” software installation.

 

As soon as I ran a repair of this specific software all problems were solved. No more unnamed Citrix sessions in Studio and consequently no more reconnection problems. So in conclusion the root cause for this problem apparently was not in the Citrix VDA agent functionality after all !

 

Then I went back to test more and found even though this "reconnection" bug is now 100% solved the other "connection interrupted" bug is still present. This is the bug which wrongfully kicks Mac users out of their active Citrix sessions with autoreconnect (session reliability) apparently not working after this happens with a false-positive pop-up that reads "connection interrupted" while there never was an interruption to be seen at the network level.

 

Through testing different builds I furthermore noticed that the "connection interrupted" bug was introduced in VDA agent 2112 and still is present in 2203 (copied over most likely) while if I uninstall the 2112/2203 VDA agent and clean install 1912 LTSR CU5 VDA agent instead the problem is not happening so that's my current workaround.

 

I hope this may help you

[Bruynzeel IT Support]

We have the same issues after updating to 2203 LTSR on our Server 2019 environment. also raised a ticket with citrix, no response yet.

Disconnected sessios are unable to connect back to in some cases, users get (ghost) sessions on the same VDA (one "active" and one "disconnected). 
What if found as workaround is that whenever a user has this problem, or gets a wrong session is that restarting the Citrix Desktop Service on the VDA helps to bring the disconnected session back alive to citrix and therefore to the user.

 

[Stacey Lewis1709159460]

4 minutes ago, Bruynzeel IT Support said:

We have the same issues after updating to 2203 LTSR on our Server 2019 environment. also raised a ticket with citrix, no response yet.

Disconnected sessios are unable to connect back to in some cases, users get (ghost) sessions on the same VDA (one "active" and one "disconnected). 
What if found as workaround is that whenever a user has this problem, or gets a wrong session is that restarting the Citrix Desktop Service on the VDA helps to bring the disconnected session back alive to citrix and therefore to the user.

 

Are you seeing issues where the Disconnected sessions are hung in a Logging Off state in Director?  I'm seeing this with 2203 LTSR on Server 2019, I've put various system processes in the LogoffCheckSysModules but its not working consistently.

[Andy Vanderbeken]

11 minutes ago, Bruynzeel IT Support said:

We have the same issues after updating to 2203 LTSR on our Server 2019 environment. also raised a ticket with citrix, no response yet.

Disconnected sessios are unable to connect back to in some cases, users get (ghost) sessions on the same VDA (one "active" and one "disconnected). 
What if found as workaround is that whenever a user has this problem, or gets a wrong session is that restarting the Citrix Desktop Service on the VDA helps to bring the disconnected session back alive to citrix and therefore to the user.

 

do you have "Microsoft Debug Diagnostic tool" installed on the VDA server or VDA image by any chance ? Citrix support often asks to install this in order to debug some problems.

[Bruynzeel IT Support]

9 minutes ago, Stacey Lewis1709159460 said:

Are you seeing issues where the Disconnected sessions are hung in a Logging Off state in Director?  I'm seeing this with 2203 LTSR on Server 2019, I've put various system processes in the LogoffCheckSysModules but its not working consistently.

not really. we see 2 symptoms really:
- either NO session is in Director or studio (But on the VDA itself in windows you see a disconnected session with open applications)
- Or in some cases director shows multiple sessions and Studio does not show the same session result, also sessions from many days old could be listed, but that is not possible since we reboot daily. if you see the sessions, they are either active or disconnected as state listed.
 

no changes made to the LogoffCheckSysModules key.
but it seems to happens to sessions that were disconnected by the user, left alone for some time and then they try to get back into it, but it does not happen all the time, so not really reproducable.
this has worked fine for multiple years, but broke after 2203 update. Workaround: reboot the Citrix Desktop Service on the VDA that users have problems on. not liking this, but it gives you a ways to solve it for now.
 

Still waiting on citrix to reply.

[Bruynzeel IT Support]

44 minutes ago, Andy Vanderbeken said:

do you have "Microsoft Debug Diagnostic tool" installed on the VDA server or VDA image by any chance ? Citrix support often asks to install this in order to debug some problems.

No, doublechecked.

[Andy Vanderbeken]

2 hours ago, Bruynzeel IT Support said:

No, doublechecked.

 

Just for the record it's not called like that if you look it up in control panel Add/Remove. Either way, even if it's not this particular piece of software I believe there is another piece of software inside your image that prevents the Citrix desktop service during the logon phase of any new session to register those sessions correctly with the broker. You can doublecheck this because troubled sessions that are e.g unable to reconnect will also always show a lack of username and/or broker username in the Studio management console or be invisible in Citrix Director.

 

If you see this same behaviour than I suggest you uninstall software piece by software on a test build and see which one is the cause in your case. In my particular case it was:

 

[Bruynzeel IT Support]

15 hours ago, Andy Vanderbeken said:

 

Just for the record it's not called like that if you look it up in control panel Add/Remove. Either way, even if it's not this particular piece of software I believe there is another piece of software inside your image that prevents the Citrix desktop service during the logon phase of any new session to register those sessions correctly with the broker. You can doublecheck this because troubled sessions that are e.g unable to reconnect will also always show a lack of username and/or broker username in the Studio management console or be invisible in Citrix Director.

 

If you see this same behaviour than I suggest you uninstall software piece by software on a test build and see which one is the cause in your case. In my particular case it was:

 

Thanks for the reply.
again i checked, no such tool installed. nothing with the name "debug" even. 
not sure the behaviour is 100% identical to what you describe. also this morning, citrix studio shows no 94 connections, director 103.... a lot of sessions dated 3 days old in director, and that is not possible, the VDA restarts every day.  i do restart the XXD as well from time to time, because that will give the most realistic situation session wise. i don't see sessions without a username, the biggest problem is some actual live sessions do not show at all. very unreall this. 
Asked for a update on our support ticket yesterday.

[Sascha Rodszies]

We have had similar difficulties.

 

We have solved our issue that the session status, after a reconnect, was no longer updated (changed not back from disconnected to active as it should).
Our cause was a Group Policy.

User Configuration > Administrative Templates > System > Ctrl+Alt+Del Options
Policy: Remove Lock Computer
Setting: Enabled

We have set the GPO to 'Disabled' on the VDAs and now everything works again.

 

Greetings, Sascha

[Andreas Roeslmaier]

Any news from Citrix about this issue? Looks like it is a known bug since VDA 2112?

[Andy Vanderbeken]

About the reconnection problem I can't give any info as it's completely solved for me. Haven't had any issues in the past month but others still seem to do.

 

About the disconnection "connection interrupted" problem I still have my case open. My latest troubleshooting has shown that the issue only happens in combination with connections through Netscaler, not through a vpn nor through direct connections (within the company network). As a result the Netscaler engineering team has been involved as well and I uploaded some specific traces of the problem to them.

 

I also found that the problem can be simulated and forced really fast (usually within 30 seconds) by peforming the following steps:

 

1. Start a Citrix session by logging into your company Netscaler Gateway website

 

2. Start a very large file copy from the client machine to the company network through a client redirected drive from within the Citrix desktop session

 

3. While this very large file copy is ongoing, start clicking on different spots of the empty desktop within the Citrix desktop session

 

 

The Netscaler engineering team confirms they clearly see a connection drop happening that is initiated from the server side. In other words they see the problem and acknowledge it but haven't found any further information yet that leads to the root cause or a solution.

[Björn Schläfli]

Found this:

https://support.citrix.com/article/CTX338807/users-may-fail-to-reconnect-or-start-a-new-session-on-virtual-apps-and-shared-desktop

Edited August 3, 2022 by Bjoern Schlaefli
hyperlink edited

[Björn Schläfli]

I'll create a new vdisk image with VDA 2203 and the fixed StatUi.dll from the 2112 fix and will report back in a few days, as I'm unable to reproduce the issue so far and need to wait for user responses.

[Björn Schläfli]

same issue with VDA 2203 and the fixed StatUi.dll from 2112. Will try with VDA 2112 and the fixed StatUi.dll.

Restart the BrokerAgent didn't work.

User today has worked in the office and later wanted to reconnect the session from private home computer which worked but it was not possible to launch any published apps. We closed one of the reconnected apps and have been unable to launch it again.

[Robert Blissitt]

On 5/23/2022 at 8:27 AM, Joe Robinson said:

 

Just wanted to point out, 2106 is vulnerable to an escalation of privledges attack.  You can read more about it here.

https://support.citrix.com/article/CTX319750

 

As far as I can tell ther eis no "good" version; you're either vulnerable, or stuff doesn't work.

 

Are the hotfixes provided at CTX319750 not sufficient to address the vulnerability?  If they are not sufficient, I could potentially uninstall 1912 LTSR CU3 and reinstall it without the CPM/UPM bits as we use FSLogix.  I assume we would just lose some reporting in Director by omitting CPM.

 

We have been seeing duplicate sessions and blank/black window issues with 1912 LTSR CU5 and 2203 LTSR "CU0" and CU1, so I am reverting to 1912 LTSR CU3.

[Björn Schläfli]

VDA 2112 with the fixed StatUi.dll didn't fix the issue.

 

Yesterday i found the cause in our site but i still don't have a solution for it. wfshell.exe is no longer running in the user context on the VDA server. That explains the behavior. But there is no event 1000 application crash in the event log. That means the wfshell.exe does not crash, but exits for an unknown reason.

[Andy Vanderbeken]

6 hours ago, Björn Schläfli said:

VDA 2112 with the fixed StatUi.dll didn't fix the issue.

 

Yesterday i found the cause in our site but i still don't have a solution for it. wfshell.exe is no longer running in the user context on the VDA server. That explains the behavior. But there is no event 1000 application crash in the event log. That means the wfshell.exe does not crash, but exits for an unknown reason.

 

I'm still left with one puzzle to solve in order to get 2203 LTSR fully working without problems here: After I solved the "reconnect" problem by repairing or uninstalling the "Microsoft Debug Diagnostic tool v2 Update 3.2" I can confirm all reconnect issues as well as unnamed sessions were fully solved but from that very same moment I started observing random wfshell.exe and winlogon.exe crashes which always seem to happen within 30sec after someone has disconnected from their Citrix session. They don't happen to every disconnected session and it seems a bit random but the fact.

 

 

Here are examples:

 

(winlogon crash)

 

WARNING - [Triggered by _ItemCount>0] - 1 event(s) of Severity Level: "Error", were recorded in the last 0.2 hours from the application Event Log. (List is on next line. Fields shown are - Logfile:TimeGenerated:EventId:EventCode:SeverityLevel:Type:SourceName:Message)

Application:20220812092031.257134-000:1000:1000:Error:Application Error:Faulting application name: winlogon.exe, version: 10.0.17763.2989, time stamp: 0x5b429668 Faulting module name: icagfxstack.dll, version: 7.33.0.50, time stamp: 0x622b7019 Exception code: 0xc0000005 Fault offset: 0x000000000000c2e8 Faulting process id: 0x6c94 Faulting application start time: 0x01d8ae2cc2cfee70 Faulting application path: C:\\Windows\\system32\\winlogon.exe Faulting module path: C:\\Program Files\\Citrix\\HDX\\bin\\icagfxstack.dll Report Id: b3dd4675-c839-4f1b-8e6d-9c79d790df57 Faulting package full name: Faulting package-relative application ID:

 

(wfshell.exe crash)

 

WARNING - [Triggered by _ItemCount>0] - 1 event(s) of Severity Level: "Error", were recorded in the last 0.2 hours from the application Event Log. (List is on next line. Fields shown are - Logfile:TimeGenerated:EventId:EventCode:SeverityLevel:Type:SourceName:Message)

Application:20220812103000.385337-000:1000:1000:Error:Application Error:Faulting application name: wfshell.exe, version: 7.33.0.51, time stamp: 0x622b72b7 Faulting module name: ntdll.dll, version: 10.0.17763.2989, time stamp: 0x4066250b Exception code: 0xc0000374 Fault offset: 0x000e0773 Faulting process id: 0x8b6c Faulting application start time: 0x01d8ae1b48514d69 Faulting application path: C:\\Program Files (x86)\\Citrix\\HDX\\bin\\wfshell.exe Faulting module path: C:\\Windows\\SYSTEM32\ntdll.dll Report Id: 0c2cacaf-2ef5-4f08-a414-a1829fa7d896 Faulting package full name: Faulting package-relative application ID:

 

 

I've gone back and forth between build updates and there seems to be no way I can get rid of the reconnect issue without at the same introducing these so I'm kind of stuck having to choose between them ?

 

Any advice is appreciated.

 

[Mike Kelly1709153237]

On 8/11/2022 at 5:02 AM, Andy Vanderbeken said:

BREAKING NEWS everyone on the specific issue of MAC (and possible other non-windows) CWA clients getting randomly disconnected unexpectedly with the "connection interrupted" pop-up message: After months of troubleshooting with both the Netscaler engineering team and VDA engineering team I just finished testing a new special build with a private fix in it that seems to solve the problem completely. After all this troubleshooting I have have identified that:

 

-The problem only happens when the connection is happening through a Netscaler Gateway. So you won't see the problem when connected through a third party VPN or internal network.

-The CWA client must be a MAC (or potentially other non-windows). The issue will not happen with a Windows CWA

-This particular problem was introduced as early as of VDA agent 2112 and is present in all later editions.

-The problem only happens when inside the Citrix session from the MAC a client-redirected device or drive is being accessed or used in any way. The easiest way to force and bring out the problem is therefore to start a continuous copy of a very large file inside the Citrix session on the from the MAC client machine towards the VDA or remote network inside the VDA. Then start clicking with your mouse on the empty desktop until the problem happens. Using this way I can consistently bring out the problem within 5-30 seconds.

 

Suspected Root cause: It's still under investigation but if this is correct it seems to be a problem introduced by a change in the winstation driver wdica.sys

Solution: obtain a new/old version of the driver through a private fix (you can refer to my Citrix CASE 81283528) and replace the current faulty version with it. Alternatively wait for a future public fix version to appear.

 

I hope this makes your day or at least helps you ! Feel free to use the upvote button

Any update on this?   Is this issue resolved for you?  Do you know if it's going to be included in the next CU for 2203?