New sessions to CVAD 2109/2112 keep hanging forever at "Please wait for FSlogix profile..." but never make it to the Windows GUI

[Andy Vanderbeken]

Dears, in a fresh QA environment with everything on 2112 including Storefront server and a copy of our existing proven productional 1912CU3 VDA image but here upgraded to 2112 (I tried 2109 as well) I keep seeing the random intermittent problem that some fresh user logons never make it to the windows GUI. Instead the end user keeps seeing "Please wait for FSlogix profile..." but the FSlogix logs clearly show that the profile load completed succesfully and fast.

 

All factors are equal to our perfect running 1912CU3 production environment except the VDA is upgraded to 2112. I also see the issue when upgrade a copy of our productional VDA to 2109. The bigger context here is that we're looking to migrate away from LTSR to CR. In this light 2006 is not an option because it has known issues which were first fixed in 2109 but now 2109&2112 are both showing this problem. When I look around the most relevant possible cause I find is CVADHELP-19182 (https://support.citrix.com/article/CTX338807) for which I currently have a case open.

 

Any advice or similar experiences from the field from fellow Citrix Administrators or particular advice/fixes ? Thx

[Erik Zwanenberg]

Quote

 

Hi Andy!

 

We had exactly the same  while building/testing a new Citrix Cloud environment on CR  (2109 and 2112) ran into this issue.....Broke my head on it and tried al versions FSLogix, Windows patches, Antivirus exclusions, Ctxhook changes directly in the registry of the MCS Master Image and not in GPO but still nothing. Then reverted back to VDA2106 and all problems were gone! Now running 2106 for about 3weeks. I've also logged a call 2 weeks ago but Citrix was still investigating the issue. The only thing I notice is some more BCR crashes in this version but thats better than frustrated end-users who can't logon and an angry customer.......Hope Citrix fixes this soon!

 

Also see: https://support.citrix.com/article/CTX338807 and https://discussions.citrix.com/topic/393608-logonuiexe-sessions-disconnected/page/9/ 

[Andy Vanderbeken]

To Anyone reading this in the future: After opening a case and requesting the specific internal fix mentioned in my OP above I received a new .dll to manually replace a faulty one in the master image build. After daily logon and testing this new build in QA with 2 admins for 4 successive workdays we have not seen the bug at all again so it looks like all problems are completely gone.

 

We'll have absolute certainty around same time next week after it's been running even longer without problems but if I remember correctly the problem would already easily reoccur within 2-3 workdays so yeah looking solid.

[Ahmad Irffan1709156583]

What was the name of dll which was replaced.

[Andy Vanderbeken]

15 hours ago, Ahmad Irffan1709156583 said:

What was the name of dll which was replaced.

 

Sorry but concerning details on which exact .dll it is as well as getting the most up-to-date replacement version of it I personally cannot help you for obvious reasons. Please simply open a case with Citrix and after the usual initial validation that your personal situation applies to this fix you'll promptly receive it directly from Citrix support. 

[Lukas Grow]

Hello there, we also seem to have this issue, after installing a VDA release higher than 2103. When the issue occurs, I can see in Process Monitor an endless loop of events for the affected session. Winlogon.exe tries to read a certain registry key. Not sure what this means. Another suspicion of mine is, that the issue only appears after installing all the latest Windows Updates.

UPDATE: we received the private fix and we will start to test soon. Replaced dll is C:\Program Files\Citrix\HDX\bin\StatUi.dll

 

[Andy Vanderbeken]

Lukas,

 

I'm not sure if your issue is the same one as mine. On the internet, people that have my issue typically report it to happen only as of 2109 and thus in 2112 but not in 2006 for instance.

 

 

[Andy Vanderbeken]

Everyone,

 

After testing more than a week in QA now we remain problem free so the fix seems to really resolve it. What baffles me though is that I personally first detected this problem when testing a build in 2109 in QA right as it was released back in september and then again in 2112 and only on 27 Jan 2022 did any official news from Citrix appear on this:

 

https://support.citrix.com/article/CTX338807

 

Meanwhile last week the article was modified and now I see it also mentions that the problem also happens in CVAD 1912 LTSR CU4. So all 3 of these fully productional (available for download from Citrix) builds of the CVAD software clearly contain this MAJOR problem yet none was recalled nor a hotfix released nor was a superseding build re-released to address it causing us all to guess for basically 4 months in a row why we have random disconnections.

 

In hindsight I now realize that on 1 particular day where we rolled out  CVAD 1912 LTSR CU4 and had massive problems that day it was due to this very problem as well. I remember that that day around 10-20% of our people logging on to that build could not connect and had permanently hanging ghost sessions because of it which then kept their FSLogix profile locked until the next nightly reboot cycle. I couldn't even kill the processes manually on the VDA servers. I'm truly amazed (in a bad way) that something this huge wasn't officially addressed or even mentioned in over 4 months ?? That makes me wonder about the QA testing internally at Citrix and honestly feel very insecure to install future version upgrades. Thank god we usually implement rigorous QA testing cycles internally in order to detect problems like this.

 

ps: Citrix Support let me know that the fix will be present in 1912 LTSR CU5 rollup pack and most likely in the next CR version (2203?) as well I guess personally.

[Lukas Grow]

On 2/14/2022 at 5:05 PM, Andy Vanderbeken said:

Lukas,

 

I'm not sure if your issue is the same one as mine. On the internet, people that have my issue typically report it to happen only as of 2109 and thus in 2112 but not in 2006 for instance.

 

 

I wrote "higher than 2103" meaning  2106, 2109, 2112. I tested all three of them and had this issue (and some really bad six past weeks) The VDA updates are necessary, as HDX engine is broken in my opinion 2103. Teams has become really unreliable in this VDA version, especially when using Igel thin clients.

 

I hope this private fix addresses the problem, as I'm desperate enough to roll this one out in production. My colleagues are replacing Igel devices by laptops already, so this is kind of urgent. Thank you for detailed post!

[Andy Vanderbeken]

Lukas,

 

I realize now that I wrote a typo: I intended to write 2106, not 2006. My bad.  I indeed interpreted higher than 2103 as you intended but on the internet, people that have my issue typically report it to happen only as of 2109 and thus in 2112 but not in 2106 for instance. In addition I see that the official Citrix documentation mentions it applies to CVAD 2109 & 2112 & 1912LTSR CU4 (but no mention of any other versions) in the original support link in my OP.

 

https://support.citrix.com/article/CTX338807

[Andy Vanderbeken]

everyone,

 

I don't know if Citrix support is reading along here after my post from yesterday but apparently yesterday a public article and hotfix appeared so no more need to open a case:

 

https://support.citrix.com/article/CTX340175

 

What a coincidence... ? 

 

 

PS: dear Citrix support, in case you are reading along here please strongly consider incorporating links to the public hotfixes always along with the official release builds on their Citrix download page so that anyone who intends to download the official CVAD 2112 product software also is made aware of any major issues such as these and sees the link to the public hotfixes immediately there as well. Kind of like it use to be with Xenapp 5-6. That would prevent Citrix admins all over the world from running into all these problems in the first place and having to go through all this.

[Lukas Grow]

I know, I was surprised to see it in 2106 as well, it is definitely affected too. That is why I think the issue might be related to current Windows updates and so the issue appears  in older VDAs as well. (retroactively you could say).

 

The now public hotfix statui.dll has a different filesize than the statui.dll I received by Citrix support, just so you know. Thank you for your quick responses!

[Andy Vanderbeken]

Lukas,

 

I see. So it's possible 2106 is affected as well but not yet (officially) discovered by Citrix at this point. Good to know. I'm going to proceed with gradual release of this hotfixed CVAD 2112 version in production now as it seems stable enough in QA so far. 

 

I downloaded the public version of the statui.dll file just now and compared its filesize to the private version I received via my support case on Feb 3. They appear to be identical so possibly the file was further modified/replaced after your case but no longer after mine.

[Christian Katterl]

Hello,

I am having this issue even with VDA 2203-LTSR and 2203-LTSR-CU1 or 2206. Trying 2209 now.

FSLogix is in the latest version - even trying the recently released public preview now.

Strange is, that even with a local user profile (local admin account), this issue occurs.

And ideas?

 

BR Christian

[Stuart Thomson1709163638]

Hi Christian, 

 

Did you manage to test 2209? 

 

For FSLogix is everyone using normal-direct profile types? From my testing the issue never appeared when using RW with failback to RO

[Phil Reeves1709156734]

I had the same issue with version 2203 and 2203.1, using multiple versions of FSLogix.  I've been able to resolve this, or at least, put in an effective workaround.

 

We restart our virtual app & desktop servers every Monday night and I found the problem occurred mostly after the restart of the servers, and potentially the first logged on user.  So I implemented the "Automatic Logon" process recommended by James Rankin here:   https://james-rankin.com/articles/how-to-get-the-fastest-possible-citrix-logon-times/

 

This has sped up average interactive session times from 41.77s to 28.22s and the problem has not happened since.

 

The process uses a service account to automatically logon at first startup, deletes the autologon password from the registry and logs off again.  This allows for faster subsequent logon times.

[Phil Reeves1709156734]

If you are still having issues, you can also try:  https://borncity.com/win/2022/09/13/windows-server-2016-fix-for-rdp-issues-in-kb5015808-and-later/