Jump to content
Welcome to our new Citrix community!
  • 1

Multi session server hangs


Sove

Question

Hi,

I have a customer with CVAD 2203upd1 on Win 2019. 

It' the third time that I cannot connect to published apps because the server hangs.

On event viewer there is error:

Faulting application name: svchost.exe_TermService

Faulting module name: ucrtbase.dll

 

After that user cannot login to the server, not even from console with administrator credentials.

 

The only way to resolve is reboot/reset the server. 

 

Do you have any suggestions?

Link to comment

22 answers to this question

Recommended Posts

  • 1

Became aware just now that 2203 LTSR CU2 released last week.
Looking in the Fixed issues section I notice the following being mentioned.
 

https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/2203-ltsr/whats-new/cumulative-update-2/fixed-issues.html#vda-for-multi-session-os

Quote

Terminal Services running on a multi-session OS VDA might unregister and exit intermittently. [CVADHELP-20671]

 

This description resembles what we are seeing, a multi-session OS VDA becomes unregistered for some unknown reason which then entails in the event viewer log which several people here have shared (Faulting application name: svchost.exe_TermService, version: 10.0.17763.3346).

 

We will test if CU2 resolves the issues we are experiencing, I suggest everyone here perform the same and share the outcome.
 

  • Like 1
Link to comment
  • 1

Apologies for long to respond, but the trust issue was/is addressed in KB5019966.

https://support.microsoft.com/en-us/topic/november-8-2022-kb5019966-os-build-17763-3650-b09dad62-5cd7-47cd-992f-b7d01f2956c1
It addresses a timing condition in Remote Desktop. It causes a device to stop working during the licensing process.

 

With that said though, the fix has to be enabled with a GPO that was given via Microsoft if you have the issue. The non GPO fix will roll out in March.

  • Like 1
Link to comment
  • 1
On 1/25/2023 at 8:26 AM, Thomas SIBILLE1709163695 said:

 

Unfortunately no, the problem persists with 2203 CU2

2023-01-24 17_13_50.png


Same here as well, 2203 CU2 VDA on Server 2019 and the same event viewer logs can be found after a server hangs.

As for our case, we instead got caught up with deploying a new Citrix desktop resource based on Windows 11 Multi-session as a replacement, possible due to having Azure as the resource layer already. This new desktop is not at full capacity yet, however we have noticed that no server hangs have been occurring.

Looking into the "C:\Windows\System32\ucrtbase.dll" file for the Windows 11 OS, it is in fact a newer version compared to the Server 2019 OS.

Windows Server 2019

image.thumb.png.2d774ba3c44907abedc2b0ba2588134a.png

Windows 11

image.thumb.png.474ea2273100666aebd84ebad54ed8c4.png

I realize it is not really helpful advice in recommending a new desktop based on another OS.

Just read this as a simple observation related to the issue in question.

  • Like 1
Link to comment
  • 0

we have been experiencing the same issues for the past 2 months

please note we run these session hosts in Azure, they are server 2019 with VDA 2203 LTSR CU1

issues started prior to our upgrade to CU1

 

Faulting application name: svchost.exe_TermService, version: 10.0.17763.3346, time stamp: 0xb6a0daab

Faulting module name: ucrtbase.dll, version: 10.0.17763.1490, time stamp: 0x48ac8393

Exception code: 0xc0000409

 

currently have tickets open with Citrix and Microsoft, no resolution yet

 

 

 

Link to comment
  • 0

Are you running a 1 way trust in your domain by any chance? We saw a similar issue with going to Server 2019 with a 1 way trust. Microsoft found that a race condition was happening with terminal server licensing and would cause the session to crash. A patch is supposed to coming this month to fix if you are running the same configuration.

Link to comment
  • 0

We are dealing with the exact same issue for one of our customers as well with the identical event viewer log mentioning svchost.exe_TermService crashing.

 

Faulting application name: svchost.exe_TermService, version: 10.0.17763.3346, time stamp: 0xb6a0daab
Faulting module name: ucrtbase.dll, version: 10.0.17763.1490, time stamp: 0x48ac8393
Exception code: 0xc0000409

 

The customer's environment details are closest to Remco Bosveld's details, only that we are using Citrix MCS to provision target machines and that no domain trusts are involved.

 

I have an ongoing case with Citrix currently but have yet to retrieve a reply from Microsoft regarding this.

 

Link to comment
  • 0

We have a client that we just upgraded the Citrix environemnt to Server 2019 and setup a new Storefront and Delivery controller. We setup brand new 2019 servers. After the cutover to the new site, we have intermittent issues with the users getting disconnected and after we are unable to login to the server with the admin credentials until the VM is restarted in VMWare. In the event logs we get the same error:

 

Faulting application name: svchost.exe_TermService, version: 10.0.17763.3346, time stamp: 0xb6a0daab
Faulting module name: ucrtbase.dll, version: 10.0.17763.1490, time stamp: 0x48ac8393
Exception code: 0xc0000409

 

Does anyone have any update on a fix? 

Link to comment
  • 0
On 12/16/2022 at 11:53 AM, Martin Becker said:

Apologies for long to respond, but the trust issue was/is addressed in KB5019966.

https://support.microsoft.com/en-us/topic/november-8-2022-kb5019966-os-build-17763-3650-b09dad62-5cd7-47cd-992f-b7d01f2956c1
It addresses a timing condition in Remote Desktop. It causes a device to stop working during the licensing process.

 

With that said though, the fix has to be enabled with a GPO that was given via Microsoft if you have the issue. The non GPO fix will roll out in March.

I didn't see a regedit mentioned on the page, would you be able to paste or link to the necessary regedits? Thanks!

Link to comment
  • 0

We were having the same issue in a pretty similar environment Windows Server 2019, VDA 2203 CU2 LTSR, MCS, all VDAs are VMs hosted in Azure.

 

We noticed a lot of rpm errors, event IDs 14 and 16 about some of our applications not being able to open virtual channels along with the terminal server error.

 

image.thumb.png.0a71db1d9167d99eef2f862558efb25d.png

 

The issue was related to this article https://www.citrix.com/blogs/2021/10/14/virtual-channel-allow-list-now-enabled-by-default/ about the virtual channel allow list enabled by default from VDA 2109 and forward.

 

We manually disabled the setting via Citrix Cloud policies hoping this will help with the issue and so far after all our session hosts have been rebooted we are not seeing the terminal service error anymore and no more unregistered machines.

 

image.thumb.png.c8770d360ebf7e48c7c4664ffa2a574f.png

 

Hope this helps someone else. We also have tickets opened with Citrix and Microsoft but so far they haven't found anything after we submitted a dump file of the terminal server process.

 

We are still monitoring the environment but so far we are good after a few days and we usually had at least one server going cuckoo every day.

Link to comment
  • 0
On 4/17/2023 at 6:51 PM, Phill Coombs said:

Victor, are you saying that even though the default is "disabled", you set it forcibly in policy to "disabled", and that seemed to have solved the issue ?

Correct Phill, seems that Citrix set the default to enabled at the VDA level bypassing the default value in the policy. You can forcibly set it to disabled or set it to enable and add the necessary virtual channels for your case. Since we did this change we haven't seen the issue happen anymore.

Link to comment
  • 0
On 4/20/2023 at 11:09 PM, Victor Miranda1709162642 said:

Correct Phill, seems that Citrix set the default to enabled at the VDA level bypassing the default value in the policy. You can forcibly set it to disabled or set it to enable and add the necessary virtual channels for your case. Since we did this change we haven't seen the issue happen anymore.

Reporting back: two weeks have now passed since I disabled the policy on the site, and no further issues have been encountered, so hopefully "solved"  

Link to comment
  • 0
On 4/10/2023 at 2:09 PM, Victor Miranda1709162642 said:

We were having the same issue in a pretty similar environment Windows Server 2019, VDA 2203 CU2 LTSR, MCS, all VDAs are VMs hosted in Azure.

 

We noticed a lot of rpm errors, event IDs 14 and 16 about some of our applications not being able to open virtual channels along with the terminal server error.

 

image.thumb.png.0a71db1d9167d99eef2f862558efb25d.png

 

The issue was related to this article https://www.citrix.com/blogs/2021/10/14/virtual-channel-allow-list-now-enabled-by-default/ about the virtual channel allow list enabled by default from VDA 2109 and forward.

 

We manually disabled the setting via Citrix Cloud policies hoping this will help with the issue and so far after all our session hosts have been rebooted we are not seeing the terminal service error anymore and no more unregistered machines.

 

image.thumb.png.c8770d360ebf7e48c7c4664ffa2a574f.png

 

Hope this helps someone else. We also have tickets opened with Citrix and Microsoft but so far they haven't found anything after we submitted a dump file of the terminal server process.

 

We are still monitoring the environment but so far we are good after a few days and we usually had at least one server going cuckoo every day.

Disabling this policy had resolved the issue for me. Thank you for the information provided.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...