Launching desktops results in ghost "active" sessions - only fix is rebooting VM

[Mick Swanson]

Here's a fun one!

 

-    Environment consists of:  
o    A little over 60 Windows Server 2019 and 2022 VDAs, all running LTSR 2203 CU1. These VDAs are VMs running on AWS EC2, with NVIDIA Tesla T4 GPUs attached.
o    Delivered via Citrix Cloud and an ADC VPX (13.1 build 37.38) running Citrix Gateway.
o    A single user is assigned to each VM. Roaming profiles are not in use, but UPM is installed.

-    What the issue looks like:
o    Between 2-5 times a day, a random user will attempt to log on via CWA, and get an error message “Resource unavailable”.
o    Upon investigation, Citrix Studio (DaaS console in Cloud) shows that the VDA is On and Registered, with an Active user session.
o    When RDPing to the affected VM, the user has an active Windows session and the VM appears to be responsive and fully operational. All Citrix services are running. If the Windows session is logged out, it still appears that the user has an Active session in Citrix Studio. Restarting the Citrix Desktop Service will reset the session length counter, but the “Ghost session” still shows as Active in Citrix Studio.
o    When looking on the Citrix Gateway console on the ADC, no connection appears to be made between the VDA and the Endpoint.
o    The only way we’ve found to fix the issue is to restart the VM. We can reproduce this issue yet, but the issue consistently happens daily across this userbase.

-    Other items of note:
o    We have several hundred other users deployed under other customer accounts in Citrix Cloud with near identical setups that do not experience this issue with regularity. Many are running the same OS and same VDA version. However, they are not running ADCs but have all been migrated to the Citrix Gateway Service over the last year. These other VDAs are also not running vGPUs.
o    Users have also been reporting an unresponsive “black screen” when logging in occasionally. Unsure if this black screen is a precursor to the “ghost session” or if it’s an unrelated issue.

-    What we’ve tried already:
o    VDA Upgrades/downgrades – This could have potentially started when we upgraded to VDA 2209 from VDA 2006. Since then, we’ve rolled back to 2203.1 hoping for more stability.
o    ADC Update to the current release. We were only behind by a build or two. We did this about two weeks ago to make it current. The issue appeared to be gone briefly, but it was just a couple of lucky days, as it’s as bad as ever still.
o    Scouring Event Viewer. Nothing even remotely interesting comes up when this occurs – We’ve looked over dozens of cases of this so far and it doesn’t leave anything of note.

[Ashok reddy]

On 2/8/2023 at 5:43 AM, Mick Swanson said:

If the Windows session is logged out, it still appears that the user has an Active session in Citrix Studio. Restarting the Citrix Desktop Service will reset the session length counter, but the “Ghost session” still shows as Active in Citrix Studio.

Interesting one, what was the situation at user end when try to launch session during this quoted scenario ?? 

[Mick Swanson]

On 2/8/2023 at 11:24 PM, Ashok reddy said:

Interesting one, what was the situation at user end when try to launch session during this quoted scenario ?? 

They get a generic "Resource Unavailable" error in CWA until I shut down their virtual machine.

 

Example:

The resource is unavailable currently. Try again later.
Transaction Id: ddb9efbc-cc5b-4f3c-ac36-23056e7f634f

 

I will say, after doing a VDA Cleanup Utility and LTSR 2203 CU2 install on the VDAs, the frequency of this issue has gone down drastically, but we're still running into one every day or two.