citrix sessions 2203cu3 disconnecting randomly every day with an event id 12

[vinod mundluri1709158178]

i am having an issue with citrix sessions disconnecting randomly every day with an event id 12 ( source RPM)

We are getting quite a few VDA with the following warnings

Connection broken unexpectedly for user XXXXX
Source RPM
ID 12

Session reliability suspended the connection for user
Source RPM
Event ID 10

any idea why?my citrix infrastructure is..delivery controller and storefront on server 2022 with 2203 cu3...have mix of 2019 and 2022 VDAs ...2022 vds are running on 2203 cu3 and 2019 vda running on2206....

 

been working with citrix support from 1st aug 2023 under the case #82146078...collected CDF logs, end user PC logs and ADc logs asking for more logs 

 

[Pascal Heinz1709163266]

On 8/25/2023 at 11:54 PM, Jeff Riechers1709152667 said:

Did you attempt to disable adaptive transport?  The newer VDA utilizes UDP communication as well as TCP, disabling that might help.

I had the same thing with my Server 2016 1912 CU7 VDAs and I "fixed" the problem by disabling the adaptive transport. 
Since this change the traffic goes over TCP only and the customers didn't have any connection timeouts in more than a month.
Their problem was only reproducible at their site. When I've tried to get a diconnect on my test-user, in my office I never had one.
Also the people working from remote never had disconnects, so I think there was a issue with their on site network and UDP traffic for the sessions.

[Jeff Riechers]

Did you attempt to disable adaptive transport?  The newer VDA utilizes UDP communication as well as TCP, disabling that might help.

[Kamel Sennoun]

Hello,

Did you get any update on your issue  and did you find a solution ?

 

We met the same behavior with unexpected broken session, but we are unable to reproduce from the holding site.

 

Actually , What we know is that it occurs only on slow network connections .

 

Usually it appears with some networks that is getting latency maybe 50ms and more in terms of latency.

 

For instance , in some case we have a lot of network hop between client and server  (more than 30 hops)..

 

We tried a lot of thing to workaroud as for example activating keep alivve instead of SR. but the issue is stilll the same .

 

HDXoverUDP has been disable by policy and we also tried to upgrade with  2203 cU4 agent but this didn't help.

 

We open a case but it's  actually in freeze state. 

 

Actually we nned to troubleshoot with some of the local IT teams.

 

they need to be be involoved and need to contribute to help.

 

Any feedback should be appreciated on your side.

 

however, we know thas this kind of event only involve with cvad starting agent 1912.

 

Before this version it didn't occurs, no event was raise on the system event (like event rpm 10-12),.

 

can you please advise on your side and  how it goes ?

 

 

[Kamel Sennoun]

On our side we are not using the IGEL thinclients so i think this is not related to this kind of devices.

[Fabian Ibanez]

Hi there,
we are facing the same issue. Session randomly close unexpectedly (event id 10)

 

Current setup is win2019, 2203 LTSR CU2, random workspace clients (win, linux), FSLogix Profiles, HDX preferred

 

We had the same behavior with 1912 LTSR CU5, but not that often
The workaround mentioned by Matteo usually also works for us.
Sometimes we even had two session of the same user on one worker server...  Although FSLogix should prevent a second session.
Does anyone know why session reliability/reconnect is not able to reconnect to an existing session in this scenario?

[Marc Gemp1709157697]

Hi @vinod mundluri1709158178

 

we have the Same Problem. We have Citrix Infrastruktur on 2203. CU2 but the VDA on 2203 CU3. The Terminalserver are running with OS2016 and OS2022. Our customers report that it currently only happens when they are inactive and not at work (time during 8-15 min or even more). This can still be a coincidence, but it has only happened to us now.

 

We assume that we did not have the error with CU2 and that the update of the VDA must be to blame.

Which anti-virus scanner do you use and do you use UDP (EDT) to the customers?

 

what kind of clients do the customers have? We have Dell Wyse ThinOS where it is currently most noticeable 

Edited August 17, 2023 by Marc Gemp
complements

[Jeff Riechers]

So that looks like something on the network is collapsing the network tunnels.  

 

Can you run some wireshark on the network so you can see if there is any resets or anything like that taking place?

[vinod mundluri1709158178]

Hi @Marc Gemp

 

Antivirus - we are using Windows defender and added all the exclusions to the defender.

Users using windows laptops/desktops with citrix workspace app 

[vinod mundluri1709158178]

Hi @Jeff Riechers

 

We have other citrix farms using same firewall and same ADC and only disconnection happens for the user when they connecting a new citrix farm running with windows 2022 with 2203 CU3..no issues when same user connecting other citrix farms that are running on server 2012r2 with 7.15 LTSR ...collected wireshark logs along with CDF logs and uploaded to citrix support 

[Sven Bender]

 

I could see the same problem in our farm.

Windows Server 2022, VDA 2203.CU3, Igel Thin Clients (11.08.230).

 

Event 12 is caused at different times and only affects individual users on a server.

Adaptive transport is turned off by Citrix Policy.

 

Are there any new insights into the citrix ticket?

Edited August 29, 2023 by Sven Bender

[Manfred Blum]

Same problem here.
Server 2019 with VDA 2203.CU3, various IGEL Clients (older ones).

 

[Thomas Gabel1709162843]

We have exactly the same. 

Citrix VDA 2203 CU3 

Server 2019 OS

 

The change to the Failback TCP is the only one solution?

[Carlo Scarpa]

Same issue happening with us using Server 2016 Standard and a VDA of 2212 and or LTSR 2203.

I went into the policy and disabled the HDX Transport and am awaiting to see if this truly does fix the issues.

[Stewart Ashens1709163249]

Same issue here.

 

Citrix VDA 2203 LTSR CU3 

Server 2022 OS

HDX Adaptive Transport is set to Preferred. 

 

Following to see if disabling the adaptive transport resolves the issue. 

[Matteo Erb1709163349]

We have a similar issue where the sessions randomly disconnects. The users cant reconnect to their sessions.
Only thing that helps in this situation is, to manually log out the user session and to set the maintenance mode for the terminalserver, because even when they try to connect to a new session, they are connected on the same server like before. And the session cant be used. The director shows the session as connected, the user device says "connection to... broken unexpectedly".
Those random dosconnects do not happen every day. Sometimes not even one in a week. 

We currently use VDA 2203 CU3.
I think it has something to do with the IGEL thinclients. 

[Kamel Sennoun]

you can't reconnect because you have a broken session that is not reliable . The only thing to do in such scenario is to hide the session until next server reboot with the cmdlet (get-brokersession -clientname "xxxxx" | set-brokersession -hidden $true ). This way user will be able to connect even on the same server. hope it helps !

[Matteo Erb1709163349]

On 2/16/2024 at 6:48 PM, Kamel Sennoun said:

you can't reconnect because you have a broken session that is not reliable . The only thing to do in such scenario is to hide the session until next server reboot with the cmdlet (get-brokersession -clientname "xxxxx" | set-brokersession -hidden $true ). This way user will be able to connect even on the same server. hope it helps !

its a workaround but does not solve the problem