Citrix Virtual Apps 2103 VDA 2103 wfshell.exe crashes

[Björn Schläfli]

Some users complain about wfshell.exe crashes since we've updated our site and vda multi-session os from 1912 to 2103. 

Virtual Apps 2103

VDA 2103

 

error 1000 in application log:

Name der fehlerhaften Anwendung: wfshell.exe, Version: 7.29.0.8, Zeitstempel: 0x603d593b
Name des fehlerhaften Moduls: VCRUNTIME140.dll, Version: 14.28.29910.0, Zeitstempel: 0x60247565
Ausnahmecode: 0xc0000005
Fehleroffset: 0x000035b0
ID des fehlerhaften Prozesses: 0xa97c
Startzeit der fehlerhaften Anwendung: 0x01d7aefb2256d55e
Pfad der fehlerhaften Anwendung: C:\Program Files (x86)\Citrix\HDX\bin\wfshell.exe
Pfad des fehlerhaften Moduls: C:\Windows\SYSTEM32\VCRUNTIME140.dll
Berichtskennung: ada5651d-41d3-4767-a0fb-3f664cb072c5
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 

 

Name der fehlerhaften Anwendung: wfshell.exe, Version: 7.29.0.8, Zeitstempel: 0x603d593b
Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.14393.4350, Zeitstempel: 0x606eb0f6
Ausnahmecode: 0xc0000374
Fehleroffset: 0x000d8ef1
ID des fehlerhaften Prozesses: 0x3f80
Startzeit der fehlerhaften Anwendung: 0x01d7af7abeaae5e8
Pfad der fehlerhaften Anwendung: C:\Program Files (x86)\Citrix\HDX\bin\wfshell.exe
Pfad des fehlerhaften Moduls: C:\Windows\SYSTEM32\ntdll.dll
Berichtskennung: 0ba766c0-7e62-4c95-b645-785f44d43e23
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 

 

It's not happening all the time and only with a few users.

 

some users complain about a strange issue, where starting a new seamless application the sessions gets disconnected with all the already active applications and reconnected with the new one.

 

 

Meanwhile with vda 2109 the issue changed to wfshell crash but with two rpm errors about URBDRC. See my post below.

[Bjoumlrn Klein]

On 11/11/2021 at 9:11 AM, Peter Loumlf said:

Got one user with this issue.

Can open the first application, but when she tries to open the second the session crashes and reconnects just the first app again.

Anyone managed so fix this issue?

Upgraded to 2109 yesterday, didnt fix it

 

I also see this issue with two users.

As soon as users is logged on, wfshell.exe crashes with reference to vcruntime140.dll.

 

This issue is with VDA 2103, 2106 and still with 2109.

 

I already have a citrix support case open for this, but yet no solution.

 

What is interesting: 

The User does not have this issue, when connected via Netscaler (Citrix Gateway). It is only when directly connected (Office LAN) to the VDA.

Maybe the protocol (UDP (LAN) vs. TCP (Netscaler)) is the difference. I will try this.

 

[Björn Schläfli]

Hi Peter,

 

I've disabled Adaptive Transport since XenApp 7.15 or something like that, since a lot of users had issues with it then. It's still disabled.

 

[Bjoumlrn Klein]

On 12/7/2021 at 7:58 AM, Björn Schläfli said:

I've rolled back to VDA 2012 and disabled the virtual channel security feature. Since then no issues. The issue with apps dis- and reconnect while launching starts with vda 2103 and persists up to 2109. see post Second App prompt a full reconnect then blink active app and open the called app - Receiver for Windows (Updater) - Discussions (citrix.com)

 

Thank you for this hint!

With this information we tried to detach several devices from the client: USB Hub, Headset Charging station, USB Camera.

 

And guess what:

The issue was solved!

It was the USB Camera attached to this device, which would cause the wfshell.exe to crash.

 

As soon as we disconnected the camera, the issue did not occur anymore.

 

So for all which are still having this issue, please check your connected devices (on the client).

 

I think I will open another case for this, so Citrix is able to resolve this.

 

Thank you!

Bjoern  Klein

 

[Björn Schläfli]

Hi Bjoern

This week we've had another case and it was solved by removing the usb Logitech webcam. Webcam like in your case. It starts definetely with VDA 2103.  Feature HDX webcam compression changed in 2103. Maybe it's related to this. Next week I'll get the same webcame type and are able to do detailled tests. I'll let you know.

[Björn Schläfli]

In our case a completey new install of our server image fixed the wfshell.exe crashes and unplug the Logitech webcam fixed the issue with sessions not launching or launching but dis- and reconnect with every app launch.

The webcam issue starts as soon as VDA 2103 and above is used. I've found some information about changes to webcam redirection in vda 2103.

"HDX webcam compression no longer requires the VDA to encode, transcode, and decode H.264 video resources. By default, the server streams video directly to the video conferencing applications running in the virtual session."

I suspect that logitech webcams can't handle this feature.

[Jakub Pruchnik1709162141]

On 1/14/2022 at 3:25 PM, Bjoumlrn Klein said:

 

Thank you for this hint!

With this information we tried to detach several devices from the client: USB Hub, Headset Charging station, USB Camera.

 

And guess what:

The issue was solved!

It was the USB Camera attached to this device, which would cause the wfshell.exe to crash.

 

As soon as we disconnected the camera, the issue did not occur anymore.

 

So for all which are still having this issue, please check your connected devices (on the client).

 

I think I will open another case for this, so Citrix is able to resolve this.

 

Thank you!

Bjoern  Klein

 

 

Who would have thought? In my case disconnecting Logitech Camera also resolved the issue. Thanks for the hint.

[Björn Schläfli]

I've installed VDA 2012 instead of 2103. No more wfshell.exe issues. 2106 has also wfshell.exe crashes. Seems to be an issue starting with version 2103.

[Miguel Resende1709151399]

There's a fix coming for VDA 2109 around wfsell.exe crashing randomly but just from the event cannot say 100% if it's the same as your issue. 

pls generate a dump of the crash and open a case with Citrix if you have a chance - That way you can confirm if it's the same issue already investigated or a different\new one ... -> and obtain a private fix for it. 

otherwise rollback as you figured out or wait\test the next release 2109  (actually just released.. https://www.citrix.com/downloads/citrix-virtual-apps-and-desktops/product-software/citrix-virtual-apps-and-desktops-2109.html || 

https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/whats-new/fixed-issues.html 

·         The wfshell.exe process might exit unexpectedly. The issue occurs when attempting to copy and paste images to and from Microsoft Excel. [CVADHELP-16299]

·         The wfshell.exe process might exit unexpectedly. [CVADHELP-16795]

·         The wfshell.exe process might exit unexpectedly. [CVADHELP-17525]

[Peter Loumlf]

Got one user with this issue.

Can open the first application, but when she tries to open the second the session crashes and reconnects just the first app again.

Anyone managed so fix this issue?

Upgraded to 2109 yesterday, didnt fix it

[Björn Schläfli]

Hi all,

 

I've tested with VDA 2012, 2103, 2106, 2109. All with the same results. Same behavior as Peter Löf has written in his post. Some users open published applications and sometimes there's a WinFrame shell has stopped working error and after it they're unable to start any new applications. If they start another application Workspace App is loading, all open applications disappear and reconnect after 1 - 2 seconds but the new one is not opened. 

Today I've seen two errors in the system event log, which are logged every time an affected user tried to start an application.

 

"NT-AUTORITÄT\SYSTEM tried to open custom virtual channel URBDRC" / Protocol System / Source rpm / Event-ID 16

"Custom virtual channel URBDRC cannot be opened by process [Not available]" / Protocol System / Source rpm / Event-ID 14

 

I've found some information about Virtual Channel Security in https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/secure/virtual-channel-security.html. These errors are listed there. Starting with VDA 2006 Citrix introduced the new feature Virtual Channel Security and with Virtual Apps & Desktops 2109 the feature is enabled by default and only Citrix virtual channels are allowed. In our 2103 environment we have no policy "virtual channel allow list" but I suppose this feature is active in VDAs 2006 and up. Why else should the vda log events 14 and 16?

If it's related to this feature it's also explaining why not all users are affected.

 

I've installed Virtual apps and desktops 2109 in a test environment and grabbed the gpo reg key to disable this feature. Since it's available from vda 2006 and above but the gpo is introduced lately in the 2109 release the older vda versions should be able to read this reg key I think.

The key is:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\VCPolicies

REG_MULTI_SZ VirtualChannelWhiteList

=disabled=

 

I'll test and let you now.

[Björn Schläfli]

Further thoughts on my part:

wfshell is responsible for the redirection of devices (as far as I know) and rpm logs a device which is not possible to redirect because the virtual channel is disabled. Sounds to me like a good point to start investigating. 

Some information about virtual channel security:

https://www.citrix.com/blogs/2021/10/14/virtual-channel-allow-list-now-enabled-by-default/

 

[Peter Loumlf]

On 11/15/2021 at 10:03 AM, Bjoumlrn Klein said:

 

I also see this issue with two users.

As soon as users is logged on, wfshell.exe crashes with reference to vcruntime140.dll.

 

This issue is with VDA 2103, 2106 and still with 2109.

 

I already have a citrix support case open for this, but yet no solution.

 

What is interesting: 

The User does not have this issue, when connected via Netscaler (Citrix Gateway). It is only when directly connected (Office LAN) to the VDA.

Maybe the protocol (UDP (LAN) vs. TCP (Netscaler)) is the difference. I will try this.

 

Same thing with my user. 

It worked when she worked from home, but not from the office.

Have you solved your issue? Have you tried disabling udp (HDX Adaptive transport off) if not i can try, my environment isnt that large.

Just the issue of needing that specific user for testing ?

 

[Björn Schläfli]

I've rolled back to VDA 2012 and disabled the virtual channel security feature. Since then no issues. The issue with apps dis- and reconnect while launching starts with vda 2103 and persists up to 2109. see post Second App prompt a full reconnect then blink active app and open the called app - Receiver for Windows (Updater) - Discussions (citrix.com)

[Ryan Kuebel1709159913]

On 11/22/2021 at 7:40 AM, Björn Schläfli said:

Hi all,

 

I've tested with VDA 2012, 2103, 2106, 2109. All with the same results. Same behavior as Peter Löf has written in his post. Some users open published applications and sometimes there's a WinFrame shell has stopped working error and after it they're unable to start any new applications. If they start another application Workspace App is loading, all open applications disappear and reconnect after 1 - 2 seconds but the new one is not opened. 

Today I've seen two errors in the system event log, which are logged every time an affected user tried to start an application.

 

"NT-AUTORITÄT\SYSTEM tried to open custom virtual channel URBDRC" / Protocol System / Source rpm / Event-ID 16

"Custom virtual channel URBDRC cannot be opened by process [Not available]" / Protocol System / Source rpm / Event-ID 14

 

I've found some information about Virtual Channel Security in https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/secure/virtual-channel-security.html. These errors are listed there. Starting with VDA 2006 Citrix introduced the new feature Virtual Channel Security and with Virtual Apps & Desktops 2109 the feature is enabled by default and only Citrix virtual channels are allowed. In our 2103 environment we have no policy "virtual channel allow list" but I suppose this feature is active in VDAs 2006 and up. Why else should the vda log events 14 and 16?

If it's related to this feature it's also explaining why not all users are affected.

 

I've installed Virtual apps and desktops 2109 in a test environment and grabbed the gpo reg key to disable this feature. Since it's available from vda 2006 and above but the gpo is introduced lately in the 2109 release the older vda versions should be able to read this reg key I think.

The key is:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\VCPolicies

REG_MULTI_SZ VirtualChannelWhiteList

=disabled=

 

I'll test and let you now.

 

 

Any updates on this by chance?

 

Edit------

 

Sorry just saw your post from yesterday.

[Björn Schläfli]

some users reported that they still have issues. Suddenly users are unable to launch a further application. Workspace app launches and disappears. On VDA server wfshell.exe in user's context is not running anymore. Seems to be no crash because there are no events in application logs which I would expect if wfshell.exe crashes. 

So with VDA 2012 no errors in log but still "crashes" of wfshell.exe.

I'll maybe rollback to vda version 1906.2, which we used before our update to virtual apps 2103.

 

 

[Daniel Sjoumlgren]

Does anyone have an update on this? I had this issue in a 2112 environment today.

Unplugged the webcam and it works.

Any official article?

 

Saw this thread with a regfix: https://discussions.citrix.com/topic/415351-logitech-webcams-session-dis-and-reconnection/

[Eduard Huumlbner1709162982]

2 hours ago, Daniel Sjoumlgren said:

Does anyone have an update on this? I had this issue in a 2112 environment today.

Unplugged the webcam and it works.

Any official article?

 

Saw this thread with a regfix: https://discussions.citrix.com/topic/415351-logitech-webcams-session-dis-and-reconnection/

I have a case open for this  issue.

My environment is also 2112.

 

Today Support asked to implement the regkey from your posted link.

 

So far I didn't have chance to test it.