Just Grey Screen instead of Published Desktop

[Rene Balz1709162432]

Hi,

 

over the last few weeks, a weird Citrix issue started to occur on several completely independent Citrix Session Hosts (from different AD domains). Some details about the environments:

 

• Server 2019 based Hyper-V architecture
• Virtual Citrix Session Host (Published Desktop) with fully patched Windows Server 2019 and VDA 7 2012

 

The issue is as follows:

 

• In the event log of the Session Host, an CtxUvi error with ID 1005 is being logged: "The Citrix Universal DLL Injection Driver has encountered an unexpected error."
• Right after that, an error 1003 (also from CtxUvi) is being logged:  "The Citrix Universal DLL Injection Driver has detected an integrity error during process creation. The Citrix Universal DLL Injection Driver has been disabled."
• From now on, all users who freshly log into the desktop just receive a blank grey screen as soon as Citrix Workspace is creating the session window.
• After a server reboot, everything works normal again.

 

Some observations / details:

 

• The issue is occuring about once every week (per Server) but there is no temporal pattern to be found.
• The CtxUvi crash is always being caused by a user who's logging off, so the issue always occurs either before lunchbreak or in the evening (~5 pm). But it's never the same user who's causing it.
• Apart from the grey screen, the session actually works. It is, as an example, possible to blindly create a text file on the desktop (with keyboard shortcuts), and the file actually gets created.
• Like I said, several customers are affected.

 

Does someone have any ideas how to fix this or tips for further diagnosis?

 

Best regards

 

 

[Jonathan Pitre]

I'm seing the same issue with these reg keys with Citrix Cloud and Azure Standard NV4as_v4 VM (AMD) with Citrix VDA 2103 on Win10 20H2. As soon as I delete the regs keys the issue is fixed.
 

"EnableWPFHook" - SOFTWARE\Citrix\CtxHook\AppIni_Dlls\Multiple Monitor Hook

"EnableWPFHook" - SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook

"Open GL" - SOFTWARE\Citrix\CtxHook\AppIni_Dlls\Graphics Helper

"Open GL" - SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper


@Citrix, since these reg keys are recommended with GPU acceleration, can you please investigate what's going on?
https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/graphics/hdx-3d-pro/gpu-acceleration-server.html

 

[James Kindon]

I’ve been spanked by that key set as well - I never set them anywhere as it’s grey screen death

[James Rankin1709162195]

I've got this issue and we don't have those Registry values set. We do use Trend though.....

[Rene Balz1709162432]

Short update: Since re-enabling Behavior Monitoring, the issue has been appearing again..

[Mark Robinson1709151368]

Hi all, 

 

An update from me.  So I enabled verbose logging and its not been totally clear if Sophos is the culprit however I spotted on one log around 1 hr before the CtxUvi error the log showed:

 

2021-04-24T12:28:17.010Z [70] [3948:8772] SaviNotify.cpp(1079) : Suppressed SOPHOS_SAVI_ERROR_SWEEPFAILURE (operation cancelled), name=C:\Program Files (x86)\Citrix\HDX\bin\PicaUiTweakHook.dll
2021-04-24T12:28:17.010Z [70] [3948:8772] SaviWrapper.cpp(1054) : Warning: Failed to sweep object, hr=0xA0040229, name=C:\Program Files (x86)\Citrix\HDX\bin\PicaUiTweakHook.dll
2021-04-24T12:28:17.010Z [80] [3948:8772] ThreatDetectionEngine.cpp(888) : [a0040229] Processor returned error for C:\Program Files (x86)\Citrix\HDX\bin\PicaUiTweakHook.dll, skipping to the decision handlers
2021-04-24T12:28:17.010Z [80] [3948:8772] ThreatDetectionEngine.cpp(216) : [a0040229] DoCheck returned error.
2021-04-24T12:28:17.010Z [80] [3948:8772] EngineManagement.cpp(259) : [a0040229] CEngineManager::Scan - Call to engine->Check(ScannableNode) for C:\Program Files (x86)\Citrix\HDX\bin\PicaUiTweakHook.dll returned an error code. Attempting to continue.
 

As I've not had much progress on this I decided to put exclusions in for ''On Access Scanning and 'Scheduled Scans' for the 3 main locations of:

 - C:\Program Files (x86)\Citrix\

 - C:\Program Files\Citrix\

 - C:\ProgramData\Citrix\

 

This was done yesterday morning and pushed out to all my Citrix servers and since then we haven't had a grey screen issues at all reported.  As mentioned previously we have approx 30 servers with 300 users daily and always get issues in the afternoon & evenings but nothing since the change.   If this stays good for the next few days I will slowly remove each exclusion and see when it fails. 

 

Its a bit early to say this is pointing in the right direction but I thought I would post an update for others.  

Also as a side note we didn't have those registry mentioned by Jonathan Pitre a few entries ago. 

[Jonathan Pitre]

I was told by a Citrix engineer that these reg keys values must be tattooed to your master image before sending an image update. I was pushing them via GPO to my target devices only.

 

I need to test and I'll report back.

[Mark Robinson1709151368]

Hi all, 

 

Further update from me.  Since doing the exclusions in for Sophos ''On Access Scanning and 'Scheduled Scans' to the 3 main folders we still have had no occurrence of the grey screen.  So I think this is definitely a solution, be it a sledge hammer to crack a nut, as we previously had the issues every single day without fail up until I did this change.  

 

As mentioned before I will now try to tie down what area is the route cause.  Also I have not made any change to behavioural monitoring.

[Chris Gundry]

We have had no further occurrences since we added our in house app to the 'trusted programs' list. Very odd as this app has been in place, processing info during user logon for the last 2-3 years, only in the last few weeks has it become a problem for Trend.

[Jeremy Saunders]

For those that experience this issue and are referring to setting the following keys via a Group Policy...

 

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook] "EnableWPFHook"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Multiple Monitor Hook] "EnableWPFHook"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Graphics Helper] "CUDA"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper] "CUDA"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\Graphics Helper] "OpenCL"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\CtxHook\AppInit_Dlls\Graphics Helper] "OpenCL"=dword:00000001

 

You need to apply these in the image and not via a Group Policy. It's a race condition related to the CtxUvi driver. Any changes to the AppInit_DlLLs structure after it has started will cause the CtxUvi driver to disable itself until the next reboot. When it disables itself it can cause a grey screen and all sorts of issues for the users. This was a change Citrix made from 7.9 and above. References: CTX220418, CTX226605, CTX223973.

 

Obviously applying via Group Policy would be hit and miss depending on when the policy applies and when the CtxUvi driver starts. Hence a race condition.

 

I feel that instead of disabling the CtxUvi driver when changes to the AppInit_DlLLs structure are detected, it would be best ignoring those changes, log an event, and continue to function as expected. That's proper engineering!

[Jonathan Pitre1709161645]

@JeremySaunders is right about the reg values. However in my case, even when they are applied manually and tattooed to the base image, I still get the gray screen of death.

The only workaround is to remove them completely. Now it is still unclear if these reg values are still required. I'm still waiting on Citrix for confirmation.

If I could get my hands on some WPF benchmarking tool I could tell if GPU offloading is working or not.

For OpenCL, GPU Caps Viewer provide OpenCL demos and they work fine without the reg value.

 

 

[Graeme Dunkley]

Hi Folks, 

RE: Grey Screen and Trend Micro 

Quick update on this issue. Citrix engineering have been investigating and the issue with CTXUVI stopping with error 1003/1005 with Trend Micro AV installed. The error occurs when the CTXUVI driver receives an ACCESS DENIED error when attempting to load one of the Citrix hooks into a process. We believe the ACCESS DENIED is caused by Trend Micro AV opening Citrix hook DLLs with exclusive access when being scanned, instead of opening them with a share access flag (eg FILE_SHARE_READ) 

 

Note that Microsoft Defender and other vendors do not appear to have the same effect and we have customer evidence that the issue is resolved by removing Trend from their VDAs, because of this we are encouraging customers to contact Trend Micro support so they are aware of the scale of the issue.  

 

The investigation is on-going but the following workaround to exclude the following executables from hooking has been known to reduce the frequency of the issue (These are active during logon/logoff, and so are often being scanned when the issue occurs)

Add Citrix hook exclusions for:
SelfService.ex;CtxWebBrowser.;Receiver.exe
Append the above to: the end of:  UviProcessExcludes under: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi\
then reboot the VDA.

 

Please note: The misspelling is deliberate, you need to ensure that the exe names added to exclusion list does not exceed 14 characters for the exclusion to take effect. (see: https://support.citrix.com/article/CTX107825 for more information)

 

Caution! Using Registry Editor incorrectly can cause serious problems that might require you to reinstall your operating system. Citrix cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk. Be sure to back up the registry before you edit it.

 

@ReneBalz, @ChrisGundry @JamesKindon - can you confirm that disabling behavioural monitoring in Trend AV resolved the issue for you?

 

RE: Grey Screen and Sophos 

The issue is slightly different in that it is not ACCESS DENIED but OPERATION CANCELLED when trying to load hooks, this seems to be resolved by adding the folders to Sophos' exclusion list.

1912:

C:\Program Files (x86)\Citrix\HDX\bin

C:\Program Files\Citrix\HDX\bin

7.15:

C:\Program Files (x86)\Citrix\System32

 

Citrix Engineering have made improvements to try and workaround these issues and this is on going, but ultimately the fix needs to come from the AV vendor.

 

Best Regards,

 

Graeme

[Rene Balz1709162432]

@Graeme Dunkley

 

Thank you so much for your reply! Very helpful information. So I was close 2 months ago when I added these processes to the CtxUvi exclusions, but I didn't know about the 14 char limit 

 

However, the issue seems to have disappeared since upgrading to VDA 2103.. But the "Sample size" is not yet large enough to be sure. I just reverted all TM related exclusion and settings to it's prior configuration.

 

can you confirm that disabling behavioural monitoring in Trend AV resolved the issue for you?

Yes, definetly. Here's the issue history from one of our customers:

 

• 01.03.2021 - CtxUvi error
• 02.03.2021 - CtxUvi error    
• 03.03.2021 - CtxUvi error
• 08.03.2021 - CtxUvi error
• 10.03.2021 - CtxUvi error
• 12.03.2021 - CtxUvi error
• 12.03.2021 - CtxUvi error
• 18.03.2021 - CtxUvi error
• 18.03.2021 - CtxUvi error
• 19.03.2021 - CtxUvi error
• 22.03.2021 - CtxUvi error
• 29.03.2021 - CtxUvi error
• 29.03.2021 - CtxUvi error
• 31.03.2021 - CtxUvi error
   
• 31.03.2021 - TM BM disabled
   
• 16.04.2021 - TM BM enabled
   
• 22.04.2021 - CtxUvi error
• 24.04.2021 - CtxUvi error
• 26.04.2021 - CtxUvi error
• 27.04.2021 - CtxUvi error
• 29.04.2021 - CtxUvi error
• 30.04.2021 - CtxUvi error
• 02.05.2021 - CtxUvi error
   
• 02.05.2021 – Upgrade to 2103

 

 

[Daniel Vester]

On 5/7/2021 at 5:53 PM, Graeme Dunkley said:

Hi Folks, 

RE: Grey Screen and Trend Micro 

Quick update on this issue. Citrix engineering have been investigating and the issue with CTXUVI stopping with error 1003/1005 with Trend Micro AV installed. The error occurs when the CTXUVI driver receives an ACCESS DENIED error when attempting to load one of the Citrix hooks into a process. We believe the ACCESS DENIED is caused by Trend Micro AV opening Citrix hook DLLs with exclusive access when being scanned, instead of opening them with a share access flag (eg FILE_SHARE_READ) 

 

Note that Microsoft Defender and other vendors do not appear to have the same effect and we have customer evidence that the issue is resolved by removing Trend from their VDAs, because of this we are encouraging customers to contact Trend Micro support so they are aware of the scale of the issue.  

 

The investigation is on-going but the following workaround to exclude the following executables from hooking has been known to reduce the frequency of the issue (These are active during logon/logoff, and so are often being scanned when the issue occurs)

Add Citrix hook exclusions for:
SelfService.ex;CtxWebBrowser.;Receiver.exe
Append the above to: the end of:  UviProcessExcludes under: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi\
then reboot the VDA.

 

Please note: The misspelling is deliberate, you need to ensure that the exe names added to exclusion list does not exceed 14 characters for the exclusion to take effect. (see: https://support.citrix.com/article/CTX107825 for more information)

 

Caution! Using Registry Editor incorrectly can cause serious problems that might require you to reinstall your operating system. Citrix cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk. Be sure to back up the registry before you edit it.

 

@ReneBalz, @ChrisGundry @JamesKindon - can you confirm that disabling behavioural monitoring in Trend AV resolved the issue for you?

 

RE: Grey Screen and Sophos 

The issue is slightly different in that it is not ACCESS DENIED but OPERATION CANCELLED when trying to load hooks, this seems to be resolved by adding the folders to Sophos' exclusion list.

1912:

C:\Program Files (x86)\Citrix\HDX\bin

C:\Program Files\Citrix\HDX\bin

7.15:

C:\Program Files (x86)\Citrix\System32

 

Citrix Engineering have made improvements to try and workaround these issues and this is on going, but ultimately the fix needs to come from the AV vendor.

 

Best Regards,

 

Graeme

Thanks a lot! I think this fixed our issues also after adding the exclusions. Cross my fingers that it will stay this way :)

[Zeno Maes]

First I tried at recommendation of Citrix to add the SelfService.exe,CtxWebBrowser.exe and Receiver.exe to the Trend Micro behavior monitor exclusions.

That didn't help. 

Then I tried (also a Citrix recommendation and also on this forum) : 

Add Citrix hook exclusions for:
SelfService.ex;CtxWebBrowser.;Receiver.exe
Append the above to: the end of:  UviProcessExcludes under: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi\
then reboot the VDA.

That didn't help either. 

But doing them both seems to do the trick, hadn't had a 1005 event for more then a week.

[Michael Shirtcliff1709159619]

Just opened a case for very similar behaviour (Windows 2019, VDA 2103 image from MCS delivered to Azure) however we are running Defender and have the documented AV exclusions in place. When the issue occurs in our environment we are seeing the CtxUvi event ID 1003/1005 occur shortly after server boot at which point any sessions complete login (profile loads, monitor reports session active and connected) but users only get gray screen.

As with the other cases documented here, reboot fixes the issue and it is only intermittent (out of 35 servers each week we only see this 1-3 times). Our instances are Autoscale enabled as well as a daily reboot schedule to help mitigate a separate issue caused by FSLogix 2009 not dismounting the profile container at logoff).

Will see what support come back with.

[Chris Gundry]

On 5/7/2021 at 4:53 PM, Graeme Dunkley said:

Hi Folks, 

RE: Grey Screen and Trend Micro 

Quick update on this issue. Citrix engineering have been investigating and the issue with CTXUVI stopping with error 1003/1005 with Trend Micro AV installed. The error occurs when the CTXUVI driver receives an ACCESS DENIED error when attempting to load one of the Citrix hooks into a process. We believe the ACCESS DENIED is caused by Trend Micro AV opening Citrix hook DLLs with exclusive access when being scanned, instead of opening them with a share access flag (eg FILE_SHARE_READ) 

 

Note that Microsoft Defender and other vendors do not appear to have the same effect and we have customer evidence that the issue is resolved by removing Trend from their VDAs, because of this we are encouraging customers to contact Trend Micro support so they are aware of the scale of the issue.  

 

The investigation is on-going but the following workaround to exclude the following executables from hooking has been known to reduce the frequency of the issue (These are active during logon/logoff, and so are often being scanned when the issue occurs)

Add Citrix hook exclusions for:
SelfService.ex;CtxWebBrowser.;Receiver.exe
Append the above to: the end of:  UviProcessExcludes under: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi\
then reboot the VDA.

 

Please note: The misspelling is deliberate, you need to ensure that the exe names added to exclusion list does not exceed 14 characters for the exclusion to take effect. (see: https://support.citrix.com/article/CTX107825 for more information)

 

Caution! Using Registry Editor incorrectly can cause serious problems that might require you to reinstall your operating system. Citrix cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk. Be sure to back up the registry before you edit it.

 

@ReneBalz, @ChrisGundry @JamesKindon - can you confirm that disabling behavioural monitoring in Trend AV resolved the issue for you?

 

RE: Grey Screen and Sophos 

The issue is slightly different in that it is not ACCESS DENIED but OPERATION CANCELLED when trying to load hooks, this seems to be resolved by adding the folders to Sophos' exclusion list.

1912:

C:\Program Files (x86)\Citrix\HDX\bin

C:\Program Files\Citrix\HDX\bin

7.15:

C:\Program Files (x86)\Citrix\System32

 

Citrix Engineering have made improvements to try and workaround these issues and this is on going, but ultimately the fix needs to come from the AV vendor.

 

Best Regards,

 

Graeme

 

Very interesting, thank you for the update.

 

In our case, we identified that it occurred when an in house developed .exe was being accessed by Trend. We added that app to Trend trusted programs and the issue has not occurred since.... So not sure we have 100% the same issue, but it is similar... This in house .exe has been in place and has been unchanged for several years, with no recent Citrix changes, so the issue has to be caused by some Trend change/definition update etc.

[Rene Balz1709162432]

Hi there!

 

Just wanted to give an update. I've created a GPO with the fix provided by Graeme Dunkley and imported it on all affected AD domains, and it seems to work flawlessly so far! Thanks so much.

[Zeno Maes]

I spoke to soon.  The problem went away for about a week , with the fix from Graeme Dunkley applied and the exe's added to the behavior monitor exclusions from Trend. The frequency definitely was less then before.  I applied the fix on halve of the servers with CU7 (24) , the other half is still on CU6 with the regkey fix for HDXMediaStreamForFlash applied. But even there I got one server every 1 to 2  weeks with the problem, only the data from the 1005 event is different. 

Citrix closed my case , saying it was a problem Trend had to solve. Trend says they can't solve it. So I'm at a los , the only thing left to try is disable behavior monitoring completely. But it can't be that you have to compromise your security to have a problem fixed that wasn't there before (CU5)  

 

[Zeno Maes]

Is there anybody here for whom the problem is completely solved ? I've added all "fixes" on this forum as well as the one I got from Citrix and Trend Micro and stil occasionally have this problem. We have 48 Xenapp servers and about 1 server a week still fails. 

The latest fix I got from Citrix, also adding  TMBMSRV.exe,NTRTScan.exe  to the UviProccessExcludes didn't help either. 

The only option Citrix has left me is to completely remove Trend micro DSA (I already disabled Behavior Monitoring). But removing all security in a production environment is one bridge to far. (Defender is not a valid alternative)

I'm also getting very tired of Citrix keep pushing me to archive the case. I still don't have a solution so the case stays open.

[Zeno Maes]

Today I received the 20.0.0.2593 version of the Trend Micro deep security agent , which should solve the problem according to trend.

We received news that the latest DSA released, Build 20.0-2593, fixes the issue concerning Citrix.
https://files.trendmicro.com/products/deepsecurity/en/20.0/Agent-Windows-20.0.0-2593.x86_64.zip
"Citrix Virtual App or Desktop users sometimes encountered a grey screen (with error code 1003/1005) when Anti-Malware was enabled for Deep Security Agent. DS-64318"
Please update your Master Image with this Build and verify whether or not the issue still persists.

 

So lets hope it actually works.


 

[Rene Balz1709162432]

Hi!

 

Unfortunately, the greyscreen issue has returned.. At least 6 occurences in the last 24 hours (3 or 4 different customers). The CTXUvi exclusions in the registry are still intact and do not help this time ?

 

There are a few differences compared to the occurences in Spring 2021:

 

• 1003 events are being logged just once when the issue occurs, not every 5 minutes (which was the case last year, IIRC)
• The issue does not seem to get triggered by a user signing out (logging off), but seemingly at random
• The problem is now occuring much more often, which is a real problem

Will keep you posted..

 

Has anyone else seen the issue coming back recently?

 

Best regards

[Chris Gundry]

3 minutes ago, Rene Balz1709162432 said:

Hi!

 

Unfortunately, the greyscreen issue has returned.. At least 6 occurences in the last 24 hours (3 or 4 different customers). The CTXUvi exclusions in the registry are still intact and do not help this time ?

 

There are a few differences compared to the occurences in Spring 2021:

 

• 1003 events are being logged just once when the issue occurs, not every 5 minutes (which was the case last year, IIRC)
• The issue does not seem to get triggered by a user signing out (logging off), but seemingly at random
• The problem is now occuring much more often, which is a real problem

Will keep you posted..

 

Has anyone else seen the issue coming back recently?

 

Best regards

 

I am interested to see that you have said you are having the issue again. Yesterday we updated an MCS image and the master image logged the CTXUvi error in it's own logs at the time the admin logged in to update the image. This has never been observed before by us. We have, so far, not had any occurrences in our production catalogs, just the single instance in the master image. But it is very odd that we had it yesterday and you have also started having issues again...

 

We have also not changed any settings, we did not use the registry options, but we excluded a particular bespoke internal program from Trend by adding it to Trent trusted programs and that resolved out problem. The setting is still in place, so no idea why we got the error this time...

[Robert Goldsmith]

Hi all,

Just wanted to join in on saying that we had the grey screen issues early last year and that was resolved... but like a couple others have posted it has returned since the new year and I can't find any solutions. Wondering if anyone has had any luck with this really annoying issue or know if there's been some sort of windows update. We do have Trend Micro installed.

 

Thanks

[Arnaud LABICHE1709152531]

Hi,

Same issue here on a Virtual Apps 1912 farm / Windows Server 2016. Windows Update seems to resolve the issue.

[Marco Wingenfeld]

Same issue. Virtual Desktops 1912 CU4/ 2012R2, Trend Micro Worry Free Business Security.

When the issue occurs, no new users can login on that server until it is rebooted.

 

CTXUVI Event logged multiple times.

 

Open support ticket for 7 days without any advice how to fix the issue.