Just Grey Screen instead of Published Desktop

[Rene Balz1709162432]

Hi,

 

over the last few weeks, a weird Citrix issue started to occur on several completely independent Citrix Session Hosts (from different AD domains). Some details about the environments:

 

• Server 2019 based Hyper-V architecture
• Virtual Citrix Session Host (Published Desktop) with fully patched Windows Server 2019 and VDA 7 2012

 

The issue is as follows:

 

• In the event log of the Session Host, an CtxUvi error with ID 1005 is being logged: "The Citrix Universal DLL Injection Driver has encountered an unexpected error."
• Right after that, an error 1003 (also from CtxUvi) is being logged:  "The Citrix Universal DLL Injection Driver has detected an integrity error during process creation. The Citrix Universal DLL Injection Driver has been disabled."
• From now on, all users who freshly log into the desktop just receive a blank grey screen as soon as Citrix Workspace is creating the session window.
• After a server reboot, everything works normal again.

 

Some observations / details:

 

• The issue is occuring about once every week (per Server) but there is no temporal pattern to be found.
• The CtxUvi crash is always being caused by a user who's logging off, so the issue always occurs either before lunchbreak or in the evening (~5 pm). But it's never the same user who's causing it.
• Apart from the grey screen, the session actually works. It is, as an example, possible to blindly create a text file on the desktop (with keyboard shortcuts), and the file actually gets created.
• Like I said, several customers are affected.

 

Does someone have any ideas how to fix this or tips for further diagnosis?

 

Best regards

 

 

[scottyt]

On 1/19/2022 at 2:18 AM, Rene Balz1709162432 said:

Hi!

 

Unfortunately, the greyscreen issue has returned.. At least 6 occurences in the last 24 hours (3 or 4 different customers). The CTXUvi exclusions in the registry are still intact and do not help this time ?

 

There are a few differences compared to the occurences in Spring 2021:

 

• 1003 events are being logged just once when the issue occurs, not every 5 minutes (which was the case last year, IIRC)
• The issue does not seem to get triggered by a user signing out (logging off), but seemingly at random
• The problem is now occuring much more often, which is a real problem

Will keep you posted..

 

Has anyone else seen the issue coming back recently?

 

Best regards

Any luck here?

We've had several customers with the issue and an open ticket with Citrix; hoping to hear back in the next day or so after CDFControl logs capturing the issue were provided to Citrix yesterday. 

Aside from the similarities with Trend AV, albeit with different engine versions and scanning options, I note that Adobe Acrobat reader 32bit has automatically updated to a new version 21.011.20039 dated Jan 11ish. Anybody else using the same version of reader or NOT out of interest?

[Isaac Laris]

Having the same issue after rolling out windows patches. Started on 1/24/22 and happened again today 1/25/22 using MCS on Server 2016 with 1912 CU3 and TM Apex One.

 

about 20% of VDA's are experiencing the ctxuvi errors over the last two days. The errors happen during peak log on hours.

 

So far Citrix Support has recommended :

1. Create the registry key HKLM\SYSTEM\CurrentControlSet\Services\CtxUvi = REG_DWORD “UviEnabled” and set it to 1

1. Uninstall VDA software and reinstall.

[Isaac Laris]

On 1/21/2022 at 8:33 AM, Arnaud LABICHE1709152531 said:

Hi,

Same issue here on a Virtual Apps 1912 farm / Windows Server 2016. Windows Update seems to resolve the issue.

 

 

So strange, January Updates seemed to trigger the issue for us.

[Gerald Muller1709156932]

7 hours ago, Isaac Laris said:

 

 

So strange, January Updates seemed to trigger the issue for us.

Hi,

we are experiencing the same issue on Windows Server 2019. Did you solve the issue at your side?

Gerald

[Loic Christ1709151777]

Same issue here but waiting for Citrix support 

[Isaac Laris]

8 hours ago, Gerald Muller1709156932 said:

Hi,

we are experiencing the same issue on Windows Server 2019. Did you solve the issue at your side?

Gerald

 

Trend Micro has sent us a patch for Apex One. We will be implementing this on our Golden Image and I will report back tomorrow morning. Had to kick users off and reboot 20% of my VDA's again this morning.  I did not go through with any registry changes or reinstalling VDA software.

[Loic Christ1709151777]

 

16 hours ago, Isaac Laris said:

 

Trend Micro has sent us a patch for Apex One. We will be implementing this on our Golden Image and I will report back tomorrow morning. Had to kick users off and reboot 20% of my VDA's again this morning.  I did not go through with any registry changes or reinstalling VDA software.

 

Could you give us the case number on TrendMicro support to have this hotfix please ?

[Fabian Wartenberg1709163012]

16 hours ago, Isaac Laris said:

 

Trend Micro has sent us a patch for Apex One. We will be implementing this on our Golden Image and I will report back tomorrow morning. Had to kick users off and reboot 20% of my VDA's again this morning.  I did not go through with any registry changes or reinstalling VDA software.

Yes i would like to have that too, referencing your case might accelerate ours. 

[Isaac Laris]

7 hours ago, Loic Christ1709151777 said:

 

 

Could you give us the case number on TrendMicro support to have this hotfix please ?

 

7 hours ago, Fabian Wartenberg1709163012 said:

Yes i would like to have that too, referencing your case might accelerate ours. 

 

 

Patch appears to be successful at remediating the issue. Zero ctxuvi errors during our peak log on hours today.

 

Running 1912 and TM Apex One on Server 2016

 

Case Number: 05273810

 

 

 

[Fabian Wartenberg1709163012]

14 hours ago, Isaac Laris said:

 

 

 

Patch appears to be successful at remediating the issue. Zero ctxuvi errors during our peak log on hours today.

 

Running 1912 and TM Apex One on Server 2016

 

Case Number: 05273810

 

 

 

Well that sounds promising! I will append my Trend case with your case number then, but is there any chance you could give me the name of the hotfix they gave you? 

[Jan-Peer Gebauer]

Hi.

We are running VDA 2109 on Server 2019 with TM ApexOne 14.0.9645 and got the same login problems with gray screens when starting a published desktop.

We contacted the Trend Micro support. They send us a new version of the file "tmlocfg.ptn". 

 

On a VDA (or VDA master image) I had to stop ApexOne, replace tmlocfg.ptn in the locations:

C:\Program Files (x86)\Trend Micro\Security Agent
C:\Program Files (x86)\Trend Micro\BM\Patterns
C:\Program Files (x86)\Trend Micro\BM\update

Restart the VDA.

 

After these steps it seems to be solved. No more gray screens in the last 2 days.

What is your experience with the ApexOne patch? Did you get the same solution?

 

JP

[lterpst151]

19 hours ago, Jan-Peer Gebauer said:

Hi.

We are running VDA 2109 on Server 2019 with TM ApexOne 14.0.9645 and got the same login problems with gray screens when starting a published desktop.

We contacted the Trend Micro support. They send us a new version of the file "tmlocfg.ptn". 

 

On a VDA (or VDA master image) I had to stop ApexOne, replace tmlocfg.ptn in the locations:

C:\Program Files (x86)\Trend Micro\Security Agent
C:\Program Files (x86)\Trend Micro\BM\Patterns
C:\Program Files (x86)\Trend Micro\BM\update

Restart the VDA.

 

After these steps it seems to be solved. No more gray screens in the last 2 days.

What is your experience with the ApexOne patch? Did you get the same solution?

 

JP

Do you have a case ID or the link to the file? We would love to try this also.

[Jan-Peer Gebauer]

4 hours ago, lterpst151 said:

Do you have a case ID or the link to the file? We would love to try this also.

Currently you can get the file only from the Trend Micro support. There is no download on official Trend Micro web pages.
TM Case: 05300965

 

[Dennis Reimer1709161851]

On 2/10/2022 at 2:43 PM, Rene Balz1709162432 said:

I have found a completely different solution, which works flawlessly so far (2 weeks):

• On all session hosts, Add RdrCEF.exe to UviProcessExcludes under HKLM\SYSTEM\CurrentControlSet\Services\CtxUvi

 

Can you tell more details. Why should this Adobe Reader process cause the CTXUvi/Greyscreen Issue? I´m asking because I have several customers with this issue and haven´t found a final solution yet. 

[mfprybell]

We just started having this problem as well 1912 CU1 on 2019 servers. We use Sophos AV on our hosts. Disabling Sophos solved the driver failure. I have now updated Sophos VM AV on the master and pushed it out as well as adding the above mentioned exclusions. Hopefully this will resolves the problem and won't have to uninstall Sophos on the hosts.

[Christian Radatz]

Sorry, no native speaker - did my best ?

 

I had the same error on 1912 CU4 on 2016 and Sophos AV. The error occours mostly five to ten minutes after a reboot and sometimes in the middle of the day.

 

I saw the fallowing errors: 

1) CtxUri-Error at the Eventlog.

2) Some Failed messages in the Sophos Log at the same second  (enabled verbose logging HKLM\SYSTEM\CurrentControlSet\Services\SAVOnAccess [DWORD]LogFlags = 0xff)

• Failed messages pointing to C:\Program Files (x86)\Citrix\System32\RandomCitrixHookingDll and  %userprofile%\AppData\Local\Adobe\AcroCef\DC\Acrobat\Cache\Code Cache
• Excluding both paths did nothing
• disable Sophos or exclude C-Drive resolves the issue

3) Process Tracking (Thanks to Rene Balz!) showed me that a AcroCef.exe always starts at the same second. 

• Adobe suggested to disable DLL-injection for Acrobat and Reader. You need need the newest version for this. Adobe had a lot of trouble with DLL injection from some AV vendors. Sadly Sophos or Citrix was not on there list.

[HKEY_LOCAL_MACHINE\SOFTWARE \Adobe\Adobe Acrobat\DC\DLLInjection] "bBlockDLLInjection"=01

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Adobe\Adobe Acrobat\DC\DLLInjection] "bBlockDLLInjection"=01

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Adobe\Acrobat Reader\DC\DLLInjection] "bBlockDLLInjection"=01

 

4) Adding AcroCef.exe and RdrCEF.exe to HKLM\SYSTEM\CurrentControlSet\Services\CtxUvi\UviProcessExcludes solves the problem. After around 700 reboots the error didn't occur. 

 

Adobe added AcroCef.exe and RdrCEF.exe some month ago. 

 

[Florent GUILLOT1709161233]

Hello,

We also have this problem in 2203 Cu1 (we also had the problem in 1912CU1)

In the windows event handler,

I get error 1005 "Citrix Universal DLL Injection driver encountered an unexpected error."

how can i see if adobe is the problem or another application?

 

thank you

[Peter Singer1709154229]

Is there a definitive fix for this as it seems to be back in 1912 CU7?