CPU spikes every 30sec on VDAs - WmiPrvSE.exe & Citrix Broker Windows service

[Dawid Bbn]

Hi all

I am raising another issue I found since migrating from CVAD 7.15 LTSR (WS 2012R2) to 2203 LTSR CU3 (WS 2019).

I get CPU spikes every 30 seconds on VDAs caused by the MSMpEng.exe (MS Defender) and WmiPrvSE.exe (WMI Provider Host) Windows processes. The height of the spikes is proportional to the amount of active sessions on the VDAs.

I think I ruled out a root cause on my side: removed all agents from the servers incl. antivirus, used a clean image, manual server build, Group Policy, tested with various published .NET & Java apps, etc. Tried to disable MS Defender completely via GPO.

Tried to use ProcMon and Process Explorer to see what exactly processes are doing but since I am not an expert on these tools could not see anything useful to base on.

The CPU spikes stop as soon as I stop the Citrix Broker Windows service. Stopping other Citrix services does not have any effect on the spikes.

No changes were made to the configuration of the site/farm after the migration from Citrix 7.14 to 2203 env. which could trigger such spikes.

As a side note, initially, on some servers not reporting VDA load to Citrix Cloud/Monitor I had some other CPU spikes present. These were fixed by rebuilding performance counters on them but the main 30 sec spikes remained.

At this stage, I think it is another 2203 LTSR VDA bug and there is nothing I can do about it.

Has anyone experienced similar issues?

Thanks for the help!


ENVIRONMENT:
- Citrix DaaS Service 2023 CU3
- Windows Server 2019
- multisession, persistent VDAs
- VMware

 

 

[Dennis Parker]

Seen this, or something very similar at least, for a very long time, but don't remember exactly what version it started with...

We had to disable resource monitoring to clear it up. 

Citrix Policy - "Enable Resource Monitoring" = Disabled
Lose some metrics like CPU and Memory to Director, but we have other tools to monitor these. 

[Dawid Bbn]

Thanks Dennis for the suggestion.

I've tried that in Citrix Policy and I am afraid this did not help.

 

[Dawid Bbn]

I've just tested 2203 CU4 VDA agent but the issue is still there.

[Dawid Bbn]

Hi all

I figured it out in the end.

My initial findings were not precise, CPU spikes of the MSMpEng.exe (MS Defender) and WmiPrvSE.exe (WMI Provider Host) Windows processes were only a result of this bug.

The actual Windows processes active during the CPU usage spikes could not be seen in Task Manager or Process Explorer - they were only visible in the CPU usage graphs.

It was the VDA Data Collection for Analytics Citrix policy that was causing the CPU spikes.

That makes sense since the CPU usage spikes did not appear on the recently deployed on-prem farm - Citrix Analytics is a cloud-only, premium service which we don't even have access to.

The strange thing was that this was only affecting our .NET & Java apps while published desktops and Chrome were fine.


Very disappointing bug, lots of wasted time on troubleshooting and testing. And this is more than two years since the VDA 2203 LTSR was released.

I thought the LTSR version should be rock-stable and polished while we only got all the memory fixes done in 2203 CU3 - to me, memory management on pre-VDA 2203 CU3 was completely broken (this bug also impacted our production systems, taking them to a standstill)..