Jump to content
Welcome to our new Citrix community!
  • 0

Servers are running out of RAM memory - termsvcs.exe process & Non-page pool bloat


Dawid Bbn

Question

Hi all,

Since the migration of our Citrix server estate from XenApp 7.15 to DaaS Service 2203 LTSR, on all VMs with published applications, we started to see memory issues.

It looks like the termsvcs.exe (svchost.exe) process is gradually taking up ram memory, then slowing down servers. Also, the Non-paged pool memory is bloated.

Do you have any idea what may be causing this?

So far I've only done basic troubleshooting, e,g, rolled back VDA to 2203 CU1, tried 2303, ruled out Group Policy.


Environment:
Citrix DaaS Service
Windows Server 2019
VDA 2203 CU2
VMware
IGEL thin clients

image.thumb.png.4bdfc085a07c3766f42e48391c8fe0e2.png

image.thumb.png.a91793e9b17449aa246c2bc47a56bd30.png

7eecccb4-749f-46a7-a84c-2d6a78c3e6ba.png

 

2023-04-1511_43_51.thumb.png.b76166ec6385f49bec7d5c37f4cf46a7.png

Link to comment

6 answers to this question

Recommended Posts

  • 0

Controlup agent virtual channel CUEPUX can be a source of matching behavior, resulting in large memory utilization by termsvc.

https://www.controlup.com/controlup-release-notes/

112130 – CUEPUX virtual channels are flooding the multi-session VDA logs with RPM events leading to performance issues.

A work around to that is to change virtual channel security back to pre 2109 defaults:
https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/ica-policy-settings/virtual-channel-allow-list-policy-settings.html

But if you do not have ControlUP agent, maybe another 3rd party component with a custom virtual channel.

Link to comment
  • 0

@Dawid Beben, did you find a solution ?

 

I'm in the same situation (Windows Server 2022 / CVAD 2203 CU2), with a VDA server 12vCPU/48GB RAM I can only host 10 to 12 users...

 

I have notice this fix issue on the new release CU3 (https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/2203-ltsr/whats-new/cumulative-update-3/fixed-issues.html) :

 

The CseEngine.exe service might consume higher memory than expected on the VDAs. [CVADHELP-19226]

Link to comment
  • 0

Hi@Baptiste Potiron

I was actually going to mention the 2203 CU3 update and the memory-related fixes there.

At first look, none of the memory fixes seem to be directly applicable to me but I am obviously now testing the new new VDA agent.

I will let you know if the upgrade helps.


In my troubleshooting so far I found out that the issue starts to happen after I move a server from a Desktop only (with published Windows desktop) Delivery Group (~12 concurrent sessions) to an Application only Delivery Group (~30 concurrent sessions).

So at this stage, to me a specific Delivery Group (functional level), applied Citrix Policy, or amount of concurrent sessions may trigger the memory bloat. I am now in the process of ruling this theory out. In my env. it takes a couple of days after the VDA reboot for the issue to re-appear.


 

Link to comment
  • 0
On 6/6/2023 at 10:39 PM, Dawid Beben said:

Hi@Baptiste Potiron

I was actually going to mention the 2203 CU3 update and the memory-related fixes there.

At first look, none of the memory fixes seem to be directly applicable to me but I am obviously now testing the new new VDA agent.

I will let you know if the upgrade helps.


In my troubleshooting so far I found out that the issue starts to happen after I move a server from a Desktop only (with published Windows desktop) Delivery Group (~12 concurrent sessions) to an Application only Delivery Group (~30 concurrent sessions).

So at this stage, to me a specific Delivery Group (functional level), applied Citrix Policy, or amount of concurrent sessions may trigger the memory bloat. I am now in the process of ruling this theory out. In my env. it takes a couple of days after the VDA reboot for the issue to re-appear.


 

 

 

Hello, Did anyone else come up with a solution?  I have the same issue.  Termsvc reached 4GB memory on one of my servers.  Session connected for 30+ days to reach that.

Citrix support gave me the, "please patch to CU3 or call Microsoft to log a support call" answer.

Link to comment
  • 0

Hi @Anthony Thompson

Sorry for not responding earlier.

In my case, the 2203 CU3 VDA version fixed the memory bloat issues.

I am happy with the improvement but actually shocked and surprised that such serious bugs were still present and not fixed in 2203 CU2 VDA.

LTSR versions are supposed to be stable and rested but in this case, it turned out it takes like a year since release to address major issues.

These memory bugs caused lots of stability issues in my production, and warehouse environments and raised complaints from operations, also affecting our customers' orders.

This is a major failure of Citrix and will take a lot of time to regain the trust of some customers.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...