Joe Robinson

March 4, 2023

On 2/28/2023 at 7:03 AM, Stephen Jeffrey1709163295 said:

Hi Joe,

Did you get anywhere with the PsIn non-paged pool memory leak? I'm getting the same symptoms on a Windows 10 VDI image and any clues would be much appreciated.

Thanks,

Steve.

I did; it was Session Recording. There was a bug that took forever to get fixed, but session recording was causing the heavy memory usage which would eventually drive the machine out of memory and crash.

The latest version is supposed to have the fix, but I completely removed it from my environment so I can't be 100% certain.

June 4, 2022

Greetings:

I've spent some time tracking down a similar problem, might be the same -- Windows Server 2016, phsysial -- 192GB of RAM. What I found was machines started to under-perform, then slowly got worse and worse until the servers would lock up and become unresponsive.

I was able to observe that the machines were using a rather large amount of non-paged pool memory. As the Non-paged pool memory grew, the available memory shrank causing the machine to page to disk. Eventually "something" would consume 100% of the RAM and -- lockup.

Using Poolmon.exe, I was able to identify the tag as PsIn. I've sent memory dumps and xperf logs off to Microsoft for Analysis. At this point in time, they believe the probelm is being caused by UPMjit.sys. This is a little bit of unfamiliar territory for me, but Microsoft mentioned that PsIn was a type of buffer where executable names were stored. Process are supposed to put data into this "pool" and then free it -- it's not supposed to stay herre very long. What they see is that the data is stored but never freed, causing the machine to leak memory and drive the machine out of memory.

If you're curious if you're seeing the same, you could try either poolmon from the Windows Driver Kit or the open source version, PoolMonx .

With PoolMonX, you can just sort by bytes and then watch it keep going up and up. When data is green, it means it went up from the last time it was polled, and red means it went down. You SHOULD see PsIn going up and down, but in my case it only ever went up.

I'm curious if this is similar behavior to what you're seeing, but its been a while so it might be too late!

May 25, 2022

Good Morning:

I have many physical servers running WEM. Each of these servers have two 10GB nics that are in a Windows team.

I've noticed that this causes duplicate devices to appear in the console. Is there a way to disable the second device from being created? I understand that it technically has a different mac, but the team will always have the same mac address.

May 23, 2022

On 5/20/2022 at 6:28 PM, Hardeep Sidhu1709163344 said:

Hi There, I wouldn't hold my breath on this one. It is a known issue with no fix. We decided to downgrade our clients from 2112 and 2109 to 2106 and have not had the issue again. 2106 is supported until the end of this year so there is time for Citrix to finally fix this bug. I did consider going to 2203 however it appears this issue has continued to through to that version as well. Another option would be 1912 CU3 which came out around the same time as 2106 and seems to be stable.

Just wanted to point out, 2106 is vulnerable to an escalation of privledges attack. You can read more about it here.

https://support.citrix.com/article/CTX319750

As far as I can tell ther eis no "good" version; you're either vulnerable, or stuff doesn't work.

Citrix Profile Management Event ID 4000 · May 7, 2022

I am also seeing this in my environment running 2016.

April 24, 2022

Hey Jon,

I remember Application Isolation Enviromnets. I was very sad when they went away and they have not returned. It was a very, very valuatble feature and the way it was implemented made it very easy to implement. I'm sad to say, there is no feature like this available in the later versions of the product.

I can suggest some alternatives that you may already be licensed for.

FSLogix, which is now owned by microsoft, has a feature called App Masking. While this isn't exact partiy to AIEs, it could help you in the scenario where applications are writing configuration files on a per user basis and multiple users would be stepping on each other. This would be more similar to File Redirection Rules back. You can take a file and redirect it somewhere else making an application THINK its writing to one location, when really it's writing somewhere else (instead of C:\Program Files\Whatever it's writing to %appdata%\Whatever). This probably isn't going to work if you need true isolation.

Another solution would be some sort of App Virtualization software. I've used Microsoft App-V for years now and have been very happy with it. It's actually built into the newer operating systems and the OS understands it. The best way for me to describe it would be Application Isolation Environments on Steroids! You could easily sequence a blank appv package and then start the executable inside this virtual environment. This should "hide" the app from other users on the same machine and isolate the process a bit more. There are lots of rules to play around with, but by default -- andything written to folders in the App-V package get virtualized on a per user basis very similar to AIEs. You can even add registry settings to virtualize as well. These settings then become part of the users profile and you can roam them like any other file and registry setting.

PRO TIP: When you start your application, you can pull it into the virtual environment created by App-V using the /appvve switch. App-V Packages have two unique GUIDs that identify them -- The Package and the Version. Each time you add rules, fules, or any changes -- the version GUID regenerates, but the Package stays the same. The OS adds this switch to all Executables and you can just run myapp.exe /appvve:<applicationguid>_<versionguid> and it will start the application inside the virtual environment for you.

A thid "option" I'm going to throw out is one that isn't going to work on it's own. Microsoft has a very powerful set of tools that allow you to create what they call Shims. These Shims are able to fake the application into behaving differently. If you look at the "compatibility" settings on an executable or shortcut, you'll see options like running it for a different OS, different color modes, etc. These are just shims, and only a very, very small subset of them. Shims can get complicated to create, but at a very high level you use the editor to create a database file. The database file is then deployed to the machine and the shims become active.

PRO TIP: These database files are really just registry settings (for all the shims I've used at least). When you install the database file, it just sets a registry key. I've taken to skipping the database file and applying the shims directly to the registry via GPO, WEM, or directly in my images/installs. I find adding processes to registry keys that I already know about to be alot easier than using the editor, creating a database, deploying the database, and then going back and editing said database every time I want to make a change (and then redeploying it to all machines).

I hope this gets you going on some of the alternatives. If you end up finding something different, please do share! AIEs were an amazing tool almost ahead of their time. I absolutely love this type of technology! Feel free to reach out if you want to chat more about any of the options above.

February 28, 2022

FSLogix stuck profiles trying to log in · February 4, 2022

Here is the community discussion regarding CTX338807. If you feel this is your issue, you may find some useful information there.

https://discussions.citrix.com/topic/393608-logonuiexe-sessions-disconnected

Profile Management - General · January 31, 2022

Take a peek at this thread:

https://discussions.citrix.com/topic/393608-logonuiexe-sessions-disconnected/?

CU2 and CU4 have a bug (as well as any newer version, at least up to 2112). In talking with citrix, it appears to be a pretty hot topic right now hitting lots of people. Seems like November Updates caused the bug to be more prevalant, and it's just gotten worse since. They've confirmed that they're seeing it with lots of customers using FSLogix, but they don't quite have an answer yet ( https://support.citrix.com/article/CTX338807 ).

I'm not 100% if this is your issue, but putting a script in to kill the logonui process has partially resolved my issue. It's the first day in a month Ithink I will sleep well!

January 31, 2022

Just going to put this out here: https://support.microsoft.com/en-gb/topic/january-25-2022-kb5009608-os-build-20348-502-preview-54285445-4a79-4bc1-9c9c-fbe7299e1c29

Addresses an issue that causes the Remote Desktop Services (RDS) server to become unstable when the number of connected users exceeds 100. As a result, users cannot access published applications using RDS on Windows Server 2019
Addresses issue that occurs in remote desktop or remote applications integrated locally (RAIL) scenarios. A window might not appear because an application has used WM_SETREDRAW to temporarily stop the window from being redrawn.

I was running into a scenario where these ghost sessions were driving up my user density. I may or may not have been running into this, but it appears there is a patch out there for what I'd consider a pretty nasty bug. This doesn't come down with Windows update, so even if you're checking for oob fixes, you won't see it unless you're looking directly for it.

I'm waiting on the fix from Citrix. Apparnetly they have identified 3 scenarios that cause this problem, and can't give me the fix until they've confirmed (via CDF Traces) that this is the issue in question. It will most likely be another day or two before I get an actual fix, which leaves me with a rather unstable environment. I've taken the advise of others here and implemented a scheduled task that runs every 5 minutes and it purges any logonui process thats older than 5 minutes. That alone has given me quite a bit of relief, and I would recommend that to anyone having problems, at least until you can get something (ANYTHING!) from Citrix directly.

This is the script I used -- might be a little bloated but I wanted some feedback so I could run it in a remote powershell session and see whats going on.

$processName = "logonui"
$threshold   = 1
$processes = get-process $processName -ErrorAction SilentlyContinue    

Write-Host "Found $($processes.count) Processes that match $($processName)"
ForEach ($process in $processes) {    
    $age = $(get-date) - $process.StartTime
    write-host "Found Process ($($process.id)) with age of $($age) Minutes"

    If ($age.TotalMinutes -gt $threshold) { 
        Write-Host "  Process $($ProcessName) ($($process.ID)) with Age of $($age.TotalMinutes) will be terminated"
        $process | stop-process -Force 
    } Else {
        Write-Host "  Process $($ProcessName) ($($process.ID)) with Age of $($age.TotalMinutes) will be ignored"
    }
}

I saved this to a file where users couldn't write, then ran the following script to register the task:

$taskName  = "Clean Ghost Users"
# Hack to get a repetitive schedule
    $t1 = New-ScheduledTaskTrigger -Daily -At 01:00
    $t2 = New-ScheduledTaskTrigger -Once  -At 01:00 -RepetitionInterval (New-TimeSpan -Minutes 5) -RepetitionDuration (New-TimeSpan -Hours 23 -Minutes 55)
    $t1.Repetition = $t2.Repetition
    
# Create the new Task
    $action   = New-ScheduledTaskAction -Execute "Powershell" -Argument "-executionpolicy bypass -file ""<full path to script file>"""          
    $settings = New-ScheduledTaskSettingsSet -Hidden -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -RunOnlyIfNetworkAvailable
    $ST       = New-ScheduledTask -Action $action -Trigger $t1 -Settings $settings
    Register-ScheduledTask $taskName -InputObject $ST -User "System"

This was all taken from this post, but I just wanted to share what I found to work as a full solution in hopes it might save some time for someone. These scripts are designed to be able to execute from a remote powershell session, so you MIGHT be able to enter-pssession to a machine with the problem and install the bandaid without rebooting.

Please test before you implement!

January 31, 2022

I've had a case open with Citrix with this issue since Jan. 11th.

Server 2022, working fine with 2109. Upgraded to 2122 and experienced the behavior you've described. After rolling back to 2109, things still do not work -- complete with VDA cleanup.

It's been a rough few months of Windows Patches triggering "bugs" in other applications, so who knows whats going on.

Profile Management - General · January 21, 2022

Greetings!

I am going to closely monitor this thread, as I'm fighting something similar.

My environment is all physical machines. I'm running Windows server 2016 and I'm seeing random lockups. I can't do anything with the machine once "it" happens. I have fslogix installed, but have removed it for testing and I still have "the problem".

My case has moved from Citrix to Microsoft, and I'm kind of in a waiting state. Microsoft has made mention of a bug in NTFS pointing fingers at a filter driver which defiantely could include FS Logix. Only problem is -- I've removed FSLogix and I still have the problem.

This problem originally started in my test environment where I'm building out some 2022 servers to migrate too. They can sit all day long with no users and be fine, but once I put users on them -- anywhere from 30 minutes to 48 hours -- the machine just freezes.

The 2016 servers are running 1912 CU4 (updated from CU2 in hopes it would resolve the problem). The 2022 servers are running 2109, haven't been able to test 2112 yet.

I'm currently removing windows updates back to November to see if I can get some stability back. If I find something that works, I'll chime in.

October 28, 2021

The fix I have is actually for 1912LTSR, CU2 and I've been told to keep using it. However, according to the patch notes, many changes have gone in that impact this DLL and I'm starting to think maybe it's causing more problems than it's fixing...

I'm wondering if making a symlink for C:\Program Files (x86)|Citrix\System32 pointing to C:\Program Files (x86)\Citrix\HDX\Bin. If something is really looking in that old folder, it might be a nice band-aid until it gets permanently fixed...

October 20, 2021

It appears that the system32 folder is finally being relocated. I found the DLLs I'm looking for in C:\Program Files (x86)\Citrix\HDX\Bin ...

October 19, 2021

Greetings:

I have an ongoing issue open with Citrix that was due to a problem with WFSHELL crashing. The fix is a new version of CTXUIMON.DLL.

I have recently started to test VDA 2109 and found that this DLL no longer is installed. I was wondering if someone else could confirm -- it used to be installed to C:\Program Files (x86)\Citrix\System32. I'm struggling with some problems that have similar symptoms (vda locks up, apps not opening - general weirdness) and it caused me to notice the dll was missing.

March 5, 2021

Found the problem with your help... you were spot on.

Our wonder HPUX LDAP service can't speak anything higher than TLS1, so I built a new profile that allowed this and the best of the weak cyphers it can use and added it to my LDAP vserver. All is good now!

Looks like its time to upgrade that LDAP service, too!

March 4, 2021

You're spot on with your recap --

As far as I can tell, everything is working 100% EXCEPT the HPUX authentication. It fails when it attempts to bind with the most generic of generic messages.

I've tried a handful of different version, and I don't have any specific versions handy right now. I do feel comfortable saying any version of 12 had the same problems. I've been trying to get this upgraded for a long time, but it's been pretty low priority. I'll take a peek at the ciphers and see if anything is different. I currently have the vserver running on 11 and 13, so I'll be able to visually compare....

Thanks for the tips!

March 4, 2021

Greetings:

I have a VPX running firmware 11.0.69.12. It's hosting an vserver that is load balancing LDAP. It's a pretty simple configuration really. It's currently load balancing the following ports to a domain controller:

88 - TCP

88 - UDP

389 - TCP

445 - TCP

464 - TCP

636 - SSL_TCP

3268 - TCP

3269 - SSL_TCP

I know that all of these are not necessary, but the vserver is currently online and running just fine. However, if I upgraded to version 12 or 13, authentication fails on our HPUX server. Everything else seems to work fine (access from Windows Applications, manual tests with ldp). If I downgrade the firmware to v11, everything starts working again.

I'm really scratching my head on this -- anyone aware of any changes in 12 that might impact load balancing LDAP?

UPDATE:

The LDAP Client was unable to use anything higher than TLS1. Built a new profile allowing TLS1 and the best of the worst Cyphers to be used, and applied that only to my LDAP Vserver.

Thanks for the help!

December 30, 2020

Greetings:

At one point, I was told that this setting just created a firewall rule for the ports it used. It's controlled via policy, so you could grab a vda and apply the policy to it and start testing!

Please don't take this as an answer -- someone else may know for sure. If it's more than just a firewall rule, then you may be able to find the correct MSI from the installation folder and apply it to the VDA. Most of the components are rather obvious once you extract the installer into the individual MSIs.

December 23, 2020

Nevermind - you tried it already.

December 8, 2020

I'm battling this problem as well. I've applied the kb4594442, which is the correct OOB Update for the version of Server 2019 I'm running, but a day later (after log on/log off) users are still receiving the error. December patches are available now (or soon), so hopefully there is something in the notes to fix that.

Posted November 12, 2020 · November 12, 2020

Greetings, and welcome to 7! It's a fun adventure, for sure!

If I understand you right, you have a delivery group hosting all your applications, but some of those servers have something else installed you want to publish -- maybe to just one of those servers. The feature that you want to look at is Application Groups.

Start by firing up Studio. Hop over to the Search node and find your servers that are hosting this Application. Right click on the server and choose Manage Tags. You'll want to create a new one that describes this app, or what makes these servers unique.

Now hop over to the Applications node of the console, you'll see Application Folders and Application groups. If you right click in the white space under Application Groups, you can create a new one with the Create Application Group option.

On the Delivery Groups tab, you'll want to select the delivery group that contains these machines. While you're here, look up top at that section you probably missed that allows you to Restrict Launches to machines with Tag. Enable this and drop down the tags listbox and choose that tag you made a minute ago. You can grant users access to users and add applications to launch in this app group while you're here, too. If you haven't published the app yet -- no worries. You don't have to do it here.

Take a peek at the properties of an app now -- and you'll see that you can publish it to a Delivery Group or an Application Group. To publish apps to these servers, now you just use the Application Group instead of the Delivery Group and you'll get the effect that you're looking for. Your app will only launch from the servers you scoped in the Application Group.

July 8, 2020

Try going into the Advanced settings of the client and disabling EDT.

June 29, 2020

Good Afternoon:

Is there an official SCUP or Third Party Updates catalog available for Workspace App? If not, any chance our Citrix friends could get this on someones radar?

I'd love to treat Workspace as just another update and let it happen monthly with my Microsoft Updates. The framework is built into SCCM these days and it would be a great addition if its not available already!!

June 29, 2020

I've been struggling to get Workspace App to update as well... not even when content is assigned via group. Just adding the app to the user's AD account. In a browser, everything works fine, but in the workspace app, it seems like it's reading too much from the cache and isn't updating.

Forums

Articles

Labs

Videos

TechZone

Citrix Community Articles

Events

Profiles

Posts posted by Joe Robinson