UPM not loading properly on servers

[Pandiarajan Kasinathan1709163297]

We have XenApp 7.19 LTSR CU3 on Windows 2016 server with Citrix UPM. Recently we are seeing odd behavior's like when user launches any application , in workspace window it loads till loading Citrix user profile and goes off . At times it loads fully till launch application as shown in attached screenshot and nothing happens. Sometimes it takes 3-5 minutes to load application. But if i stop Citrix profile management service on VDA servers when we encounter this issue, it launches the application fine. UPM is hosted in NAS filer. 

 

Also when I check the system profiles list under system properties as shown in screenshot below. I see mostly local profiles not roaming profiles.

 

 

was it because of timeout issue between Citrix UPM & VDA server?

Can someone help me to identify why its happening like this?

[Ruben Centeno1709155520]

You have Windows Roaming Profile configured, too? UPM is intended as a replacement for Roaming Profiles. Use just one or the other. Also, when using UPM, Windows will always see it as a local profile; this is normal.

[CarlStalhood]

What do you see in the UPM logs? Default location is C:\Windows\System32\LogFiles\User Profile Manager\pm.log.

[Pandiarajan Kasinathan1709163297]

Thanks Carl for your response.

 

I am attaching most recent log, When i launched an application it took almost 5 minutes to launch it. Here is the log for that

2022-03-17;13:17:17.716;INFORMATION;CORP;z027996;64;6908;SessionCount::RealTimeCount - User: z027996, Domain: CORP, Session Count: 0.
2022-03-17;13:17:33.310;INFORMATION;CORP;z027996;65;6908;XenApp Optimization, enabled: 0, definition path: 
2022-03-17;13:17:33.310;INFORMATION;CORP;z027996;65;6908;SessionCount::RealTimeCount - User: z027996, Domain: CORP, Session Count: 0.
2022-03-17;13:17:33.326;WARNING;CORP;z027996;65;6908;QueryLocalProfile: Deleting stale ProfileList entry for user with SID <S-1-5-21-1608400361-1401040151-28319967-3026511> pointing to <c:\users\z027996>.
2022-03-17;13:17:34.200;;CORP;z027996;65;6908;Copied NTUSER.DAT from \\tec01.na0vlb\data2\upmprofile\z027996\UPM_Profile\NTUSER.DAT to C:\Users\z027996\NTUSER.DATThe operation completed successfully.
2022-03-17;13:17:34.200;INFORMATION;CORP;z027996;65;6908;Now start to copy from\\tec01.na0vlb\data2\upmprofile\z027996\UPM_ProfileTo C:\Users\z027996
2022-03-17;13:17:35.919;INFORMATION;CORP;z027996;65;6908;Finished copying from \\tec01.na0vlb\data2\upmprofile\z027996\UPM_Profile To C:\Users\z027996 , error code: 0
2022-03-17;13:22:01.196;ERROR;CORP;ay31465;62;3092;CJitThreadInfo::SaveChangedFiles: failed to copy 'AppData\Local\Microsoft\Windows\WebCache\V01.log' to the userstore. Error 0x57: The parameter is incorrect.
2022-03-17;13:22:01.212;ERROR;CORP;ay31465;62;3092;CJitThreadInfo::SaveChangedFiles: failed to copy 'AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat' to the userstore. Error 0x57: The parameter is incorrect.

[Pandiarajan Kasinathan1709163297]

2022-03-17;14:51:48.110;ERROR;CORP;i020467;334;24336;Open registry file c:\users\i020467\NTUSER.DAT failed with nError:32
2022-03-17;14:51:48.110;ERROR;CORP;i020467;334;24336;CRegistryHive::Load: Check signature of <c:\users\i020467\NTUSER.DAT> to <upm_S-1-5-21-1608400361-1401040151-28319967-985295_logoff> failed. The hive file is probably corrupt. Message: The process cannot access the file because it is being used by another process.
2022-03-17;14:51:48.110;ERROR;CORP;i020467;334;24336;CRegistryHive::Load: RegLoadKey of <c:\users\i020467\NTUSER.DAT> to <upm_S-1-5-21-1608400361-1401040151-28319967-985295_logoff> failed with: The process cannot access the file because it is being used by another process.
2022-03-17;14:51:48.110;ERROR;CORP;i020467;334;24336;ProcessLogoff: Failed to open registry file NTUSER.DAT for cross-platform processing during logoff. Error 0x20: The process cannot access the file because it is being used by another process.

[Ruben Centeno1709155520]

To me this screams anti-virus. UPM performance suffers heavily without the proper anti-virus exclusions in place. Also make sure you have AppData\Local\Microsoft\Windows\WebCache  and AppData\Local\Microsoft\Windows\Cache listed as a folders to mirror.

 

https://docs.citrix.com/en-us/tech-zone/build/tech-papers/antivirus-best-practices.html#antivirus-exclusions

 

[Pandiarajan Kasinathan1709163297]

Thanks @Ruben Centeno

 

Antivirus exclusions looks good.

 

I could see central profile configured in different path for users in their AD object. Will it conflict with Citrix UPM store & block it from launching?

[Pandiarajan Kasinathan1709163297]

Sorry for responding very late.

 

I have removed roaming profile from User's AD object. Now we have only UPM, Still it fails to load the application or else takes almost 5 minutes to launch the application.

 

UPM Log files entries show no issues, Pls see the logs below. 

2022-04-05;13:08:19.262;INFORMATION;CORP;z027996;10;8728;XenApp Optimization, enabled: 0, definition path: 
2022-04-05;13:08:19.268;INFORMATION;CORP;z027996;10;8728;SessionCount::RealTimeCount - User: z027996, Domain: CORP, Session Count: 0.
2022-04-05;13:08:19.300;WARNING;CORP;z027996;10;8728;QueryLocalProfile: Deleting stale ProfileList entry for user with SID <S-1-5-21-1608400361-1401040151-28319967-3026511> pointing to <c:\users\z027996>.
2022-04-05;13:08:20.695;;CORP;z027996;10;8728;Copied NTUSER.DAT from \\tec01\data2\upmprofile\z027996\UPM_Profile\NTUSER.DAT to C:\Users\z027996\NTUSER.DATThe operation completed successfully.
2022-04-05;13:08:20.695;INFORMATION;CORP;z027996;10;8728;Now start to copy from\\tec01.\data2\upmprofile\z027996\UPM_ProfileTo C:\Users\z027996
2022-04-05;13:08:21.402;INFORMATION;CORP;z027996;10;8728;Finished copying from \\tec01\data2\upmprofile\z027996\UPM_Profile To C:\Users\z027996 , error code: 0
 

[Pandiarajan Kasinathan1709163297]

On 3/18/2022 at 6:32 PM, Ruben Centeno1709155520 said:

You have Windows Roaming Profile configured, too? UPM is intended as a replacement for Roaming Profiles. Use just one or the other. Also, when using UPM, Windows will always see it as a local profile; this is normal.

Sorry for very late response.

As you rightly pointed out, Issue was the antivirus exclusion. Found the affected servers were out of antivirus exclusion list. I was in an impression that these servers were part of it, because when a new Citrix server is deployed in our organization, it will be automatically put in Citrix container in AV. 

 

Thank you