XenApp slow logon times, user get black screen for 20 seconds.

[Andy White1709154166]

Hello,

 

We use XenApp 7.6 with Windows 2012R2 on VMware 5.5.

 

Logon times have become slow recently for users, they get a black screen just before their XenApp desktop shows up for about 20 seconds.  How do we find out the cause of this?  Most profiles are under 100mb.

 

If I turn all the GPOs off apart from the UPM GPO it's still slow, if I turn it off logons are fast.

 

The first user logon on the VDAs everyday is always slow (we reboot every night) even with the Citrix Profile Manger service off and all GPOs.

 

Thanks

[Sascha Matter1709156668]

No, published apps are working pretty fast. You also have ClassicShell installed on your W2K12 desktops?

 

I noticed that my MCS based desktops dont have this massive black screen phase, like the PVS based desktop. Both desktops are in the same AD OU, so they use the same GPO's.

 

But thanks for the hint about the Shell-Core log, I will check that later.

 

Edit: I just checked it, the regkey was added with the ActiveSetup, saw it in the shell core log. I also checked how long it takes until the first LogonTasks were started and when the "AppResolver finish" log info was created. It took about 47s for those tasks and I saw a lot of entries about "shortcut blablabla added to app resolver cache."

[Mark DePalma1709157107]

No, published apps are working pretty fast. You also have ClassicShell installed on your W2K12 desktops?

 

I noticed that my MCS based desktops dont have this massive black screen phase, like the PVS based desktop. Both desktops are in the same AD OU, so they use the same GPO's.

 

But thanks for the hint about the Shell-Core log, I will check that later.

 

Edit: I just checked it, the regkey was added with the ActiveSetup, saw it in the shell core log. I also checked how long it takes until the first LogonTasks were started and when the "AppResolver finish" log info was created. It took about 47s for those tasks and I saw a lot of entries about "shortcut blablabla added to app resolver cache."

 

I use Classic Shell for my Win2k12R2 VDAs. If I eventually move to 2016 I'll go back to native, but the Windows 8 style menu is unacceptable for users coming from a Win7/Win10 world.

 

If that is taking 47 seconds then most likely the registry value isn't taking effect. I do notice that AppResolver runs more than once (after shell has loaded and the person is logged on), so you have to be sure that you're looking at the first instance of AppResolver associated with that specific logon. You should be looking at the time between these:

 

-Start-

Event ID 62170 - Logon task 'AppResolver' started with flags 1034.

Event ID 28125 - Starting to refresh app resolver cache for scenario 2 with flags 79.

........

-Stop-

Event ID 28018 - AppResolver Scan Stopped.

 

You have to be sure you are looking the first instance for that logon session. Also look at excluding 'AppData\Local\Microsoft\Windows\Caches' from UPM and wiping from an existing user.

[Sascha Matter1709156668]

@Christopher Yue

 

Are your vm's PVS or MCS based? Cause I can confirm that with the second login. The first login takes about 20-30s but the second login is below 10s.

[Christopher Yue]

@Christopher Yue

 

Are your vm's PVS or MCS based? Cause I can confirm that with the second login. The first login takes about 20-30s but the second login is below 10s.

MCS.

 

Local profiles are deleted on logoff after a 1 minute delay.

 

If a user logs off and back onto the same server before the local profile is deleted, then the login is quick without the black screen.

 

The next thing I might try is disabling Classic Shell and return back to the native Start Screen and test again.

[Mark DePalma1709157107]

@Christopher Yue

 

Are your vm's PVS or MCS based? Cause I can confirm that with the second login. The first login takes about 20-30s but the second login is below 10s.

 

 

MCS.

 

Local profiles are deleted on logoff after a 1 minute delay.

 

If a user logs off and back onto the same server before the local profile is deleted, then the login is quick without the black screen.

 

The next thing I might try is disabling Classic Shell and return back to the native Start Screen and test again.

 

Could you both try this? I suspect that the ActiveSetup registry values I gave you aren't actually executing properly for some reason. This is what I initially did before creating the ActiveSetup fix to test my theory...

 

1.) Determine the AD SID of the user

2.) Isolate the user so that they launch the desktop only on one server and make sure there are no current profiles for the user on this server

3.) Ensure the 'Remote Registry' service is enabled and started on the server

4.) Launch PowerShell from another computer that has admin privileges on the affected server

5.) Run the loop below in PowerShell:

 

While ($true) {

reg query \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache

reg add \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache /t REG_DWORD /d 0 /f

Write-Host ""

Start-Sleep -Milliseconds 100

}

 

6.) Launch the published desktop as the affected user.

 

 

This will basically go in a fast loop querying and changing the 'ResetCache' value back to 0. It will give output on what the value was before changing it. The new value of '0' should register before Explorer loads and reads it avoiding the 'AppResolver Scan'. If this speeds up all logons for the user, then we know ActiveSetup wasn't working for you and that can be diagnosed separately.

[Christopher Yue]

Hi Mark,

 

Will give this a look when I get a moment and post back if I have any tangible results.

[Balaji Baskaran1709154564]

Mark,

 

I had the exact issue of 47 seconds (xenapp vda 7.11/12/13 on W2k12) of overall log on time with 20 seconds of black screen. Tried your powershell script to execute from a remote computer to reset the cache, the black screen was now down to 2-3 seconds instead of 15-20 seconds. What's next to fix this permanently?

 

Regards,

Balaji

[Balaji Baskaran1709154564]

Ran procmon, userprofilemanager.exe is setting the cache value to 4 & finally explorer reset it to 0.

[Mark DePalma1709157107]

Mark,

 

I had the exact issue of 47 seconds (xenapp vda 7.11/12/13 on W2k12) of overall log on time with 20 seconds of black screen. Tried your powershell script to execute from a remote computer to reset the cache, the black screen was now down to 2-3 seconds instead of 15-20 seconds. What's next to fix this permanently?

 

Regards,

Balaji

 

Great to hear this is working for other people. The temporary fix is to put in my Active Setup registry patch. I'm going to try to reach out to Citrix through alternate channels to get some higher tiers to look at this.

 

 

Ran procmon, userprofilemanager.exe is setting the cache value to 4 & finally explorer reset it to 0.

 

That is what happens, yes.

[Christopher Yue]

Mark,

 

I have made the changes as suggested and have seen a slight improvement.

 

Now for the weird stuff.

 

Using the same set of GPOs I built another Machine Catalog from the same Gold Image I use for my production site except this Catalog only has a single server (the production has 20).

 

The black screen phase does not appear in the MC with a single server only the MC with multiple servers in.

[Balaji Baskaran1709154564]

Mark,

 

I have made the changes as suggested and have seen a slight improvement.

 

Now for the weird stuff.

 

Using the same set of GPOs I built another Machine Catalog from the same Gold Image I use for my production site except this Catalog only has a single server (the production has 20).

 

The black screen phase does not appear in the MC with a single server only the MC with multiple servers in.

 

Did you verify the policies are applied properly from HKLM\software\policies\citrix\userprofilemanager?

[Mark DePalma1709157107]

Mark,

 

I have made the changes as suggested and have seen a slight improvement.

 

Now for the weird stuff.

 

Using the same set of GPOs I built another Machine Catalog from the same Gold Image I use for my production site except this Catalog only has a single server (the production has 20).

 

The black screen phase does not appear in the MC with a single server only the MC with multiple servers in.

 

 

Did you verify the policies are applied properly from HKLM\software\policies\citrix\userprofilemanager?

 

FYI, I have opened up a support case through alternative channels to get more eyes on this issue. It looks like it is going to move along quickly.

 

That is weird that the catalog from the same image is faster. Definitely check that the exact same set of GPOs are applying.

[Pierre Boisvert1709157167]

i'm trying alternate method of changing permission on the "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore" key.

As now seem much faster. I Will continue my tests.

[Mark DePalma1709157107]

i'm trying alternate method of changing permission on the "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore" key.

As now seem much faster. I Will continue my tests.

Seems like that would accomplish the same thing, but I wouldn't want to restrict the OS from accessing or writing to the whole key. Not sure what other implications there are for that. Preventing just a change to the one value is the most direct workaround.

[Pierre Boisvert1709157167]

Seems like that would accomplish the same thing, but I wouldn't want to restrict the OS from accessing or writing to the whole key. Not sure what other implications there are for that. Preventing just a change to the one value is the most direct workaround.

I tried with the powershell script, and it worked, but with the regfile it doesn't, so tried something else. Also tried on windows 8 pc and i saw improvement in logon. We are using classic shell on our desktop also.

[Balaji Baskaran1709154564]

i'm trying alternate method of changing permission on the "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore" key.

As now seem much faster. I Will continue my tests.

 

How to do this for all users during logon? 

[Christopher Yue]

FYI, I have opened up a support case through alternative channels to get more eyes on this issue. It looks like it is going to move along quickly.

 

That is weird that the catalog from the same image is faster. Definitely check that the exact same set of GPOs are applying.

 

Yes the GPOs are identical.

 

In fact I turned them all off and the behaviour is the same.

 

It seems for sure when the Machine Catalog only contains a single server, I do not get the black screen.

 

Be very interested in the outcome from Citrix support.

[Joe Robinson]

I had a case open with Citrix regarding this issue.  They chalked it up to something environmental.  I'll share what I found during my investigations.

 

Prior to this issue, I had a case open regarding 2012 R2 machines hanging during logon (Mostly on the Welcome screen).  This problem turned out to be a deadlock with UPM where it was looking for some file and it would never give up.  I was told that the solution was to put a timeout on the command they were calling which caused UPM to give up and move on.  When I received the private, this issue was fixed, but I now had the 15-20 second delay when logging in where it would hang at the black screen.

 

We spawned that off into a new case and the fix for the original problem was put into UPM with Xen 7.11.

 

While investigating this new problem, I found that UPM would stop writing log files during this time.  I used a program called CMTrace to monitor the UPM log remotely.  CMTrace allows you to view text in a text file as it's written to the file.  As the user would start the login process, text would be written like crazy to the log.  The second the black screen showed up, no text was written to the log.  Once the black screen went away, the logs would "catch up".  

[Balaji Baskaran1709154564]

I have a Citrix support ticket for this for more than 2 weeks now. They tried everything but nothing fixed the problem. Took CDF & other proc mon logs. I also showed them how changing the value of registry using script is fixing the problem. Ticket is now escalated to other tiers.

[Mark DePalma1709157107]

I have a Citrix support ticket for this for more than 2 weeks now. They tried everything but nothing fixed the problem. Took CDF & other proc mon logs. I also showed them how changing the value of registry using script is fixing the problem. Ticket is now escalated to other tiers.

 

Good tohear. I think they'll be on this very soon. I had this escalated very quickly to UPM engineers on Wednesday.

[Pierre Boisvert1709157167]

I tried with the powershell script, and it worked, but with the regfile it doesn't, so tried something else. Also tried on windows 8 pc and i saw improvement in logon. We are using classic shell on our desktop also.

Thanks Mark DePalma , finally found that GPO were blocking REG.exe and find a way to write to reg of user with vbs. Now it work very fast. 

[Christopher Yue]

OK, I thought I would summarize my environment again and list out all the suggested changes to make sure I have not missed anything out:-

 

Environment

XenApp 7.11

Publishing Server 2012R2 Desktops

Using Classic Shell

 

Config Changes I have applied

On gold image server applied the following registry fix:-

 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\DisableUPMResetCache]

@="DisableUPMResetCache"

"Version"="1,1,1,1"

"StubPath"="REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache /t REG_DWORD /d 0 /f"

"Locale"="*"

 

VIA GPO, I have applied the suggested changes via GPO (see attached).

 

With the various changes via the GPO I did not think I needed to apply the following Powershell script Mark has referred to, namely:-

 

While ($true) {

reg query \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache

reg add \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache /t REG_DWORD /d 0 /f

Write-Host ""

Start-Sleep -Milliseconds 100

}

 

 

If I still need this, what is the best way to automate the distribution?

 

[Mark DePalma1709157107]

 

OK, I thought I would summarize my environment again and list out all the suggested changes to make sure I have not missed anything out:-

 

Environment

XenApp 7.11

Publishing Server 2012R2 Desktops

Using Classic Shell

 

Config Changes I have applied

On gold image server applied the following registry fix:-

 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\DisableUPMResetCache]

@="DisableUPMResetCache"

"Version"="1,1,1,1"

"StubPath"="REG ADD HKCU\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache /t REG_DWORD /d 0 /f"

"Locale"="*"

 

VIA GPO, I have applied the suggested changes via GPO (see attached).

 

With the various changes via the GPO I did not think I needed to apply the following Powershell script Mark has referred to, namely:-

 

While ($true) {

reg query \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache

reg add \\#SERVERNAME#\HKU\#USERSID#\Software\Microsoft\Windows\CurrentVersion\ImmersiveShell\StateStore /v ResetCache /t REG_DWORD /d 0 /f

Write-Host ""

Start-Sleep -Milliseconds 100

}

 

 

If I still need this, what is the best way to automate the distribution?

 

I wouldn't do a DELETE on StateStore via GPO. You can never guarantee that the GPO will apply BEFORE the shell is loaded. If you already applied my 'Active Setup' changes then those take care of that. You also have to apply the #2 and #3 changes I laid out here: https://discussions.citrix.com/topic/381735-xenapp-slow-logon-times-user-get-black-screen-for-20-seconds/?p=1955157

[Christopher Yue]

Thanks for the rapid response Mark,

 

I will remove the StateStore delete via GPO just in case and have noted your other points.

What might be of interest is that I am also looking at an alternative to UPM called Profile Container (from FSLogix) and this behaviour of the long black screen appears even when UPM is not used. I thought this new solution would fix it but I was wrong.

 

Does seem to be a Server 2012R2 thing.

 

I even tested all this without Classic Shell but no difference.

 

It's enough to make me want to try Server 2016 to see if the matter resolves itself in a later OS.

[Balaji Baskaran1709154564]

For me the active setup method never worked. Powershell script worked which is only to prove the reset cache is causing the issue.