Jump to content
Welcome to our new Citrix community!
  • 0

Citrix WEM 2305 - Service Preventing Logins


Greg Beck

Question

Recently we were doing maintenance on our environment but nothing Citrix specific. I thought it would be easier, so I just shut down the SQL server that hosts the WEM database, thinking the cache would handle the settings while we were doing our work. We didn't notice it right away but maybe 20 to 30 minutes later we were getting alerts about users not being able to sign into Citrix.  The sessions were getting stuck at waiting for the WEM service while the session was building, then the session would timeout and close.  To get things going I started the SQL server back up.  

 

We are on all prem and pretty simple setup. Only the SQL server was shutdown.  

WEM SQL -> WEM App -> VDA*

 

I haven't been able to recreate the issue yet. I didn't have WEM debug logging enabled at the time. I am not finding helpful logs either.  The interesting messages I found in the event log are below. 

 

Searching I did find this older discussion thread: https://discussions.citrix.com/topic/403982-wem-1906-citrix-wem-user-logon-service-adds-5-seconds-to-logon-time

 

  • The thread suggests that the service is only used handling group policy applied via WEM. We aren't applying any Group Policy Settings via WEM.  Is it still save to set the service to manual? 
  • Is there something I am missing? Shouldn't WEM be ok if the SQL server is not available for some reason? I was under the impression that since everything was cached I didn't need a highly redundant SQL server but maybe I need to look at that. 

 

 

 

Event logs: 

I thought it was interesting.  I see the service start processing, then timeout after a minute, then the user gets logged off. 

Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:03:20 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Start logon processing for user EXAMPLE\user.


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:03:20 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Start processing policies for user EXAMPLE\user


Log Name:      Application
Source:        Microsoft-Windows-Winlogon
Date:          9/19/2023 1:04:20 AM
Event ID:      6005
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
The winlogon notification subscriber <WemLogonSvc> is taking long time to handle the notification event (Logon).


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:20 AM
Event ID:      0
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Processing policies for user EXAMPLE\user takes too much time.


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:20 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Finished logon processing for user EXAMPLE\user.


Log Name:      Application
Source:        Microsoft-Windows-Winlogon
Date:          9/19/2023 1:04:20 AM
Event ID:      6006
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
The winlogon notification subscriber <WemLogonSvc> took 60 second(s) to handle the notification event (Logon).


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:21 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Start logoff processing for user EXAMPLE\user.


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:21 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Start processing logoff for user EXAMPLE\user, sid: ####


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:22 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Succeeded to process logoff script for user EXAMPLE\user.


Log Name:      Application
Source:        Citrix WEM User Logon Service
Date:          9/19/2023 1:04:22 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
Finished logoff processing for user EXAMPLE\user.


Log Name:      Application
Source:        Citrix Desktop Service
Date:          9/19/2023 1:04:22 AM
Event ID:      1030
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server1.example.com
Description:
The Citrix Desktop Service detected that a user session has ended. Session ##### for user '' has ended; reason code Logoff.

 

Link to comment

12 answers to this question

Recommended Posts

  • 0

Yeah that doesn’t sound right at all. That service should be self regulating these days, it had challenges way back when it was only used for CEM stuff

 

Can you post your settings for your offline processing here? (Enable offline mode etc) 

 

will see if we can get some eyes from those in the know on this post 

Link to comment
  • 0

We don't use UPM at all.  For the server I grabbed the event logs from it is using Drive actions (map a network drive), Registry Entries, External Tasks, and Folder and Files (to copy files around).

 

If it makes a difference the server above is publishing Apps.  Server 2016.   

 

For all of our configuration sets we have the agent options to configured to "Enable Offline Mode" and "Use Cache to Accelerate Actions Processing" enabled.  Attached a screenshot of the configuration screen from the server above. 

 

wem-agent-options.png

Link to comment
  • 0
13 hours ago, Greg Beck said:

We are using BIS-F on the non-persistent servers which is running RefreshCache when the machine starts up too.  

By default BIS-F removes the cache when you seal the image, any chance the problem users were landing on a rebooted VM that might not have had any form of cache?

Link to comment
  • 0
11 hours ago, James Kindon said:

By default BIS-F removes the cache when you seal the image, any chance the problem users were landing on a rebooted VM that might not have had any form of cache?

 

Everything is published apps. All the VDA would have been running for at least 24 hours at the point of our maintenance window.  

Link to comment
  • 0

I am able to recreate the problem now.  I created two new VMs.  One for SQL and one for infrastructure services.  I restored the existing database to my test database server and configured the infrastructure services to use that. I have a test VDA which I pointed at the new infrastructure server.  

 

After verifying things seems to be working with the test services, I turned off the SQL server.  Right after shutting down the SQL server, I did some test login, and everything was working as I expected. I signed off and let things sit.  I tried again about 30 minutes later.  Now it is getting stuck at the "Please wait for the Citrix WEM User Login Service".  

 

Before shutting down the SQL server the one change I did make was enabling "Advanced Settings -> Service Options -> Enable Debug Mode".  I verified the "C:\Program Files (x86)\Citrix\Workspace Environment Management Agent\Citrix WEM Agent Host Service Debug.log" was logging.   

 

Now that it is broke, the "Citrix WEM Agent Host Service Debug.log" is getting anything written. 

 

The troubleshooting guide has a section for Windows Communication Foundation traces which I haven't tried but I am not of time for today and out tomorrow. 

 

Link to comment
  • 0

To close the loop with this.  I have been working with support on the issue.  

 

The final answer is this is the expected behavior.  The infrastructure server requires that the SQL database is available.  Their suggestion is to stop the infrastructure service on the server.  That will get the agents to switch over to using their cache.

 

Or put the database in a highly available SQL server.

Link to comment
  • 0
On 12/14/2023 at 7:33 AM, Greg Beck said:

To close the loop with this.  I have been working with support on the issue.  

 

The final answer is this is the expected behavior.  The infrastructure server requires that the SQL database is available.  Their suggestion is to stop the infrastructure service on the server.  That will get the agents to switch over to using their cache.

 

Or put the database in a highly available SQL server.

This does not sound right at all. There is a Cache on the Broker that is supposed to sustain the loss of SQL connectivity. The product team have eyes on this, but will need more info, I will DM you

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...