PVS target devices intermittently unresponsive

[Steffen Valentin1709160582]

Hello everybody,

 

for about 3 month now, we are facing an issue with our pvs targets.

 

The targets become unresponsive. Citrix Studio shows unregistred.

You can no longer logon from the vsphere console as ctrl + alt + del is not recognized.

vCenter does not show IP and fqdn anymore for the target device.

 

The only temporary solution is to reset the virtual machine and boot it again.

 

It seems as somehow the machines are going to sleep or hibernation. But everything like this is disabled.

 

Our environment looks like this:

 

We have 3 PVS farms containing 2 PVS servers each (Windows Server 2019). PVS Version is 2203 LTSR CU2

 

These 3 farms contain about 50 target devices in total.

 

The target device OS is Windows Server 2019.

Target device drivers we used 1912 CU1 / 2203 CU1 / 2203 CU2. All with the same issue.

Hypervisor version is VMware 7.0.3 and VMware Tools 12130

 

Not all server stop responding at the same time. Some servers seem to work fine.

The server stop responding with and without user connected to them. Often the stop in the first 30 minutes after booting.

 

We also have a call open with Citrix. But the only thing they could see from memory dumps is that the image seems not to be reachable anymore at this time.

 

We also disable all firewalls between PVS servers and targets and uninstalled anti virus software. Everthing without improvement.

 

Citrix support also wanted us to check cache partition space. But there has always been enough space free.

 

Does anyone had a similar behavior before?

Does anyone have an idea wht to check next?

 

Any idea is appreciated.

 

Thanks in advance. 

 

 

Have to add:

 

We tested with servers booting locally from the reversed image. This way the problem did not occur.

So it seems to be PVS specific.

 

[Jeff Riechers]

I have seen things like this if the writecache has become full.  Even with Windows Updates disabled windows pulls down micro updates that can fill that cache quickly.

 

What writecache method are you using, and what size writecache do you have deployed?

[Steffen Valentin1709160582]

Write cache is RAM with overflow to harddisk (4 GB Ram + 40 GB disk) 

 

Cache disk is not full. We checked it serveral times. There were at least 8GB free.

 

And the problem also occurs with no user on it and an empty cache file (4MB) 

[Jeff Riechers]

Ok, if you run a show usage on the vdisk do you see a bunch of retries on the machine?

[Steffen Valentin1709160582]

Retries are low for all servers (single digit numbers)

[Jeff Riechers]

I would look for ghost nics, and would update the drivers in the image.

 

Are the PVS server and target device in the same vlan?  I have also seen if the Server and Target device are in different vlans, then running through the default gateway can cause disconnections.

[Steffen Valentin1709160582]

PVS servers and targets are in a dedicated streaming network.

 

We will look for ghost nics and repond with the result.

[Steffen Valentin1709160582]

Ok, we have looked for ghost nics and found one.

 

We have removed the nic and now we will see if the problem is resolved.

 

Thank you for your support, so far.

[Steffen Valentin1709160582]

Unfortunately removing the ghost nic did not solve the issue ?

Several server stuck again.

[Jeff Riechers]

Are your event logs set to go to the persistent drive?  There may be some service or something dying there, possibly Windows itself having the issue instead of the PVS engine.

 

Have you applied all the best practice for non-persistent and provisioned environments?  I have GPO and WEM examples of these up on my site if you want to compare what works for me.

 

https://www.jeffriechers.com/wiki/wem-based-non-persistent-image-optimizations/

 

https://www.jeffriechers.com/wiki/gpo-configuration/

 

 

 

[Steffen Valentin1709160582]

The event logs are persistent, but do not seem to have any suspicious entry.

 

Thank you for your links. I will check it.

[Steffen Valentin1709160582]

Sorry for the late answer.

We evaluated your configs, but couldn't find any obvious different configuration in our environment.

 

Today we updated fslogix to the newest hotfix. As the version 2210 should be buggy, maybe this is a solution.

 

I will give an update

[Steffen Valentin1709160582]

Unfortunately the servers are still crashing.

[Dmitriy Kytmanov]

Hello everybody,
@Steffen Valentin, did you manage to solve this problem?
We are experiencing the same issue on one of our sites: target devices and provisioning servers are 1912 CU4.
We don't have any problem with retries or with free space on cache drives.
The only difference from your description is that our targets become unresponsive not in the first 30 minutes after booting but randomly.
Reboot always helps.

[Steffen Valentin1709160582]

Citrix and we are still investigating the issue.

Since about 10 days, servers have not been crashing anymore.

 

 

So far nobody knows why it was crashing and why it didn't crashed the last days.

[Dmitriy Kytmanov]

On 3/30/2023 at 7:16 PM, Steffen Valentin1709160582 said:

Citrix and we are still investigating the issue.

Since about 10 days, servers have not been crashing anymore.

 

 

So far nobody knows why it was crashing and why it didn't crashed the last days.

Hello, @Steffen Valentin1709160582  maybe there is new information?

[Steffen Valentin1709160582]

3 hours ago, Dmitriy Kytmanov said:

Hello, @Steffen Valentin1709160582  maybe there is new information?

"Unfortunately" the problem did not occur in the last weeks.

We still have no clue what caused it and what solved it for now.

 

Maybe any Windows Update.

[Jeff Riechers]

I ran into a similar issue at 2 different clients.

 

Check the install date of the Visual C++  2015-2022.  If it shows as being installed after your VDA, you will need to remove your VDA.  Remove Visual C++.  And then reinstall the VDA and let it install C++.

 

At both clients when they install a Microsoft Servicing Stack update it re-installed the Visual C++ components.  Even though they were the exact same version their VDA and other software didn't work correctly.  They needed to have the C++ installed first as part of the pre-requisites before their application was installed.  The reinstall after that app and VDA were installed caused them to crash, and unregister.