Jump to content
Welcome to our new Citrix community!
  • 1

Short session freeze, and packet loss towards VDA with new golden image


Ronnie Pedersen

Question

Hi, 

 

hope everyone is slowly gearing down for the holidays :)

 

But before doing so, I'm being taunted by our Xenapp environment at the moment, and could use the help of some fresh minds. 

 

Short story long: whenever I try to create a new golden image using MDT, and implement this via our PVS, I end up with a VDA that - when "pushed" - experiences short hang/freeze moments, upto 1-2 seconds. This is the same across several physical hosts. 

 

In this period the VDA doesn't even respond to ping - and users that are logged in also notice that the HDX realtime connector status displays "connected" when the screen freeze is done - kind of like the server looses all connection to the outside world. 

 

Depending on the amount of users, this can go from slightly annoying to absolutely unbearable. And it doesn't take much more than 2-3 users for this to be noticable. 

 

The MDT process is locked down pretty well, so there really isn't any difference in how the master images are prepared.

 

I've tried more of less every thinkable combination of: 

 

VDA: 7.15, 7.18, 1808

Xentools: Three different version spanning 7.1, 7.1CU and latest as of a few weeks ago

PVS: 7.15, 7.15.9

WEM: 4.2, 4.7, 1808

HDX: 2.3, 2.6

Windows 2016: Clean from ISO, and varying levels of cumulative updates. 

Antivirus: on / off

GPO's on / off

 

Client software has been tried also, Receiver 4.9, 4.11, 1808 - HDX 2.3, 2.6 - all same result. 

 

Nothing definitive seems to make a difference. Even a bone stock W2016 with the bare minimum of software needed: VDA, PVS - has the same issues. 

 

But - the image that we currently have in production is working just peacy. No issues at all. This image has been thru a couple of versions over the last six months, and so its a bit difficult for me to replicate it 100% - although, I doubt that we are faced with a situation where only the magic combination of software works. 

 

Windows Event logs don't tell any story. 

 

If I don't put any user load on the new golden image, I can still see small "spikes" via Pingplotter with 50-100ms delay that aren't there, with a fully loaded server running the healthy image. 

 

Looking at the Xenservers running the VM, the packet loss matched a high CPU load on one of the VM cores. This led us to try out a different Xentools version, but alas no change was seen. 

 

Yeah... a lot of blabla. So, basicly - I get short screen freezing on my VDA, and actual packet loss when pinging the VM at the same time. Regardless of software load. But not on VM's with another golding image, running on the exact same hosts, in the same AD OU and so on. 

 

Any feedback, even "you're a moron if you haven't tried this:" - I'll take anything :)

 

Cheers

Ronnie

Link to comment

17 answers to this question

Recommended Posts

  • 1

The mechanism of XenTools auto-update through windows-update is described in this article: https://support.citrix.com/article/CTX224933

The article mentions the URL: "https://pvupdates.vmd.citrix.com/updates.latest.tsv?_ga=2.68898006.63278142.1546854859-893697423.1495544881"

 

This file contains links to the installer of Version 7.1.0.1323:

x86: http://downloadns.citrix.com.edgesuite.net/14262/managementagentx86.msi

x64: http://downloadns.citrix.com.edgesuite.net/14263/managementagentx64.msi

 

As of the following post this should be the version which contains XENVIF 8.2.1.175 and which you are looking for:

https://discussions.citrix.com/topic/400561-vm-randomly-freeze-after-xentools-upgrade-from-65-to-76/#comment-2031100

 

I did not verify it myself yet ...

  • Like 1
Link to comment
  • 0

Thank you for the reply. 

 

The installer does not get the /enable_HDX_3d_PRO switch - and apparently it actually ignores these now, choosing the correct mode automatically. Besides confirming that this policy is disabled (default) I cant find a sure way of confirming that 3d pro is actually not running. 

 

https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/ica-policy-settings/graphics-policy-settings.html#optimize-for-3d-graphics-workload

 

 

Link to comment
  • 0

Hello sglrope,

 

We have also the same issue, currently we're building a new golden image with Server 2016.

The old golden image is working with the same hardware / network and Citrix components, and is working well.

 

When the user is working on the new image we see packet loss and extreme high latency to the VDA, also from VDA to PVS server. This is for a user a complete system freeze for one or two seconds. We can simulate this just to start an Office application or doing some Internet browsing.

 

The golden image is setup with only server 2016 installed and the VDA, so there is no other software interfering and no optimization is done.  We also have the issue doing a RDP session to the VDA server. So this will almost exclude the VDA software.

 

Because we had 3 versions on the VDISK we did a merge, then we see that the package loss is decreasing and the latency is not impacted.

Next step  is updating PVS software.

 

We have created a support case with Citrix, and keep you informed.

 

Did you also upgrade XenServer OS or just the client tools?

 

Greetings,

 

Dennis

 

 

 

Link to comment
  • 0

Hi Dennis, 

 

I've just created a case today, as I'm going nowhere fast. We are still on XS 7.1CU1 across all hosts, so only tried different versions of client tools. 

 

I've also tried just changing the VBD driver to the one included with 71CU2, but that (as expected) did nothing to change the situation. 

 

When trying to run the CQI, it is green all the way through a 2 second 100% packet loss... how about that :D. 

 

I'm happy that I'm not the only one, it seems ever so strange that the old images are ok, and tens of combinations of new (or similar) builds fall flat on their face. 

 

 

Link to comment
  • 0

A small update on this. 

 

Have been in contact with Citrix support, and the first major step was to go thru several PVS best practise settings. I took the short route, and simple re-created the problem with a Reverse imaged version of the current "problem" PVS image, running directly on the hypervisor storage. That should rule out PVS. 

 

We then made a fresh vanilla image, directly on the hypervisor also - just running Chrome. No Citrix or other third party software load. This did not have the issues when connected via RDP. So the hypervisor does not seem to be causing the issues either. 

 

Next came the CleanupSessionListenersCancelDelay setting, but that didn't seem to make an impact. So we will most likely be deep diving traces next. 

Link to comment
  • 0
1 hour ago, Ronnie Pedersen said:

It seems the buzz word is "Micro lockup" and that the latest xentools update solves it. Saw a massive improvement yesterday, and will continue testing today. XENVIF needs to be 8.2.1.175  

 

 

Hi sglrope,

 

Where can i get the latest xentools update? Should i download the latest XenServer iso or is there another repository with Xentools versions?

 

Regards,

 

Jeroen

Link to comment
  • 0
On 1/8/2019 at 8:13 AM, Ronnie Pedersen said:

It seems the buzz word is "Micro lockup" and that the latest xentools update solves it. Saw a massive improvement yesterday, and will continue testing today. XENVIF needs to be 8.2.1.175  

 

 

Wonder the same thing, where can we get the latest XenTools with XENVIF 8.2.1.175?

The latest XenTools that we got from XenServer 7.6 has XENVIF 8.2.1.170 and we can't find any newer driver on Internet either.

 

Regards

Andreas

Link to comment
  • 0

For my specific problem, I simply did an online driver update of the "XenServer PV Network Class" in Device manager -> System devices  (went into driver, and hit update driver). Not something I would normally do, but since the issues appears to be specifically with this, why not. 

 

I don't have any information regarding an actual new version of Xen tools yet. Rumours are of a private fix, that someone has shared. I'm guessing it will be out soon, since its on Windows update as of the day before yesterday... 

Link to comment
  • 0

Download and open the tsv file from the link below.

https://pvupdates.vmd.citrix.com/updates.latest.tsv

 

When the tsv file is openend you get the links to the latest versions of XenTools

 

http://downloadns.citrix.com.edgesuite.net/14247/managementagentx86.msi 7.1.0.1305           5595136                x86

http://downloadns.citrix.com.edgesuite.net/14246/managementagentx64.msi 7.1.0.1305           5627904                x64

Link to comment
  • 0
4 minutes ago, Stefan Klopp said:

The mechanism of XenTools auto-update through windows-update is described in this article: https://support.citrix.com/article/CTX224933

The article mentions the URL: "https://pvupdates.vmd.citrix.com/updates.latest.tsv"

 

This file contains links to the installer of Version 7.1.0.1323:

x86: http://downloadns.citrix.com.edgesuite.net/14262/managementagentx86.msi 7.1.0.1323

x64: http://downloadns.citrix.com.edgesuite.net/14263/managementagentx64.msi 7.1.0.1323

 

As of the following post this should be the version which contains XENVIF 8.2.1.175 and which you are looking for:

https://discussions.citrix.com/topic/400561-vm-randomly-freeze-after-xentools-upgrade-from-65-to-76/#comment-2031100

 

I did not verify it myself yet ...

 

Thanks :2_grimacing:

 

You beat me to it :7_sweat_smile:

Link to comment
  • 0

Thank you guy's, this helped a lot! 

I've never seen that tsv link file before, guess I never really read the XenTools auto update article in full... :)

 

And wow, it actually solved the problem that we have been struggling with for weeks now.

Before the update we had big issues with micro freezes in the session (1-2sek freeze every 10-15 sek) but not anymore!

The weird thing is that if we don't run the machine with PVS we didn't see the problem at all... So we thought for a while that it was PVS related.

 

 

Regards

Andreas

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...