MCS Pooled / AHV slow master image update

[Andrew Gresbach1709152664]

We are setup with around 270 MCS Pooled Random desktops connected to a Nutanix AHV datastore.    When we update a small 5-10 desktop test pool, all machines shutdown, update and power back on completing within 5 min.   However when we update the master image on our production 270 desktop pool, it immediately shuts down large batches of desktops but they then sit in that state for several minutes before any update process actually happens on those shutdown desktops.  After that completes they power on fine and all is good but it takes approximately 30 min before the first machine flips from pending update "yes" to "no" .  From what I have gathered from other experiences w/ AHV and MCS, this process should be much quicker (i've heard benchmarks of about 5 min max to update a 100 desktop pool) so curious if maybe we have some settings off on either end?  any recommendations would be appreciated !

[Andrew Gresbach1709152664]

Old thread here but i'm still working with support on this one....BUT messing around on my own i changed my Studio settings to below and it seems much faster now.  (basically the only thing i changed was the maximum new actions per minute from the default 10 up to 100)  When i say fast , after the image finishes copying (~2ish min), i start to see VDA's change to "pending update no" after about 3 -4 min now (before it was about 20 before we'd start to see Disk Image Update tasks in AHV and see this start.  

 

[Vijay Gandagatri1709156937]

Apparently changing the display setting to "Never" helps....

 

The ACPI call sent to the guest VM is used to turn-off the display and only subsequent calls from the Studio initiate the actual VM reboot.

[Andrew Gresbach1709152664]

this is our connection info for AHV if it helps

 

[Richard Mash]

We have the same issue... We're running 2 different hypervisors as we migrate to AHV and there is a noticeable delay in the AHV process. Updating 100 images on the XenServer 7.1 hypervisor hosts is very fast.  Updating 100 desktops on AHV is painfully slow. It takes a long time before the VMs even start to power off. It seems like the bulk power off/on actions are slow using the Nutanix MCS connector. Changing any of the settings in the screenshot you posted doesn't seem to make a difference.

We just upgraded to AHV 5.10 and CVAD 1906.2 but the issue persists.

[Andrew Gresbach1709152664]

Thanks for your info!  i was curious if anyone had tested on another hypervisor as a comparison.  I already had opened a Citrix ticket and confirmed it probably was not on their end so i'm opening a Nutanix ticket this morning to see what we can do.  do you know what version of the Nutanix MCS connector tool that you have on your DDCs?  i'm sure that will be something that gets looked at too

[Richard Mash]

Our MCS Plugin is: "Nutanix AHV Plugin for Citrix XenDesktop" - Version 2.3.0.0

 

I updated a pool of 95 desktops on the AHV cluster last night. In case it helps, it looks like the delay (at least on our installation) is in the VM shutdown process on the cluster. When I monitor the VM shutdown/startup in Stiudio and PRISM this is what I see:

• Update the Machine Catalog. The master image gets copied very quickly.
• On the AHV cluster the "Preparation - xxx" VM gets created, started and removed very quickly.
• There is then a short delay where nothing seems to happen in PRISM.
• The PRISM Task list then shows a lot of "VM set power state" tasks" all at once, so it looks like the MCS connector is sending the guest shutdown commands in bulk.
• There is a long delay before the first VM starts to shut down. I think this is where most of the lag is coming from.
• When all the VMS are shut down there is another slight delay before the PRISM task list starts to show "VM disk update" and 'VM set power state" as the VMs are refreshed with their new image and start to power on again.

We have also noticed that if we create a new machine catalog, the master image copy and VM creation/startup is very fast, which supports my theory that the delay is in the VM shutdown task when updating an image.

[Andrew Gresbach1709152664]

awesome thank you for all the detailed steps! i opened a Nutanix case this morning and included your troubleshooting steps and observations along with mine if you dont mind.  i will report back here if we make any progress

[Andrew Gresbach1709152664]

quick update here.......still working w/ Nutanix support on this one.  they had me simulate an update while running a CDF trace and document details.   Afterwards i had to run logbay bundle captures on each CVM and upload so working on that tonight.  Hopefully can make some headway soon as it took about 45 min to update 173 machines tonight so that seems pretty extreme compared to what is supposed to be possible w/ this hardware

[Andrew Gresbach1709152664]

yup good addition......i completely forgot about that one too , we ran into that a couple of years ago when we first implemented AHV.   we had that but also needed the studio settings i posted above to get things better

[Vijay Gandagatri1709156937]

Thanks...we just figured this yesterday with Support.....testing Citrix deployment on AHV....not as straightforward as VMware integration.