PVS Slow Boot - Large amount of data

[Paul Hite1709155384]

We have been fighting an ongoing slow boot issue since moving to Windows 10 1809 from Windows 7 in our Citrix PVS environment for all VDAs at all times (even during non-peak hours with one VDA booting and only a handful running). It's steadily gotten worse over the years, to the point where it's causing lots of desktops to go to a "failed to boot" state. If we look at the Statistics tab of the Virtual Disk tray tool, we can see the following:

 

Windows 7 Boot stats from the old image show 262 MB read at 20 Mbps, 101 second boot time.
Windows 10 stats from the current image show 2.2 GB read at 76 Mbps, 244 second boot time.

 

 

I have no idea if it's normal for Win 10 to require so much additional data to be read. We've tried standing up an image with most everything removed, including security software, and most startup items disabled. No change. What's also odd is that the "Throughput" number in the Windows 10 image is wildly inaccurate. It shows "4,185,995 KB/sec" which would be nearly 33 Gbps. I'm not sure if this version of the tool has a bug, but calculating the throughput manually (bytes read / boot time) gives us 76 Mbps. Watching network stats in vSphere confirms this rough speed (50-100 Mbps). The virtual disk tray tool on the old Win7 image on PVS 7.6 shows an accurate number for throughput. iPerf testing done between VDA<>PVS once the image is loaded shows over 6 Gbps TCP and close to 10 Gbps UDP. I believe the speed of the boot process is normal for BIOS mode (I'm aware we could possibly get better performance with UEFI) - what concerns me is the amount of data that must be read. What are you seeing for "Bytes read" during boot in your Windows 10 environment?

 

Environment is Citrix 7.15, single image, PXE boot, non-persistent, dedicated Citrix Desktop, on vSphere 7. Two PVS servers (Server 2016, 4 CPU, 32GB RAM) with local storage for images (Pure NVMe Array). VDAs are anywhere from 2 to 4 CPUs, 6 to 16GB RAM, using Cache in RAM with Overflow, 16 to 32GB cache disks, BIOS boot, and VMXNET3 interfaces. Adding more resources to either PVS or VDAs doesn't change the issue. We've even tried booting the image from a PVS server with no connections to a VDA on a huge ESXi host with nothing running on it, same result. TCP offload disabled, RSS enabled (tried disabling), MTU is 9014 (confirmed 8972 MSS works between VDA and PVS, and also tried reducing MTU to default). We haven't really tweaked much else in this environment. I've checked to verify the "RunAsPPL" regkey is not present as mentioned here: https://support.citrix.com/article/CTX476110/low-pvs-boot-throughput. Also tried disabling NetQueue, but didn't see any improvement: https://support.citrix.com/article/CTX139498/provisioning-services-target-devices-boot-slow-in-esx-5x.

[Carl Fallis]

So you already check the article I was going to mention,   you should check if you have security software that scans at boot or some other software that performs scanning it is the only thing I can think of that could be adding such high reads.  I am not sure why you are using such old version of win 10, 1809 is already EOL isn't?  Adding more resources will not change the amount of data read,   Converting to UEFI would only reduce the bootstrap time probably by 20-25%, that is not the entire boot time. 

[Paul Hite1709155384]

On 12/31/2022 at 3:26 PM, Carl Fallis said:

So you already check the article I was going to mention,   you should check if you have security software that scans at boot or some other software that performs scanning it is the only thing I can think of that could be adding such high reads.  I am not sure why you are using such old version of win 10, 1809 is already EOL isn't?  Adding more resources will not change the amount of data read,   Converting to UEFI would only reduce the bootstrap time probably by 20-25%, that is not the entire boot time. 

 

Sorry - We are using Windows 10 Enterprise LTSC 2019, which is equivalent to Win10 1809 and still supported - just wanted to reduce any confusion for those unfamiliar with LTSC. We do have plans to move to LTSC 2021 (21H2) in our next major update.

I'll double-check, but removing security software was a step I'm fairly certain we tried a few months ago with no appreciable difference in the boot times.