Jump to content

Citrix DaaS Power Management of Azure Virtual Machines with PowerShell

  • Contributed By: Gerhard Krenn Special Thanks To: Nitin Mehta, Yuhua Lu, Ivan Viamonte, Steve Beals

Introduction

With Citrix DaaS, you can power manage Machine Creation Services (MCS) provisioned virtual machines across various supported hypervisors and cloud services. The power management operation provides you:

  • Optimal user experience
  • Cost management and power savings

In Azure environments, you can create an MCS machine catalog that supports hibernation. Using this feature, you can suspend a virtual machine when the session is idle for some time and then reconnect to the previous state of the virtual machine when a user signs in again.

Quote

Note:

The hibernation capability applies to only Single-session OS (persistent and non-persistent) machine catalogs. When hibernating a virtual machine, Azure signals the guest operating system to perform suspend-to-disk. During hibernation, Azure persists the memory (RAM) contents of the virtual machine in the OS disk and deallocates the virtual machine. Later, when starting the virtual machine, the RAM contents are restored back from the OS disk, and applications and processes that were previously running in the virtual machine resume from their last state.

This guide only covers the Microsoft Azure Hibernation functionality with Powershell.

Once a virtual machine is in a hibernated/deallocated state, you are not billed for the virtual machine usage. You only pay for the storage (OS disks, data disks) and networking resources (IPs, and so forth) attached to the virtual machine. As a result, hibernating a virtual machine can be useful in scenarios such as:

  1. Virtual desktops and development and test scenarios where the virtual machines can be hibernated/deallocated during non-business hours.
  2. Applications that require considerable time to load due to memory components. These applications can be initialized on virtual machines and hibernated/deallocated. These pre-warmed virtual machines can then be quickly started when needed, with the applications already up and running in the desired state.

     

What happens when hibernating a virtual machine? When hibernation is triggered on a virtual machine using the Azure Portal, CLI, PS, SDKs, or APIs, Azure signals the guest operating system to perform suspend-to-disk (S4).

During hibernation, the contents of the RAM are saved on the OS disk, and the virtual machine is deallocated (that is, the virtual machine releases the lease on the underlying hardware and is powered off). Virtual machine states and billing have more details on the virtual machine deallocated state.

During hibernation, data in the temporary disk does not persist.

Once a virtual machine is hibernated/deallocated, the OS disk, data disks, and NICs remain attached to your virtual machine. Static IP addresses do not change when the virtual machine is suspended.

Supported virtual machine sizes

Virtual machine sizes with up to 64 GB RAM from the following virtual machine series support hibernation.

  • Dasv5-series
  • Dadsv5-series
  • Dsv5-series
  • Ddsv5-series
  • Easv5-series
  • Eadsv5-series
  • Esv5-series
  • Edsv5-series

VM sizes with up to 112 GB RAM from the following GPU VM series support hibernation.

  • NVv4-series
  • NVadsA10v5-series

Supported Operating Systems

  • Linux
  • RHEL
  • Ubuntu Server 22.04 LTS
  • Ubuntu Server 20.04 LTS
  • Ubuntu Server 18.04 LTS
  • Debian 10 with backport kernel
  • Windows
  • Windows Server 2022
  • Windows Server 2019
  • Windows 11 Pro
  • Windows 11 Enterprise
  • Windows 11 Enterprise multi-session
  • Windows 10 Pro
  • Windows 10 Enterprise
  • Windows 10 Enterprise multi-session

Known issues

  • Debian 11 cannot be tested due to Azure issues.

Azure-related prerequisites to use hibernation

You must enable the feature for your subscription before you can use hibernation. 
Hibernation can be enabled on the virtual machine at the time of creating the virtual machine.

  • Hibernation must be supported by the virtual machine size.
  • Hibernation must be supported by the OS.
  • If a virtual machine is being created from an OS disk or a Compute Gallery image, then the OS disk or Gallery Image definition must support hibernation.
  • You can use a persistent OS disk which is large enough to store the contents of the RAM, OS, and other applications running on the virtual machine.
  • The virtual machine must have the Azure virtual machine Agent installed if you are using the Windows or Linux Hibernate Extensions.
Quote

Note:

MCS automatically sets the hibernation property for the generated resources. You do not need to configure the properties of the master resources to support hibernation.

Limitations

  • When a virtual machine is hibernated/deallocated, you cannot attach/detach any disks or NICs to the virtual machine. To do so, you must move the virtual machine to a stopped/deallocated state by stopping the virtual machine and then attach/detach disks and NICs.
  • When a virtual machine is hibernated/deallocated, you cannot modify the disks and NICs associated with the virtual machine. To do so, you must move the virtual machine to a stopped/deallocated state by stopping the virtual machine and then modify the disks and NICs.
  • When a virtual machine is hibernated/deallocated, there is no capacity guarantee to ensure that there will be sufficient capacity to start the virtual machine later. In rare cases, if you encounter capacity issues, you can try starting the virtual machine later.
  • If a virtual machine has a Capacity Reservation associated with it and is hibernated/deallocated, the Capacity Reservation does not ensure that the virtual machine will have the capacity to resume.
  • You can only hibernate/deallocate a virtual machine using the Azure Portal, CLI, PowerShell, SDKs, and API. Hibernating the virtual machine using guest OS operations does not result in the virtual machine moving to a hibernated/deallocated state and the virtual machine remains billed.
  • You cannot disable hibernation on a virtual machine once it's enabled.
  • Hibernation is only supported with Nested Virtualization when Trusted Launch is enabled on the VM.

The following are not supported with hibernation:

  • Ephemeral OS disk
  • Shared disk
  • Availability Sets
  • Virtual machines Uniform
  • Spot virtual machines
  • Managed images
  • Azure Backup
  • Capacity reservations

Windows Limitations:

  • The page file cannot be on the temp disk.
  • Applications such as Device Guard and Credential Guard that require virtualization-based security (VBS) are only supported with hibernation when Trusted Launch is enabled on the virtual machine and Nested Virtualization is enabled in the guest OS. Hibernation is supported with Nested Virtualization only when Trusted Launch is enabled on the virtual machine.

Linux Limitations:

  • Hibernation is not supported with Trusted Launch for Linux virtual machines.

IMPORTANT:
The mentioned limitation of Windows "The page file cannot be on the temp disk for a hibernation-capable VM" is not valid for MCS-based deployments:
The page file of the Windows image can be on the temporary disk, MCS will move it to the C drive during image preparation!

Required permissions for Power Management

Minimum permissions give better security control. However, new features that require more permissions fail if only minimum permissions are given.

    "Microsoft.Compute/virtualMachines/read", "Microsoft.Resources/subscriptions/resourceGroups/read", "Microsoft.Compute/virtualMachines/deallocate/action", "Microsoft.Compute/virtualMachines/start/action", "Microsoft.Compute/virtualMachines/restart/action",

Create hibernation-capable virtual machines and Machine Catalogs

In Azure environments, you can create an MCS machine catalog that supports hibernation. Using this feature, you can suspend a virtual machine when the session is idle for some time and then reconnect to the previous state of the virtual machine when a user signs in again.

Quote

Note:

The hibernation capability applies to only Single-session OS (persistent and non-persistent) machine catalogs.

To use the hibernation capability, you can do the following. However, before proceeding, see the prerequisites (link to be added) and limitations (link to be added).

  • Create and manage a hibernation-capable machine catalog using the Full Configuration interface or PowerShell commands. You must select a machine profile that supports hibernation. (link to the topics to be added).
  • Create a machine catalog for existing hibernation-capable virtual machines using the Full configuration interface. You can create a machine catalog containing both hibernation-capable and incapable virtual machines. However, if you want hibernation-related functionality, create the machine catalog with only hibernation-enabled virtual machines. (link to the topic to be added)
  • Enable hibernation on existing MCS-provisioned virtual machines using PowerShell commands. (link to the topic to be added)
  • Check the hibernation property of a machine catalog, virtual machine, and broker machine using PowerShell commands. (link to the topic to be added).

Prerequisites to use hibernation

To use hibernation, make sure to complete the following tasks on your Azure subscription:

  1. Enable the feature for your subscription.

  2. Install the Azure Virtual Machine Agent on the master image for both Windows and Linux. The page file of the Windows image can be on the temporary disk. MCS sets the page file location to the C: drive in the base disk when hibernation is enabled on the machine catalog.

  3. On Windows-based virtual machines, the Agent is normally installed automatically.

  4. To check if the Agent is installed, run the following PowerShell script.

    Get-Azvirtual machine  -Name "CTX-HibTest-M" -ResourceGroupName "HibTest" -status
    

     

    deployment-guides-citrix-azure-hibernation-poshwin.png

     

     

    deployment-guides-citrix-azure-powermanagment-azureportal1.png

     

  5. Use a virtual machine size in your subscription that supports hibernation as detailed in Supported virtual machine sizes.

  6. Create a hibernation-capable machine profile (virtual machine or template spec) so that virtual machines inherit the hibernation-capability. To create the virtual machine, see the Microsoft user guide on hibernation. To create the template spec, open the Azure Portal. Choose a virtual machine whose configuration you want to use in the template. Select Export template in the left pane.

     

    deployment-guides-citrix-azure-powermanagment-azureportal2.png

     

  7. Clear the Include parameters checkbox. Copy the context and save it as a JSON file, for example VMExportTemplate.json.

     

    deployment-guides-citrix-azure-powermanagment-azureportal3.png

     

  8. Modify the local template file VMExportTemplate.json. The parameter hibernationEnabled must be true. You can specify a supported virtual machine size. However, you can also specify the machine size while creating the catalog.

  9. Add the template for the network interface resource to the JSON file VMExportTemplate.json. As a result, you have an ARM template file containing two resources.

     

    deployment-guides-citrix-azure-powermanagment-azureportal4.png

     

  10. Select Azure Portal > Template specs > Import template > Choose local template file to import this template file as an ARM template spec.

     

    deployment-guides-citrix-azure-powermanagment-azureportal5.png

     

  11. After the ARM Template specification is created, you can use it as a machine profile.

Quote

Note:

It might take a few minutes to sync to Citrix Studio.

Creating a Hibernation-enabled virtual machine using Azure Portal

Assuming all prerequisites are met, you can create a Hibernation-enabled Azure virtual machine using Azure Portal.

 

deployment-guides-citrix-azure-powermanagment-azureportal6.png

 

If you do not see the “Enable Hibernation” checkbox, the feature activation is not completed or failed. Keep in mind to choose the correct machine size otherwise an error occurs.

Creating a Hibernation-enabled virtual machine using PowerShell

  1. Assuming all prerequisites are met, the following PowerShell script creates a Hibernation-enabled virtual machine - in this example Windows 11 Enterprise in the newest available version.

        // Create a dedicated Resource group - not required, but recommended
        PS C:\TACG> New-AzResourceGroup -Name HibTest -Location "Germany West Central"
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate1.png

     

        //Check which virtual machines are available for creation
        $publname="MicrosoftWindowsDesktop"
        $ location="Germany West Central"
        Get-Azvirtual machineImageOffer -Location $location -PublisherName $publname | select offer
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate2.png

     

        $offername="windows-11"
        Get-Azvirtual machineImageSKu -Location $location -PublisherName $publname -Offer $offername | Select SKus
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate3.png

    deployment-guides-citrix-azure-powermanagment-poshcreate4.png

     

        $Skuname="win11-22h2-ent"
        Get-Azvirtual machineImage -Location $location -PublisherName $publname -Offer $offername -Skus $Skuname |  Select-Object -Last 1
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate5.png

     

        $ImgResult=Get-Azvirtual machineImage -Location $location -PublisherName $publname -Offer $offername -Skus $Skuname |  Select-Object -Last 1
        $URN = $Imgresult.PublisherName + ":" + $Imgresult.Offer + ":" + $Imgresult.Skus + ":" + $Imgresult.version
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate6.png

     

        // Example settings for creating a virtual machine
    
        $ResourceGroupName="HibTest"
        $Location="GermanyWestCentral"
        $Name="CTX-HibTest-M"
        $ImageName=$URN
        $OpenPorts="3389"
        $VirtualNetworkName="CTX-HibTest-VNet"
        $SubnetName="CTX-HibTest-Subnet"
        $SecurityGroupName ="CTX-HibTest-NSG"
        $Size ="Standard_D2S_v5"
        $cred = Get-Credential -Message "Enter username and password for the virtual machine."
    
        //Creating the virtual machine based on the previous parameters
    
        New-Azvirtual machine -ResourceGroupName $ResourceGroupName -Location $Location -Image $ImageName -OpenPorts $OpenPorts -VirtualNetworkName $VirtualNetworkName -SubnetName $SubnetName -SecurityGroupName $SecurityGroupName  -HibernationEnabled -Credential $cred -Name $Name -Size $size
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate7.png

     

        // To check already created virtual machines if Hibernation is supported:
        // Deallocated virtual machines:
    
        Get-Azvirtual machine -ResourceGroupName "HibTest" -Name "CTX-HibTest-M" -Status
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate8.png

    deployment-guides-citrix-azure-powermanagment-poshcreate9.png

     

        //Running virtual machines - you can determine if the virtual machine has the required Azure agent installed and the Agent is running:
    
        Get-Azvirtual machine  -Name "CTX-HibTest-M" -ResourceGroupName "HibTest" -status
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate10.png

    deployment-guides-citrix-azure-powermanagment-poshcreate11.png

    deployment-guides-citrix-azure-powermanagment-poshcreate12.png

     

        //Checking the Hibernation capabilities in the Client OS of a running virtual machine: 
        1.  Check, if the Pagefile is on the C:\ drive
        2.  Check Output of powercfg /a
    
        C:\Windows\system32> .\powercfg.exe /a
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate13.png

     

        // Try out Hibernation using Powershell
        Stop-Azvirtual machine -ResourceGroupName "HibTest" -Name "CTX-HibTest-M" -Hibernate
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate14.png

     

        Get-Azvirtual machine  -Name "CTX-HibTest-M" -ResourceGroupName "HibTest" -status
    

     

    deployment-guides-citrix-azure-powermanagment-poshcreate15.png

     

  2. Create the Master Image-virtual machine based on the virtual machine you created. When the Master image is ready we can create a Machine Catalog.

Creating a Hibernation-capable Machine Catalog using the Citrix Cloud GUI

The following screenshots show the steps for creating a Hibernation-capable Machine Catalog.

  1.  

    deployment-guides-citrix-azure-powermanagment-mc1.png

     

  2.  

    deployment-guides-citrix-azure-powermanagment-mc2.png

     

  3.  

    deployment-guides-citrix-azure-powermanagment-mc3.png

     

  4.  

    deployment-guides-citrix-azure-powermanagment-mc4.png

     

  5.  

    deployment-guides-citrix-azure-powermanagment-mc4.png

     

  6.  

    deployment-guides-citrix-azure-powermanagment-mc5.png

     

  7.  

    deployment-guides-citrix-azure-powermanagment-mc6.png

     

  8.  

    deployment-guides-citrix-azure-powermanagment-mc7.png

     

  9. Hovering over the virtual machine-based Machine Profile reveals that Hibernation is supported on this profile.

  10.  

    deployment-guides-citrix-azure-powermanagment-mc8.png

     

  11.  

    deployment-guides-citrix-azure-powermanagment-mc9.png

     

  12.  

    deployment-guides-citrix-azure-powermanagment-mc10.png

     

  13.  

    deployment-guides-citrix-azure-powermanagment-mc11.png

     

  14.  

    deployment-guides-citrix-azure-powermanagment-mc12.png

     

  15.  

    deployment-guides-citrix-azure-powermanagment-mc13.png

     

  16.  

    deployment-guides-citrix-azure-powermanagment-mc14.png

     

  17.  

    deployment-guides-citrix-azure-powermanagment-mc15.png

     

    Creation of the Machine Catalog is complete.

  18. To determine the Hibernation capabilities of the Machine Catalog and the virtual machines in the MC we run some checks using PowerShell.

        //Check Hibernation capability of Provisioning scheme just created
    
        (Get-ProvScheme -ProvisioningSchemeName MC-AZ-HibTest).virtual machineMetadata -join "" | ConvertFrom-Json | Select HibernationEnabled
    
        //Check Hibernation capability of virtual machine(s) in MC
    
        (Get-Provvirtual machine -virtual machineName "HibTest-1").Customvirtual machinedata | ConvertFrom-Json | Select SupportsHibernation
    

     

    deployment-guides-citrix-azure-powermanagment-posh1.png

     

Creating a Hibernation-capable Machine Catalog using PowerShell

After you meet all the requirements to use hibernation (link to the prerequisites topic), you can create a hibernation-capable machine catalog using the New-ProvScheme command. For information on how to create a catalog using the Remote PowerShell SDK, see here.

  1. Create the Machine Catalog.

        New-BrokerCatalog -AllocationType 'Static' -IsRemotePC $False -MinimumFunctionalLevel 'L7_34' -Name 'MC-AZ-HibTest-PSH' -PersistUserChanges 'OnLocal' -ProvisioningType 'MCS' -Scope @() -Sess
        ionSupport 'SingleSession`
    

     

    deployment-guides-citrix-azure-hibernation-posh3.png

     

  2. Create an IdentityPool.

        New-AcctIdentityPool -AllowUnicode -Domain 'hib.the-austrian-citrix-guy.at' -IdentityPoolName 'IP-AZ-HibTest-PSH' -NamingScheme 'HibTest-PS-#' -NamingSchemeType 'Numeric' -OU 'CN=Computers,D
        C=hib,DC=the-austrian-citrix-guy,DC=at' -Scope @()
    

     

    deployment-guides-citrix-azure-hibernation-posh4.png

     

  3. Create a Provisioning Scheme.

        New-ProvScheme -HostingUnitName 'NW-TACG-HibTestNN' -IdentityPoolName 'IP-AZ-HibTest-PSH' -MasterImagevirtual machine 'XDHyp:\HostingUnits\NW-TACG-HibTestNN\image.folder\CTX-Hibtest2.resourcegroup\CTX-Hi
        bTest-M_OsDisk_1_53e27e588432470da645b2fd51afe622.manageddisk' -ProvisioningSchemeName 'PvSch-Hibtest-PSH' -Scope @() -MachineProfile 'XDHyp:\HostingUnits\NW-TACG-HibTestNN\machineprofile.folder\CTX-Hibtest2.res
        ourcegroup\CTX-HibTest-M.vm' -NetworkMapping @{'0'='XDHyp:\HostingUnits\NW-TACG-HibTestNN\virtualprivatecloud.folder\East US.region\virtualprivatecloud.folder\CTX-Hibtest2.resourcegroup\TACG-HibTest-DCCC-vnet.vi
        rtualprivatecloud\default.network'}
    

     

    deployment-guides-citrix-azure-hibernation-posh5.png

     

  4. Check if the Provisioning Scheme supports Hibernation.

        (Get-ProvScheme -ProvisioningSchemeName "PvSch-Hibtest-PSH").virtual machineMetadata -join "" | ConvertFrom-Json | Select HibernationEnabled
    

     

    deployment-guides-citrix-azure-hibernation-posh6.png

     

  5. Update the BrokerCatalog with the unique Id of the provisioning scheme created earlier.

        Set-BrokerCatalog -Name 'MC-AZ-Hibtest-PSH' -ProvisioningSchemeId 'b659c922-5f1a-4163-a3ec-f09b57c081ef`
    

     

    deployment-guides-citrix-azure-hibernation-posh7.png

     

  6. Add controller addresses to the provisioning scheme object.

        Add-ProvSchemeControllerAddress -AdminAddress 'tacg-hibtest-dc.hib.the-austrian-citrix-guy.at' -ControllerAddress @('tacg-hibtest-dc.hib.the-austrian-citrix-guy.at') -ProvisioningSchemeName 'PvSch-Hibtest-PSH'
    

     

    deployment-guides-citrix-azure-hibernation-posh8.png

     

  7. Create the required AD computer accounts in Active Directory.

        New-AcctADAccount -Count 1 -IdentityPoolUid fe8bd3e8-1de6-40da-9a53-37b6b3ebd454 -ADUsername 'TACG-HIB\azadmin'
    

     

    deployment-guides-citrix-azure-hibernation-posh9.png

     

  8. Create a virtual machine, based on the template definition in the provisioning scheme created earlier.

        New-Provvirtual machine -ADAccountName @('TACG-HIB\Hibtest-PS-1\$')  -ProvisioningSchemeName 'PvSch-Hibtest-PSH'
    

     

    deployment-guides-citrix-azure-hibernation-posh10.png

     

  9. Create a Broker Machine object.

        New-BrokerMachine -CatalogUid 23 -HostedMachineId 'TACG-HIB\HibTest-PS-1' -HypervisorConnectionUid 13 -MachineName ' TACG-HIB\HibTest-PS-1'
    

     

    deployment-guides-citrix-azure-hibernation-posh11.png

     

  10. Check successful virtual machine creation.

        Get-BrokerMachine -MachineName 'TACG-HIB\HibTest-PS-1
    
  11. Creation of the Machine Catalog is now complete. To determine the Hibernation capabilities of the virtual machines in the MC we run some checks using PowerShell.

        (Get-Provvirtual machine -virtual machineName "HibTest-PS-1").Customvirtual machinedata | ConvertFrom-Json | Select SupportsHibernation
    

     

    deployment-guides-citrix-azure-hibernation-posh13.png

     

Editing/Updating a Machine Catalog

If you already have hibernation-capable virtual machines and want to use DaaS to suspend and resume them, create machine catalogs to import those virtual machines to DaaS for power management.

Quote

Note:

You can create a machine catalog containing both hibernation-capable and -incapable virtual machines. However, if you want hibernation-related functionality, you must create the machine catalog with only hibernation-capable virtual machines.

If the current Machine Catalog supports Hibernation, you cannot:

  • Change the virtual machine size Service Offering to a Hibernation-incapable size
  • Change the Machine Profile to a Hibernation-incapable profile

If the current Machine Catalog does not support Hibernation, you can:

  • Change the machine profile to a Hibernate-enabled profile using the Full Configuration interface or use PowerShell commands.

You can enable Azure Hibernation on the following existing Machine Catalogs:

  • Windows MCS-provisioned virtual machines of a machine catalog created without a temporary disk.
  • Linux MCS-provisioned virtual machines of a machine catalog created with and without a temporary disk.
Quote

Note:

The existing MCS-provisioned virtual machines must have an Azure virtual machine Agent installed (see above).

Check hibernation property

You can check the hibernation property of a machine catalog, virtual machine, and a broker machine using the PowerShell commands already mentioned and shown earlier.

    (Get-ProvScheme -ProvisioningSchemeName MC-AZ-HibTest-NoHib).virtual machineMetadata -join "" | ConvertFrom-Json | Select HibernationEnabled

    (Get-Provvirtual machine -virtual machineName "HibTest-NoHib-1").Customvirtual machinedata | ConvertFrom-Json | Select SupportsHibernation

Using the Full Configuration interface:

 

deployment-guides-citrix-azure-powermanagment-config1.png

 

 

deployment-guides-citrix-azure-powermanagment-config2.png

 

 

deployment-guides-citrix-azure-powermanagment-config3.png

 

Virtual machine deployed in a Machine Catalog created without a Machine Profile results in Hibernation being unsupported.

  1. We can change the Provisioning Scheme of the Machine Catalog to use a Machine Profile.

        Set-ProvScheme -provisioningSchemeName 'MC-AZ-HibTest-NoHib' -machineProfile 'XDHyp:\HostingUnits\NW-TACG-HibTestNN\machinepro
        file.folder\CTX-Hibtest2.resourcegroup\CTX-HibTest-M.vm' -ServiceOffering 'XDHyp:\HostingUnits\NW-TACG-HibTestNN\serviceoffering.folder\Standard_D2
        s_v5.serviceoffering'
    
  2. Request update on existing virtual machines in a machine catalog.

        Set-Provvirtual machineUpdateTimeWindow -ProvisioningSchemeName` 'MC-AZ-HibTest-NoHib' -virtual machineName 'HibTest-NoHib-1`
    
  3. Restart the virtual machine.

        New-BrokerHostingPowerAction -MachineName 'HibTest-NoHib-1' -Action Restart
    

     

    deployment-guides-citrix-azure-powermanagment-config7.png

     

  4. Check if MC is now enabled to support Hibernation.

        (Get-ProvScheme -provisioningSchemeName 'MC-AZ-HibTest-NoHib').virtual machineMetadata -join "" |  ConvertFrom-Json | Select HibernationEna
        bled
    

     

    deployment-guides-citrix-azure-powermanagment-config8.png

     

     

    deployment-guides-citrix-azure-powermanagment-config4.png

     

  5. Check if the virtual machine is now enabled to support Hibernation.

        (Get-Provvirtual machine -virtual machineName 'HibTest-NoHib-1').CustomVmData | ConvertFrom-Json | Select SupportsHibernation
    

     

    deployment-guides-citrix-azure-powermanagment-config9.png

     

     

    deployment-guides-citrix-azure-powermanagment-config5.png

     

     

    deployment-guides-citrix-azure-powermanagment-config6.png

     

We updated an existing Machine Catalog which initially had no support for Azure Hibernation using PowerShell to finally support Azure Hibernation.

How to use Hibernation feature

Hibernation-capable virtual machines can be hibernated/deallocated in two different ways:

  1. Initiate Hibernation of the virtual machine via the Studio action menu

    • Full Configuration interface:
      • Select a virtual machine in the hibernation-capable machine catalog and select Start. -After the Power State is On, right-click the virtual machine, and select Suspend. Click Yes to confirm the action. -The Power State changes from Suspending to Suspended. -You can check the status of the virtual machine in the Azure portal.
  2. To resume and reconnect back to a hibernated/deallocated virtual machine, use one of the following:

    • Administrators can resume the virtual machine using the Full Configuration interface.
    • End users can start the virtual machine using the Citrix Workspace menu.
Quote

Note:

You can do the following power management operations on the hibernated/deallocated virtual machines: (Suspend virtual machine from the running state, Resume virtual machine from the suspended state, Force shut down virtual machine from a suspended state, and Force restart virtual machine from the suspended state)

 

deployment-guides-citrix-azure-powermanagment-hibernate1.png

 

Enabling and Configuring Autoscale using the Full Configuration Interface

Citrix Autoscale provides a consistent, high-performance solution to proactively power manages your machines. It aims to balance costs and user experience. Autoscale incorporates the deprecated Smart Scale technology into the Manage console’s power management solution.

Autoscale enables proactive power management of all registered single-session and multi-session OS machines in a delivery group. It supports all platforms that Citrix DaaS supports - Citrix Hypervisor, AWS, Google Cloud Platform, Microsoft Azure Resource Manager, virtual machineware vSphere, and many more.

Autoscale powers on and off machines in a delivery group based on a schedule that you set. A schedule includes the number of active machines for each time slot, with peak and off-peak times defined.

Autoscale supports both multi-session OS and single-session OS delivery groups. There are three user interfaces to be aware of:

  • Autoscale user interface for multi-session OS delivery groups (formerly RDS delivery groups)
  • Autoscale user interface for single-session OS random (pooled) delivery groups (formerly pooled VDI delivery groups)
  • Autoscale user interface for single-session OS static delivery groups (formerly static VDI delivery groups)

For managing virtual machines with Azure Hibernation enabled keep in mind that currently only single-session OS delivery groups are supported.

Enabling Autoscale

  1. You can enable Autoscale by choosing the corresponding Delivery group and click Manage Autoscale.

     

    deployment-guides-citrix-azure-powermanagment-enable1.png

     

  2. In the next window click Enable autoscale.

     

    deployment-guides-citrix-azure-powermanagment-enable2.png

     

Basic concepts of Citrix Autoscale for Azure Hibernation-enabled virtual machines.

Schedules:

  • Autoscale powers machines on and off based on the selected schedule. Autoscale lets you set multiple schedules that include specific days of the week and adjust the number of machines available during those times. If you expect a set of users to consume the machine resources at a specific time on specific days, Autoscale helps provide an optimized experience.
  • The schedule is based on the time zone of the delivery group.
  • Autoscale treats only those machines that are registered with the site as part of the available capacity in the calculations it makes. “Registered” means that the machine is available for use or already in use. Doing so ensures that only machines that can accept user sessions are included in the capacity for the delivery group.
Quote

Note:

Those machines are powered on during the schedule, whether there are sessions running on them.

Capacity buffer:

  • A Capacity buffer is used to add spare capacity to the current demand to account for dynamic load increases.

Autoscale lets you set the capacity buffer separately for peak and off-peak times. A lesser value in the capacity buffer field decreases the cost because Autoscale powers on less spare capacity. A greater value ensures an optimized user experience so that users do not have to wait for more machines to power on when launching sessions. By default, the capacity buffer is 10%. For single-session OS delivery groups, the capacity buffer is defined as a percentage of the total number of machines in the delivery group.

Quote

Note:

The capacity buffer results in machines being powered on when the total spare capacity drops to a level below “X” percent of the total capacity of the delivery group. Doing so reserves the required percentage of spare capacity.

Power policies:

  • The Power policies control the behavior of the Autoscale depending on Session status:
    • The session on the virtual machine is disconnected. The virtual machine is put into hibernation or shuts down when the specified disconnection time elapses, depending on the action you configured (the action to choose to put the machine into Hibernation is “Suspend”). By default, no action is assigned to disconnected machines. You can define actions separately for peak and off-peak times.

The session is logged off:

  • The virtual machine is suspended or shut down when the specified logoff time elapses, depending on the actions you configured (the action to choose to put the machine into Hibernation is “Suspend”). By default, no action is assigned to logged-off machines. You can define actions separately for peak and off-peak times.

 

deployment-guides-citrix-azure-powermanagment-enable3.png

 

Setting the Peak Times on the schedule:

  • You can set the Peak Times in 30min-intervals by clicking in the time-schedule. Each blue frame represents a timeslot marked as Peak Time. The Peak Times need not to be consecutive, non-consecutive timeslots are possible.

 

deployment-guides-citrix-azure-powermanagment-enable4.png

 

  • Non-consecutive timeslots are possible.

 

deployment-guides-citrix-azure-powermanagment-enable5.png

 

Enabling and Configuring Autoscale using PowerShell

  1. First we check whether Autoscale is already activated in the respective desktop group.

        Get-BrokerDesktopGroup -Name "BG-AZ-HibTest" | Select AutoscalingEnabled
    

     

    deployment-guides-citrix-azure-powermanagment-enable6.png

     

  2. If it is not enabled we can enable it.

        Get-BrokerDesktopGroup -Name "BG-AZ-HibTest"  -AutoscalingEnabled $true
    
  3. Check if a schedule is already configure by getting the Uid and then using the Get-BrokerPowerTimeScheme command.

        Get-BrokerDesktopGroup -Name "BG-AZ-HibTest" | Select Uid
    
        Get-BrokerPowerTimeScheme -DesktopGroupuid <Your Uid #.>
    

     

    deployment-guides-citrix-azure-powermanagment-enable7.png

     

  4. To create a schedule the syntax of the script is important to know:

    • The time slots representing the Peak Hours are created in 30min-based time frames starting to count at 0 representing 12pm.
    • If the Peak Hours are be between 9am and 5pm we must set the time slots 18 (9am) to 34 (5pm).

     

    deployment-guides-citrix-azure-powermanagment-enable8.png

     

  5. The schedule is now created and enabled.

    Quote

    Note:

    The script produces an error if another schedule has the same time slots marked as Peak Times!

  6. Now we can set the other important settings like Capacity Buffer, Power-Off delay.

        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -PeakBufferSizePercent 0` -OffPeakBufferSizePercent 0
    
        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -PowerOffDelay 1
    
  7. At last we must set the Power policies. As we want to hibernate/deallocate all virtual machines when not in use we set the action to “Suspend” and the time before the action occurs to one minute.

        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -PeakLogoffTimeout 1 -PeakLogOffAction Suspend
        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -PeakDisconnectTimeout 1 -PeakDisconnectAction Suspend
        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -OffPeakLogoffTimeout 1 -OffPeakLogOffAction Suspend
        Set-BrokerDesktopGroup -Name "BG-AZ-HibTest" -OffPeakDisconnectTimeout 1 -OffPeakDisconnectAction Suspend
    
  8. Now all relevant Autoscale settings for hibernating Azure-based virtual machines are set and active.

Troubleshooting guide

Unable to create a virtual machine with hibernation enabled

If you are unable to create a virtual machine with hibernation enabled, ensure that you are using a virtual machine size, OS version that supports Hibernation. Refer to the supported virtual machine sizes, OS versions section in the user guide in addition to the limitations section for more details. Some common error codes that you might observe include:

ResultCode Error Message Action
OperationNotAllowed The referenced OS disk must support hibernation for a virtual machine with hibernation capability. Validate that the OS disk has hibernation support enabled.
  The referenced platform image must support hibernation for a virtual machine with hibernation capability. Use a platform image that supports hibernation.
  The referenced shared gallery image must support hibernation for a virtual machine with hibernation capability. Validate that the Shared Gallery Image Definition has hibernation support enabled.
  Hibernation capability is not supported for Spot virtual machines.  
  User virtual machine Image is not supported for a virtual machine with Hibernation capability. Use a platform image or Shared Gallery Image if you want to use the hibernation feature.
  Referencing a Dedicated Host is not supported for a virtual machine with Hibernation capability.  
  Referencing a Capacity Reservation Group is not supported for a virtual machine with Hibernation capability.  
  Enabling/disabling hibernation on an existing virtual machine requires the virtual machine to be stopped/deallocated first.
  Hibernation cannot be enabled on Virtual Machine since the OS Disk Size ({0} bytes) must at least be greater than the virtual machine memory ({1} bytes) Ensure that the OS disk has enough space to be able to persist the RAM contents once the virtual machine is hibernated/deallocated.
  Hibernation cannot be enabled on Virtual Machines created in an Availability Set. Hibernation is only supported for standalone virtual machines and virtual machineSS Flex virtual machines.

Unable to hibernate a virtual machine

If hibernation is enabled on the virtual machine, check if hibernation is successfully enabled in the guest OS.

For Windows, you can check the status of the Hibernation extension to see if the extension was able to successfully configure the guest OS for hibernation.

 

deployment-guides-citrix-azure-powermanagment-troubleshooting1.png

 

The virtual machine instance view would have the final output of the extension :

 

deployment-guides-citrix-azure-powermanagment-troubleshooting2.png

 

Also, confirm that hibernation is enabled as a sleep state inside the guest. The expected output for the guest will look like this.

 

deployment-guides-citrix-azure-powermanagment-troubleshooting3.png

 

If hibernate is not listed as a supported sleep state, there is a reason associated with it which will help determine why hibernate is not supported. For example, the following would happen if guest hibernation has not been configured for the virtual machine.

 

deployment-guides-citrix-azure-powermanagment-troubleshooting4.png

 

If either the extension, or the guest sleep state reports an error, you need to update the guest configurations as per the error descriptions to resolve the issue. After fixing all the issues, you can validate that hibernation has been enabled successfully inside the guest by running the powercfg /a command - which returns Hibernate as one of the sleep states. Also validate that the AzureHibernateExtension returns to a Succeeded state. If the extension is still in a failed state, then you need to update the extension state by triggering the Reapply virtual machine API.

Quote

Note:

If the extension remains in a failed state, you will not can hibernate the virtual machine.

Commonly seen issues where the extension fails

Issue Action
Page file is in temp disk. Move it to OS disk to enable hibernation.

Move the page file to the C: drive and trigger reapply on the virtual machine to rerun the extension.

IMPORTANT:
The page file of the Windows image can be on the temporary disk, MCS will move it to the C drive during image preparation!

Windows failed to configure hibernation because of insufficient space for the hiberfile. Ensure that C: drive has sufficient space. You can try expanding your OS disk, your C: partition size to overcome this issue. Once you have sufficient space, trigger the Reapply operation so that the extension can retry enabling hibernation in the guest and succeeds.
Extension error message: A device attached to the system is not functioning Ensure that C: drive has sufficient space. You can try expanding your OS disk, your C: partition size to overcome this issue. Once you have sufficient space, trigger the Reapply operation so that the extension can retry enabling hibernation in the guest and succeeds.
Hibernation is no longer supported after Virtualization Based Security (VBS) was enabled inside the guest. Enable Virtualization in the guest to get VBS capabilities along with the ability to hibernate/deallocate the guest. To enable virtualization in the guest, refer to this document
Enabling hibernation failed. Response from the powercfg command. Exit Code: 1. Error message: Hibernation failed with the following error: The request is not supported. The following items are preventing hibernation on this system. The current Device Guard configuration has disabled hibernation. An internal system component has disabled hibernation. Enable Virtualization in the guest to get VBS capabilities along with the ability to hibernate/deallocate the guest. To enable virtualization in the guest, refer to this document.

Windows Guest Unable to Hibernate

If a hibernation operation succeeds, the following events are seen in the guest:

 

deployment-guides-citrix-azure-hibernation-faq1.png

 

If the guest fails to hibernate/deallocate, then all or some of these events are missing.

Commonly Seen Issues

Issue Action
Guest fails to hibernate because Hyper-V Guest Shutdown Service is disabled. Ensure that Hyper-V Guest Shutdown Service is not disabled. Enabling this service resolves the issue.
The guest fails to hibernate because HVCI (Memory integrity) is enabled. If Memory Integrity is enabled in the guest and you are trying to hibernate/deallocate the virtual machine, then ensure that your guest is running the minimum OS build required to support hibernation with Memory Integrity. Win 11 22H2 – Minimum OS Build - 22621.2134
Win 11 21H1 - Minimum OS Build - 22000.2295 Win 10 22H2 - Minimum OS Build - 19045.3324

Unable to Resume a Virtual Machine

Starting a hibernated/deallocated virtual machine is similar to starting a stopped/deallocated virtual machine. For errors and troubleshooting steps related to starting a virtual machine, refer to this guide.

In addition to commonly seen issues while starting virtual machines, certain issues are specific to starting a hibernated/deallocated virtual machine.

ResultCode errorDetails
OverconstrainedResumeFromHibernatedStateAllocationRequest Allocation failed. virtual machines with the following constraints cannot be allocated, because the condition is too restrictive. Remove some constraints and try again. Constraints applied are: Networking Constraints (such as Accelerated Networking or IPv6), Resuming from hibernated/deallocated state (retry starting the virtual machine after some time or alternatively stop/deallocate the virtual machine and try starting the virtual machine again).
AllocationFailed Virtual machine allocation failed from hibernated/deallocated state due to insufficient capacity. Try again later or alternatively stop/deallocate the virtual machine and try starting the virtual machine.

Windows guest resume status through virtual machine instance view

For Windows virtual machines, when you start a virtual machine from a hibernated/deallocated state, you can use the virtual machine instance view to get more details on whether the guest successfully resumed from its previous hibernated/deallocated state or if it failed to resume and instead did a cold boot.

  1. When the guest successfully resumes, the virtual machine instance view output is:

     

    deployment-guides-citrix-azure-hibernation-faq2.png

    deployment-guides-citrix-azure-hibernation-faq3.png

     

  2. If the Windows guest fails to resume from its previous state and instead cold boots, then the virtual machine instance view response is:

     

    deployment-guides-citrix-azure-hibernation-faq4.png

     

Windows Guest events while resuming

If a guest successfully resumes, the following guest events are available:

 

deployment-guides-citrix-azure-hibernation-faq5.png

 

If the guest fails to resume, all or some of these events are missing.

To troubleshoot why the guest failed to resume, the following logs are needed:

  1. Event logs on the guest: Microsoft-Windows-Kernel-Power, Microsoft-Windows-Kernel-General, Microsoft-Windows-Kernel-Boot.
  2. On bug check, a guest crash dump is needed.

FAQs

  1. What are the charges for using this feature?

    • Once a virtual machine is placed in a hibernated/deallocated state, you are not charged for the virtual machine, just like how you are not charged for virtual machines in a stopped/deallocated state. You are only charged for the OS disk, data disks and any static IPs associated with the virtual machine.
  2. Can I enable hibernation on existing virtual machines?

    • No, you cannot enable hibernation on existing virtual machines. You can only enable hibernation at the time of creating a virtual machine.
  3. Can I resize a virtual machine with hibernation enable?

    • No. Once you enable hibernation on a virtual machine, you cannot resize the virtual machine.
  4. Can I modify a virtual machine once it is in a hibernated/deallocated state?

    • No, once a virtual machine is in a hibernated/deallocated state, you cannot perform actions like resizing the virtual machine and modifying the disks. Also, you cannot detach any disks or networking resources that are currently attached to the virtual machine or attach new resources to the virtual machine. You can however stop/deallocate or delete the virtual machine if you want to detach these resources.
  5. What is the difference between stopping/deallocating and hibernating/deallocating a virtual machine?

    • When you stop/deallocate a virtual machine, the virtual machine shuts down without persisting the memory contents. You can resize a stopped/deallocated virtual machine and detach/attach disks to the virtual machine. When you hibernate/deallocate a virtual machine, the memory contents are first persisted in the OS disk and then the virtual machine hibernates. You cannot resize virtual machines in a hibernate/deallocated state, nor detach/attach disks and networking resources to the virtual machine.
  6. Can you disable hibernation?

    • No, you cannot disable hibernation on a virtual machine.
  7. Can I initiate hibernation from within the virtual machine?

    • To hibernate/deallocate a virtual machine you can use the Azure Portal, CLI, PowerShell commands, SDKs, APIs, and Citrix Autoscale. Triggering hibernation from inside the virtual machine will still result in your virtual machine being billed for the compute resources.
  8. When a virtual machine is hibernated, is there a capacity assurance at the time of starting the virtual machine?

    • No, there is no capacity assurance for starting hibernated/deallocated virtual machines. In rare scenarios if you encounter a capacity issue, then you can try starting the virtual machine later.

deployment-guides-citrix-azure-powermanagment-cloudprereq1.png

deployment-guides-citrix-azure-powermanagment-azureprereq1.png

deployment-guides-citrix-azure-hibernation-posh.png

deployment-guides-citrix-azure-powermanagment-azurevm1.png


User Feedback


Very helpful document!  Thank you!  Looks like many hours spent.  I have 2 or 3 suggestions.

I think maybe the title should be changed.  The article is all about hibernation but hibernation is not mentioned.  Also the title mentions that this is for Powershell but the instructions have both Powershell and GUI versions throughout.  Maybe "Using hibernation with Azure VMs in Citrix DaaS" or something?

A second thing is that many of the images are hard to read, too small when seeing them in the web page and then not enough quality to read them if I enlarge.   The screen by screen for the Create Machine Catalog Wizard is the main one that I had a bad time with.

Lastly, perhaps this is a wider site issue but the navigation columns on the right and left of the web page are so prominent that the content is only in the center third of my screen.

 

Share this comment


Link to comment
Share on other sites



Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...