Azure VMware Solution Architecture and Deployment Guide

The goal of this document is to provide architecture guidance to enterprises that are moving towards deploying Citrix Virtual Apps and Desktops (CVAD) to Azure VMware Solution (AVS). This document covers the following topics:

Architecture
Deployment Guidance
Scalability Guidance
Migration Guidance

This document provides general guidance and best practices that are specific for moving your Citrix Virtual Apps and Desktops workload into AVS.

Architecture

The architecture consists of Citrix Virtual Apps and Desktops Service (Citrix DaaS), Microsoft Azure, and Azure VMware Solution (AVS).

Citrix Virtual Apps and Desktops Service

The Citrix Virtual Apps and Desktops Service secures the delivery of Windows, Linux, Web or SaaS applications and desktops to any device. The delivery of secure desktops and applications empowers today’s modern digital workspace. Citrix Virtual Apps and Desktops provides advanced management and scalability. Citrix can deliver a rich multimedia experience over any network and offers self-service applications with universal device support. Citrix supports a full range of mobile endpoints—including laptops, tablets, smartphones, PCs, and Macs.

Available as a hybrid cloud solution, Citrix DaaS allows you to choose the workload deployment option that best aligns with your enterprise cloud strategy. When deployed on Azure, Citrix DaaS gives IT departments the flexibility of delivering infrastructure services with the elastic scale of the public cloud. This elasticity allows you to easily extend and integrate current investments from on-premises environments into Azure.

The Citrix Cloud Virtual App and Desktop Service was used for the control and management of the test workloads. The numbers focus specifically on the scalability and performance of a single virtual machine instance running Citrix’s Server OS Virtual Delivery Agent (VDA).

Microsoft Azure and Azure VMware Solution

Microsoft Azure is a reliable and flexible cloud platform that allows applications to be quickly deployed across Microsoft-managed data centers. By provisioning Citrix DaaS workloads on Azure Cloud Services, businesses can avoid expenses for internal infrastructure. Instead you can rely on Microsoft to supply the compute, networking, and storage resources for user workloads.

AVS has the following limits:

Clusters per private cloud: 12
Maximum hosts per cluster: 16
Minimum hosts per cluster: 3
Maximum hosts per private cloud: 96
vCenter per private cloud: 1
vSAN capacity limit: 75% of total usable space

Value of Citrix on AVS – Choice and ease of migration, familiarity

Azure VMware Solution is a managed VMware Software Defined Data Center (SDDC) that contains vSphere clusters built on dedicated bare-metal Azure servers. The minimum deployment is three nodes (hosts), with a maximum of 16 nodes per SDDC cluster. All SDDC deployments include a vCenter server for administration, vSAN with Azure-backed storage, vSphere hosts and NSX-T for networking. The SDDC is connected to your Azure tenant via an internal Express Route connection which is included in the pricing for the service. Microsoft manages the infrastructure and software, you manage the workloads running on the SDDC.

Since AVS is a full VMware deployment, you can use the same skills and processes to manage the AVS deployment as you do your on-premises VMware data center. Use of Azure VMware Solution (AVS) provides the following advantages:

Cloud Migration Support: Easily migrate desktops and applications between VMware deployments including replication of up to 500 VMs simultaneously
Familiar Administration: Administrators are familiar with the VMware interface and are comfortable with the administration tools and interfaces
Data center extension: Use burstable capacity and remote locations for periods of high demand, disaster recovery or business continuity

Use Azure VMware Solution (AVS) to provision your Citrix VDA workload just as you provision it with vSphere in your on-premises data center.

Architecture and Build Guidance

This section outlines the AVS environment architecture and provides high-level guidance. Here we show you how to build a Citrix environment within the AVS SDDC and connect your on-premises data center to the Citrix Cloud.

Architecture Diagram

The diagram below shows the architecture of an AVS environment with Citrix VDA Hosts in the AVS SDDC. This architecture uses an ExpressRoute connection back to your on-premises data center.

Figure: Azure VMware Solution ExpressRoute Architecture

If an ExpressRoute connection is not available to your on-premises data center, then use a site-to-site VPN and a Virtual WAN to connect with the AVS SDDC. The figure below shows the site-to-site VPN architecture.

Figure: Azure VMware Site-to-Site VPN Architecture

Citrix Virtual Apps and Desktops service supports AVS. Azure VMware Solution provides cloud infrastructure containing vSphere clusters created by Azure infrastructure. Use Citrix DaaS with AVS for provisioning your VDA workload in the same way that you use vSphere in an on-premises environment.

Citrix Build Guidance for AVS

The detailed deployment of the AVS Solution on Azure can be found here. This section contains the high-level steps and other information to help you migrate successfully.

The deployment consists of the following steps:

Request a host quota:
- Completed through an Azure Portal Support Request.
- Requires Azure Enterprise Agreement.
- Takes up to 5 days to complete.
Register the Microsoft.AVS resource provider:
- Completed through the Azure Portal Subscription List.
Network Checklist
Create an Azure VMware Solution private cloud:
- Created through the Azure Portal new resource (Azure VMware Solution).
- Recommended that you use a new resource group to hold the AVS resources.
- Use a minimum of a /22 CIDR block for the IP address space and it cannot overlap with any existing address space within your network.
- Do not specify a Virtual Network now, you add the ExpressRoute (ER) later.
- Takes 3-4 hours, and new hosts take 30-45 minutes to add to the cluster.
Access an Azure VMware Solution private cloud:
- Create a Windows VM that can be used as a bastion host to access the environment.
- Use the bastion host to connect to the AVS vCenter and NSX-T Manager to continue the configuration and setup.
- Enable DHCP in NSX-T so the VMs can get IP addresses at startup automatically:
  - Allow NSX-T to host your server or add a DHCP Relay address.
  - Set up a Network Segment in NSX-T, which must be on the Azure Gateway subnet.
  - Configure DHCP and DNS settings.
Connect the on-premises data center to Azure
- If using an ExpressRoute connection, connect it to the Virtual Gateway used by the AVS ExpressRoute.
- If using a site-to-site VPN or point-to-site VPN, create a Virtual WAN following the instructions here.
  - Create a Virtual Hub.
  - Connect ExpressRoute from AVS to the Hub.
  - Connect point-to-site or site-to-site VPN to the Hub.
  - Link the connections and verify that routes are being propagated.
Verify Azure VMware Solution environment:
- Ping through and verify the connectivity between the on-premises data center and the AVS SDDC.
- Ping between Azure subnets and AVS SDDC.
Enable AVS VMs to access internet.
Deploy Citrix Infrastructure VMs:
- Domain controller to join the on-premises domain.
- Two Citrix Cloud Connectors which link the AVS environment to the Citrix Cloud.

Best Practice Recommendations

When building the AVS SDDC, the following best practices and recommendations are provided.

VMware HCX requires an ER connection between the on-premises data center and Azure. The ER connection can then be connected to a Virtual WAN. Connecting both the on-premises ER and the AVS SDDC ER to the Virtual WAN creates a WAN bridge.

AVS gateways require 4-byte Autonomous System Numbers (ASNs).
AVS resources are not granted internet access by default.
AVS requires a minimum /22 RFC 1918-compliant subnet range that does overlap with anything in the cloud or on-premises.
An Azure VNet is required along with a gateway subnet (/27 or larger) named GatewaySubnet.
Define two other subnets: One subnet for the bastion host. One subnet for the Azure Bastion service that must be named AzureBastionSubnet.
Any workloads running on AVS require both DHCP and DNS services. AVS is not able to provide these services automatically.
Reduce costs by purchasing with 1-year and 3-year pricing.
The following resources incur charges:
- Virtual Network Gateways
- ExpressRoute Circuits
- Azure Bastion Services
- Azure Virtual Machines
- Azure Virtual Machine Disks
- Public IP addresses
- Network traffic egressing AVS to on-premises data centers or outside of the AVS region

Scalability Testing

To determine the capacity of the 3-node cluster, we loaded each AVS node with Citrix shared desktop hosts then ran two workloads. The two workloads that we used for testing are described below.

Task Worker Workload: This workload includes segments with Microsoft Office 2016 Outlook, Word, Excel, and Internet Explorer, Adobe Acrobat and PDF Writer. The Task Worker workload does not place a high demand on the environment and represents users that do not access the system heavily.
Knowledge Worker (2 vCPU) Workload: This workload includes the Task Worker workload plus PowerPoint 2016, FreeMind, PhotoViewer, Doro PDF Writer and several 360p movies. The Knowledge Worker workload places a higher demand on the environment and represents users that access the system heavily.

The dual vCPU Knowledge worker profile provides the lower bound for user density. The lightweight single vCPU Task worker profile provides the upper bound for user density. These two bounds provide the expected range for user density.

Login VSI is the industry-standard in benchmark and load testing for the end-user computing and application markets. Login VSI's workloads are developed specifically for VDI and server-based computing solutions from Citrix, Microsoft, and VMware. Login VSI is used to right-size production environments at the lowest cost while maximizing desktop and application performance. This test environment used Login VSI version 4.1.40 to create simulated task worker and knowledge worker sessions against a VMware 3-node cluster. For more information about Login Enterprise see https://www.loginvsi.com

The number of users successfully completing their sessions provides a key performance indicator under real-world conditions. This value, referred to as the VSImax session count, is used for the comparative analysis. At the beginning of the test a baseline value is taken with only a single user on the system. The Login VSI workloads calculate the VSImax session count by observing when the user response time has significantly increased above the baseline value.

Lab Environment Setup

The Citrix VDA hosts were built in the AVS environment on the 3-node cluster. Each cluster node was a Dell PowerEdge R640 server with 18-core Intel Xeon Gold 6140 2.30 GHz CPUs and 512 GiB RAM. The Citrix Cloud Connectors and the Login VSI launchers were installed in the Azure virtual network. The Citrix Workspace clients on the launchers connected to the VDA hosts over the internal ExpressRoute connection. The diagram of the lab setup is shown below.

Figure: Login VSI AVS Lab Environment

All test runs used the default installation settings for Citrix policies, Windows Server 2019, Office 2016, and Windows Defender.

Methodology

All testing used the hosted shared desktop model with Windows Server 2019 and the Citrix Server OS VDA version 2106. We first looked at single-server scalability (SSS) to identify the most efficient vCPU and memory configurations. The SSS results were used to select the best configuration for maximizing the users on the 3-node cluster. After we determined the best approach for maximizing the users on the system, we used Login VSI to generate the maximum load possible.

These tests help identify the approximate number of users to expect on a 3-node cluster. The results vary and are heavily influenced by the workload. We strongly urge you to test your own workload and select a configuration that works best for your workload.

Scalability Test Results

This section covers the scalability testing performed on AVS within the lab environment. We used LoginVSI to determine the expected number of users for three popular VDA host configurations. Here are the results for the Task Worker profile:

Configuration	VSImax Users	Users per core
4 vCPUs 16 GiB RAM	18	4.5
8 vCPUs 16 GiB RAM	30	3.75
8 vCPUs 24 GiB RAM	34	4.25

Table: VSImax Single Server Scalability Results

The results from our SSS testing indicate the most efficient use of compute resources was the 4vCPU x 16 GiB RAM configuration. We used this information to determine the VM density for loading the 3-node cluster. We built 50 4vCPU VDA hosts using 200 vCPUs. We then ran a cluster load test to determine the scalability of the cluster in this configuration. The results of the load test are shown in the table below:

Configuration	Task Worker	Knowledge Worker
Single Server (4xCPU 16 GiB RAM)	18	13
50 VMs (Citrix VDAs) on a 3-node cluster	602	378

Table: VSI Max score by Profile Type for Server 2019 4vCPU with 16 GiB RAM

The blog “Rule of 5 and 10” provides SSS guidance for on-premises data centers. Our 4vCPU results are consistent with his recommendations of about 10 CVAD session users per physical core. With 4 vCPUs we were using 2 physical cores (4 hyperthreaded cores) for each Citrix session host. Applying the rule of 10, gives us about 20 users per VM, and a VSImax of 18 is within the margin provided in the blog. These results are expected since AVS is using the same VMware nodes you expect in an on-premises data center.

In line with previous scalability testing, a single VM does not always scale linearly when multiple VMs are running in parallel on the same host. With the Task Worker, we were expecting to get closer to 900 (18 * 50) users but VSImax showed 602. With the Knowledge worker we were expecting to reach around 650 (13 * 50) but the most we observed was 378.

NOTE: All the sessions in the run were able to get launched and logged on. However, the user experience declined far enough that the VSImax recommended user count was about two-thirds of the session count.

Scalability Analysis

In this section, we provide the VMware performance graphs for the time when the test runs were being executed. We show both a Windows server VM and a cluster node to provide a point of reference. LVS-SVR-25 represents the 50 Windows servers while ESX17-R16 represents the cluster nodes. We review the four most common areas where performance bottlenecks can be found, CPU, Memory, Disk, and Network.

CPU

The CPU graphs show an average CPU utilization based on available capacity for the both the Windows server VM and the cluster node. The single server peaks below 85% and the node peaks around 97%. Both of these graphs though are reporting the average CPU utilization, not the maximum CPU utilization, which was not available in the VMware console.

Figure: Single Server CPU Utilization

Figure: Cluster Node CPU Utilization

We can get a better picture of what was happening by looking at two of the VSI Max performance indicators. The Zip Low Compression (ZLC) and Zip High Compression (ZHC) indicators are tied to CPU intensive actions. On the task worker profile, we see the operations go from a baseline of around 1 second to almost a maximum 13 seconds. The majority of that delay doubling after the 600-user mark where the maximum was 6 seconds. A similar increase in delay is seen on the knowledge worker profile when the user count exceeds 380. This behavior leads to the conclusion that the CPU was indeed reaching its full capacity. Typically, the bottleneck for this type of workload is the processor.

Figure: Task Worker Zip Compression Performance

Figure: Knowledge Worker Zip Compression Performance

Memory

The memory utilization for both the single server and the cluster node is as expected. Both are using around 75%-80% of the available memory, which is the sweet spot for scalability. No unexpected findings here.

Figure: Single Server Memory Utilization

Figure: Cluster Node Memory Utilization

Disk

Similar to the memory utilization, the disk utilization is as expected. The single server shows disk latency up to 1 ms, while the cluster node peaks at 5 ms, both of which are within expected performance parameters.

Figure: Single Server Disk Utilization

Figure: Cluster Node Disk Utilization

Network

The network traffic is the last metric to review. The single server shows the peak during testing of about 2000 Kbps of traffic and the cluster node shows the peak under 100,000 Kbps.

Figure: Single Server Network Utilization

Figure: Cluster Node Network Utilization

At the beginning of the project, concern existed around the internal ExpressRoute (ER) connection between AVS and Azure. Specifically, the team was concerned the circuit might become overloaded when the clients in Azure used the ER connection to the AVS SDDC. We used Azure Monitor to view the Packets per second (PPS) and Bits per second (BPS) under our heaviest load of 850 users.

The results we observed during the load test are encouraging. The 850 users were generating about 17,240 packets per second at throughput of under 50 Mbps. Since a standard ExpressRoute connection handles around 100,000 PPS and 1 Gbps of throughput, the ER connection should support about 4000 users comfortably.

Figure: Express Route Packets per Second

Figure: ExpressRoute Bits per second

Scalability Recommendations

The analysis shows that the CPU processing power was the bottleneck in our testing. The other three metrics reviewed, (memory, disk, and network), were all within the acceptable range. The CPU processing power is typically the limiting factor on scalability. In a 3-node system with 72 cores per node, we had 216 total cores available. Reserving some compute capacity for the hypervisor and infrastructure VMs, about 200 vCPUs are available for workloads.

Based on these test results, a good rule of thumb to follow for planning your deployment on AVS is 3.5 users per vCPU for a light workload and about 2 users per vCPU for a heavy workload.

These are ball park numbers generated in a controlled lab. They represent only an approximate number of users to expect on a 3-node cluster under the two workload profiles tested. We strongly urge you to test your own workload and use that data to decide the best configuration for your workload.

Migration Guidance

The first step is to migrate your Citrix infrastructure to the Citrix Cloud. The second step is to migrate your VMware workloads to AVS. Using VMware’s HCX appliance is the recommended approach to moving VMware worklaods between the on-premises data center and the AVS SDDC.

Citrix Infrastructure

The complete set of detailed instructions for a VMware to Azure migration can be found here. This link does presume you are migrating to Azure infrastructure instead of Azure VMware Solution. When migrating Citrix infrastructure to AVS, here are the key steps to include in the migration plan:

Set up a Citrix Cloud Environment
1. Obtain a Citrix Virtual Apps and Desktops Service subscription
2. Deploy two Cloud Connectors to the on-premises VMware environment
3. Rename the Resource Location
Migrate the Citrix Virtual Apps and Desktops on-premises to Citrix Virtual Apps and Desktops service
1. Establish Host Connection
2. Enable Citrix Cloud API access
3. For MCS, replicate your VM configurations to AVS
  - Create Machine Catalogs
  - Create Delivery Groups
  - Migrate Applications and Policies using the Auto Config tool
4. For PVS, use the Auto Config tool to import the PVS information
  - Verify that the Machine Catalogs are created correctly
  - Verify that the Delivery groups are present
  - Migrate Applications and Policies using Auto Config tool
Workspace Environment Management on-premises to Workspace Environment Management service
1. Verify system requirements
2. Switch to Service Agent mode
On-premises StoreFront and Citrix Gateway to Citrix Workspace and Citrix Gateway service
1. Set up a Workspace Configuration
2. Configure authentication
3. Grant access to users

Migrating Citrix Workloads

To migrate the workloads between an on-premises data center and a cloud-hosted VMware implementation, the best approach is to use VMware’s migration tool HCX. HCX’s core function is to transparently migrate virtual workloads between VMware vSphere environments.

NOTE: HCX Requirements are stringent, including an ExpressRoute connection between the on-premises data center and AVS deployments. See the full list of requirements here.

HCX requires several pairs of appliances, one each in the on-premises data center and one in the AVS SDDC.

Manager: responsible for providing all the management functionality of HCX
WAN Interconnect Appliance: responsible for facilitating workload migration and replication
WAN Optimization Appliance: responsible for WAN optimization and data duplication used by the WAN Interconnect Appliance and requires storage capable of 2500 IOPS
Network Extension: responsible for extending the layer 2 network between vSphere environments
Proxy Host: responsible for providing a local target for migrations and vMotion activities that the WAN Interconnect Appliance uses

Planning Citrix Workload Migration

Here are the high-level steps to planning the workload migration. More detailed information for these steps can be found on VMware’s site here.

Generate an inventory of workloads to migrate, this includes:
1. Citrix VDA hosts, Citrix PVS Server and its database.
2. File Servers that host user and application data.
3. Understand networking and performance requirements.
Generate an inventory of workloads that cannot be migrated.
1. Workloads on non-x86 platforms such as mainframes.
2. Understand the integration and dependencies with Citrix applications that are migrating.
Develop a strategy for IP addresses:
1. Ideally, the Citrix workloads keep their existing IP addresses and the Network Extension appliance can be used to make the cloud transition.
2. Workloads that cannot keep their IP addresses need an IP migration plan for the transition. These include any workloads on subnets where the whole subnet is not migrating to the cloud.
Understand all traffic flows:
1. Egress traffic from the Citrix servers to an on-premises data center is charged. Migrate systems with their data.
2. Understand which systems should transfer together. For instance, PVS and its SQL database or all the Citrix servers that share a base image.
3. Account for the additional replication and network extension traffic.
Generate a topology diagram with all the on-premises network devices in the migration path between the on-premises data center and the AVS cluster:
1. Setup monitoring on network devices. Watch for situations where the bandwidth decreases or the latency increases significantly.
2. Watch for bottlenecks in the migration traffic. This monitoring is especially key when moving high-utilization servers in hot or warm mode. High-utilization servers include the PVS server or the SQL database server.
3. HCX requires a minimum of 100 Mbps connection between the sites.
Understand all external storage devices used by the workloads:
1. Identify any applications that rely on the external storage devices.
2. Identify applications that rely on physical external devices such as dongles for licensing.
Review utilization levels for workloads and plan migrations during the times when the workloads are at the lowest utilization. This practice reduces the user impact.
Plan migration waves with the following in mind:
1. Keep interdependent groups of applications as close as possible to one another, such as Citrix workloads that share a base image.
2. Isolate large complex applications to their own waves.
3. Networking traffic increases during the migrations.
4. Keep workloads between 25-50 servers to reduce network traffic.
5. Monitor storage IOPS on source and destination.

Citrix Workload Migration Process

Once HCX is deployed and the migration planned, the next step is to migrate the Citrix workloads. More detailed information on this process can be found here:

Ensure that HCX appliances have sufficient resources (CPU and Memory). Verify HCX appliances are not included in any Distributed Resource Scheduler (DRS) automation.
Configure the High HA Restart policy for all HCX appliances.
The WAN Interconnect Appliances replicate workloads over dedicated IPsec tunnels.
The Network Extension Appliances provide layer 2 networking extensions over dedicated IPsec tunnels.
The extended networks are isolated within the SDDC until after migration is complete.
The extended network is dismantled and normal routing is restored via SDDC.

Three types of migration processes are available:

Cold Migration: Migrate the VMs while they are powered off.
Bulk Migration: Replicate workloads in waves to the cloud and then power off on-premises copy while simultaneously powering on the cloud workload. This approach is recommended for the PVS Server and its SQL database when they cannot be migrated via Cold Migration.
HCX vMotion: Standard vMotion or Replication Assisted vMotion (restrictions apply) This approach is recommended only when Cold Migration and Bulk Migration are not an option.

To migrate your Citrix deployment, use the following order:

Migrate PVS and MCS images (files and folders via data migration).
Migrate PVS Server and Database using HCX Bulk Migration.
Recreate the Machine Catalogs and Delivery Groups using the new AVS VMs.
Reboot the servers in the cloud to bring them online.
Post Migration cutover decommission the on-premises resources.

Best Practice Recommendations

When migrating to Azure, the following best practices and recommendations are provided to assist you in planning the transition.

VMware Site Recovery Manager (SRM) requires an ExpressRoute connection between the on-premises data center and the AVS SDDC.
If deploying more than one cluster, place VMware management VMs on the first cluster (NSX-T, vCenter, HCX).
HCX is able to help with the following use cases:
- Migrate workloads between legacy versions and modern versions of vSphere.
- Migrate non-VMware environments to vSphere.
- Migrate workloads without requiring IP address changes.
- Migrate workloads over a WAN optimized links with data deduplication enabled.
- Migrate workloads securely by using NSA Suite B cryptography.

Conclusion

We have provided high-level guidance to migrate your on-premises Citrix Deployment to the Azure VMware Solution in Microsoft Azure. The URL links throughout the guidance and within the References section offer detailed steps for the migration. If your enterprise is already running VMware vSphere, moving to AVS is the easiest path for migration. If your enterprise is running another virtualization solution on-premises, the VMware HCX appliances and AVS can still provide you a smooth transition to Azure.

Azure VMware Solution Architecture and Deployment Guide

Architecture

Citrix Virtual Apps and Desktops Service

Microsoft Azure and Azure VMware Solution

Value of Citrix on AVS – Choice and ease of migration, familiarity

Architecture and Build Guidance

Architecture Diagram

Citrix Build Guidance for AVS

Best Practice Recommendations

Scalability Testing

Lab Environment Setup

Methodology

Scalability Test Results

Scalability Analysis

CPU

Memory

Disk

Network

Scalability Recommendations

Migration Guidance

Citrix Infrastructure

Migrating Citrix Workloads

Planning Citrix Workload Migration

Citrix Workload Migration Process

Best Practice Recommendations

Conclusion

References

Table of contents

User Feedback

Create an account or sign in to comment

Create an account

Sign in