These applications have application data that must be accessible regardless of where the application resides.
Here are the questions that you need to answer about Applications and Application Data:

How do I integrate the VDA-hosted applications with Microsoft 365?

• Use Office 365 ProPlus when installing Office

• Microsoft 365 requires a plan that supports shared computer activation, which is required for any multi-user session hosts

• After installing Office on a golden image, do not open any Office applications. If you open one Office application, you must reset the image to remove the temporary key which prevents user-level activation. To reset the image, uninstall Office, reboot and then reinstall Office.

• For earlier versions of Office that use KMS licensing, such as Office 2010 and 2013, you must verify that your KMS server is reachable from the Azure cloud. You can make your KMS server accessible to your Citrix workloads by one of these methods:

  • Migrate your KMS server to the Azure cloud
  • Connect your on-premises data center to Azure using either an ExpressRoute or a Site-2-Site (S2S) Virtual Private Network (VPN)

• When using FSLogix and Office 365 containers, follow these steps to integrate it with Windows Search, rebooting between each step:

  • Configure Automatic Startup (not delayed) for the Windows Search service. This configuration must be completed before installing Office so Office sets the required hooks.
  • Install Microsoft Office.
  • Install the FSLogix agent.
  • If you do not need Windows Search, you can disable the service. Before disabling the service, go ahead and complete the install steps to save on compute resources, then disable the service. With this approach, if later it is needed, you can enable the service easily.

Should I use FSLogix with Citrix workloads?

• Use Microsoft GPOs to manage all Microsoft 365 Office settings

• Microsoft FSLogix is the recommended approach to handle Microsoft 365 integration. It handles the Outlook Search, Outlook PSTs and Office activation seamlessly.

• Microsoft recommends using SSO (ADFS) with Microsoft 365 Apps 1704 and above:

  • When ADFS is available, enable the "Automatically activate Office with federated organization credentials” GPO and configure the automatic logon in GPO Security Logon
  • if ADFS is not available, use FSLogix or Citrix Profile Manager to synchronize the following registry key %localappdata%\Microsoft\Office\16.0\Licensing to roam the Microsoft 365 token with the user

How should I configure Outlook? (Cached mode or online mode)?

• Use Cached Exchange Mode when the following conditions are true:

  • A Profile Management solution such as FSLogix or Citrix Profile Manager is available to manage the OST file and the search index
  • Users require a more responsive email system
  • Connections between the Outlook client and the mail server have high latency or are frequently disrupted

• Use Online Mode when the following conditions are true:

  • Low latency network connection is available

• Use Active Directory Group Policy to configure Outlook Exchange mode, recommended settings include the following:

  • File > Cached Exchange Mode
  • Sync Settings
  • Disable Fast Access
  • Use Cached Exchange mode
  • Cache file

What settings should I use for Microsoft 365 when using Citrix Profile Management?

• When using Citrix Profile Management use these items to provide a robust user experience and support the OST/PST storage locations and the Search Index locations

  • Use the latest version of Citrix Profile Manager. The latest version has features such as Native Outlook Search and Large File Handling which provide optimizations for Outlook.
  • Enable Large File Handling to allow storing OST/PST files on Azure Files.
  • Include these folders and registry in the Citrix Profile Management configuration:
    • %localappdata%\Microsoft\Office\16.0\Licensing
    • %localappdata%\Microsoft\Credential
    • AppData\Local\Microsoft\Credentials
    • AppData\Local\Microsoft\Windows\WebCache
    • AppData\LocalLow\Microsoft\CryptnetUrlCache
    • AppData\Local\Microsoft\Outlook
    • AppData\Local\Microsoft\Vault
    • AppData\Local\Microsoft\Office
    • AppData\Local\Microsoft\Office\*.qat
    • AppData\Local\Microsoft\Office\*.officeUI
    • AppData\Local\Microsoft\Windows\UsrClass.*
    • HKCU\Software\Microsoft\Office\16.0\Common\Identity\DisableADALatopWAMOverride

Where should I store my application data?

• Use Azure Migrate or Movere to assess and plan the application migration

• Check with the application vendors to determine if the software is supported in Azure. If planning to use PVS for streaming, also verify that the vendor supports Gen 2 VMs.

Where should my data be located relative to my application?

• Data leaving Azure incurs an egress charge. Try to keep the applications and their data as close as possible to one another. Although ideal, this configuration is not always possible. When you cannot keep your data close to the application, focus on minimizing the latency between them.

What costs should I consider when determining my data location?

• When working in a hybrid cloud environment that prevents both the application and its data from moving to Azure together, move the application first then move the application data. With this approach, the data egress charges are reduced.

How do I integrate Citrix Workspace with Microsoft 365 and Microsoft Teams?

• For multi-session hosts, install Microsoft Teams after the VDA is installed on your golden image and install it under c:\program files using the ALLUSER=1 flag

• Updates to the Microsoft Teams agent require an uninstall of the previous version before installing the new version

• Set the Citrix Microsoft Teams redirection policy to allowed

• Microsoft Teams relies on Azure Transport Relays the following ports and IP address ranges must be accessible

  • UDP 3478–3481
  • TCP 443
  • 137.106.64.0/18
  • 52.112.0.0/14
  • 52.120.0.0/14

• Use Citrix Director’s Activity Manager to monitor Microsoft Teams applications such as WebSocketAgent.exe, WebSocketService.exe, and CtxSvcHost.exe

Links to Other Resources

Citrix Optimization for Microsoft Teams on macOS

Deploy Microsoft 365 Apps as part of an operating system image

Deploying Azure Files for Citrix Profile Management and Citrix User Personalization layers

FSLogix Office 365 Container is Now Citrix Ready

Microsoft 365 network connectivity principles

Microsoft Teams optimization in Citrix Virtual Apps and Desktops environments

Monitor, troubleshoot, and support Microsoft Teams

The goal of the document is to help answer FAQs on Autoscale to achieve the best cost optimization. It provides guidance for configuring Autoscale for different admin use cases and their infrastructure and technical requirements. This document keeps evolving as more use cases and capabilities are introduced within Autoscale.

Use cases related questions

Can an admin configure Autoscale to first exhaust on-premises resources and only then on-demand burst to the cloud?

• The Autoscale selective power management feature is useful here. The admin must tag the cloud workloads within the delivery group that will be used for burst. Then configure Autoscale to power manage only these tagged workloads. The admin can then use zone preference to prefer on-premises workloads (which are always powered on) for incoming requests. After this configuration, incoming requests will first be brokered to on-premises resources until they are fully used. Cloud resources will be powered on only after the on-premises capacity is used.

Can an admin use their on-premises resources for normal operations and Cloud resources only in a business continuity or disaster recovery scenario?

• Use the preceding configuration

Can an admin configure Autoscale for users ramping up slowly / seasonal users / not worry about capacity planning upfront?

• When an organization has users onboarding slowly and does not want to provision the entire capacity ahead of time, then the dynamic provisioning feature with Autoscale can help. This feature constantly evaluates the load on the delivery group and creates or deletes machines accordingly. An admin must define the high and low watermarks for load on the delivery group to create machines. Admins can optionally define the maximum number of machines to be provisioned and tags to be applied to these machines.

How can the admin of an university with varying schedules and lectures during the day configure Autoscale?

• The admin must understand the capacity utilization and lecture schedule and feed it into Autoscale schedule-based scaling to step up and step-down capacity during the day. Autoscale allows admins to vary the powered-on capacity with 30 min granularity and allows for different schedules on different days. If a volatile workload is expected, it is advised to configure the capacity buffer to avoid users having to wait for machines to power on.

How can the admin of an outsourcing firm with agents logging in at fixed timings configure Autoscale?

• Use schedule-based scaling depending on the shift schedules. Autoscale has 30 mins granularity. Admins can configure Autoscale to power on different workloads at separate times of the day. If the number of users is unexpected, then admins also define the buffer capacity. Some platforms can take several minutes to switch on machines so do factor that while creating these schedules.

How can the admin configure Autoscale if they are using heavy machine instances?

• If the organization provides static desktops to its users, there might be a use case where the admin wants the machines to be started when the user clicks the desktop icon (say for knowledge workers who occasionally work on those large machines) but shut them down at a specific time of the day say at 6 PM. In this case the admin can define the property AutomaticPowerOnForAssigned to false.

How can an admin configure Autoscale to save storage costs apart from compute?

• With dynamic provisioning machines can be created and deleted using Citrix Machine Creation Services. This provides extra storage cost savings by de-allocating the disk when the machine is not needed.

Infrastructure questions

How can the admin choose the number of reserved instances (RI) to buy?

• Observe the capacity using data under Monitor > Trends > Machine Usage over a time period (that covers all variations such as seasonality, work force changes and so on.) in the environment. The minimum capacity that is powered on for this time period can be a good indication of the minimum reserved instances you might buy. Some other parameters that define this are:

  • Is there a fixed schedule say 9 AM - 5 PM? That schedule would mean that machines can be shut down 66% of the time and achieve close to reserved instances cost savings of ~70%.
  • More RIs than the minimum capacity usage can be bought depending on workload usage and savings. For instance, say if there are 10% more machines running more than 50% of the time.
  • The organizations expansion plan must also be factored in.

• Using Reserved Instances can save up to 70% costs compared to pay as you go machines over a three year duration.

How can the admin choose the number of burstable instances to purchase?

• For virtual desktop users with high memory consumption (For instance applications such as Chrome browser, Office applications) but low CPU consumption, admins can consider using burstable VMs. These VMs might be half the cost of their peer machines (as is the case in Azure for B series and D series). In Citrix Director, check Resource utilization and average CPU consumption and match it with the base CPU% guarantee of the burstable VMs. It is recommended to use these machines for single session machines only. The reason for this is that the impact is limited compared to a multi-session machine where many users can be impacted if the CPU gets throttled.
• This mechanism can be used to lower costs by ~50% from the regular pay as you go model.

How can the admin choose the right instance size?

• Admins can optimize costs if they right size the machine instances in public clouds. Smaller instances host fewer user sessions than larger instances. Therefore, Autoscale powers off machines faster because it takes less time for the last user session to be logged off, thus reducing costs. Citrix recommends provisioning smaller instances if they match workload performance and capacity requirements. An effective way to understand this would be to have no more than 15-20 sessions per machine.

What scale limits can the admin operate under?

• It depends on the limits of the chosen cloud platform and the provisioning technique. For example, if the cloud platform is Azure and the provisioning technique is MCS (Machine Creation Services) then the guidance is ~1200 machines per subscription.
• It is also recommended to evaluate other limits like # of machines in a resource location and hosting connections. Visit the Limits page for more info. This is an evolving page.

Technical questions

How can the admin start new machines based on session count?

• Look at the load index. By default, it is based on session count (250) and is the factor considered for powering on more machines.

How does Autoscale turn off machines?

• Autoscale always attempts to scale down the number of powered-on machines in the delivery group to the configured pool size and capacity buffer. It does so by putting the excess machines with the fewest sessions into “drain state” and powering them off when all sessions are logged off. This occurs when session demand lessens, and the schedule requires fewer machines than are powered on. More info.

How does Autoscale turn on machines?

• It powers on machines based on the schedule and load-based settings. If the current machines cannot accommodate the incoming load and respect the capacity buffer configured, then it will power on more machines. See this working example for a better understanding of how it works.

Where do I see machines which are candidates for shutting down?

• Just like maintenance mode, a list of machines in drain state is maintained. If a machine is in drain state, then it is a candidate to be shut down. Once all the sessions are drained off, the machine is shut down. To find out which machines are in drain state, use the PowerShell command: Get-BrokerMachine -DrainingUntilShutdown $true. More info

How do I see my cost savings?

• The Monitor > Trends > Machine Usage page also displays the estimated cost savings achieved by enabling Autoscale in the selected delivery group. Both $ saved and % savings using Autoscale are displayed. These savings can be sent as monthly savings reports to higher management.

A subscription is an agreement with Microsoft to use one or more Microsoft cloud platforms or services. Charges accrue based on either a per-user license fee or on cloud-based resource consumption. Subscriptions can be used to further subdivide the costs or administrative access as required.

Management groups are used within Azure to efficiently manage access, policies, governance, and compliance across subscriptions. They are invaluable for operating multi-subscription tenants in Azure at scale. Each subscription automatically inherits the conditions, policies, and access of its parent management group.

Here are the questions that you need to answer about Azure infrastructure

How many Azure tenants do I need?

• Use a single Azure tenant for the Citrix resources and the users and devices that access those resources

• Use multiple tenants where multiple Azure Active Directories are required. Development/Test having a separate authentication directory or an enterprise that has multiple on-premises AD directory services are two examples.

• The Azure account owner must be associated to the same tenant where the subscriptions for the account are provisioned

• Azure account owners are automatically subscription owners for all the subscriptions in the account

What Microsoft license models should I use?

• Apply the Hybrid Use Benefit (HUB) of your current EA license if it includes Windows Server Software Assurance. HUB significantly reduces compute costs in the cloud. This licensing model can save you up to 40% of the hourly cost because you can use the base VM pricing for Windows Server or SQL Server instances in Azure.

• If using the Microsoft Office suite, use per user licenses that include the Windows 10 Virtual Desktop licenses such as the E3/E5 subscriptions

  • Microsoft 365 E3/E5: Includes Azure Virtual Desktop licenses and Microsoft Office licenses
  • Microsoft 365 Business Premium: Includes Azure Virtual Desktop licenses and Microsoft Office licenses
  • Windows 10 Enterprise E3/E5: Includes Azure Virtual Desktop licenses

How many Azure subscriptions will I need?

• All subscriptions within the same management group must trust the same Azure Active Directory tenant

• A subscription can be associated with only a single account at a time and must have an associated account owner

• Subscriptions cannot share networks, but they can communicate through VNET peering and Azure ExpressRoute

• Subscriptions are boundaries for Azure policies, management, governance and administrative, so plan subscriptions for business units that have separate administrative or billing requirements

• Multiple subscriptions reduce the blast radius and exposure in case credentials are compromised

• Plan to isolate development and test subscriptions from production subscriptions to provide extra performance, security, governance, and compliance

• Some environments such as production and user acceptance testing or preproduction can be shared in a single subscription

• Dedicating subscriptions to Citrix workloads simplify administration and policy management

• Citrix recommends limiting a subscription to 2,500 Virtual Delivery Agents (VDAs)

• Use subscriptions as scale units and scale them out as needed to support the required resources

• Microsoft sets limits on resources within a subscription and those limits must be considered when determining how many subscriptions are necessary to support the Citrix workloads

How many management groups will I need?

• Subscriptions can belong to only one management group at a time

• Management groups are associated with a single parent

• Management groups can be up to 6-levels deep and Microsoft recommends keeping the management group hierarchy as flat as possible

• Management groups are used for policies, not for billing or line-of-business groups. Create management groups based on policy requirements such as instance types, firewall rules, logging, storage, encryption, RBAC model, and so forth

• Limit the number of Azure policy assignments at the management group root, instead of placing them on the individual management groups

• Citrix recommends creating a management group for Citrix workload subscriptions

• Management groups are used for aggregating Azure Policies, so group subscriptions with similar policy requirements together under the same management group

• Use resource tags that can be referenced by Azure policy

For Citrix Cloud to connect and deploy machine catalogs in the Azure cloud, a service principal account is required. That account needs the correct permissions to create, delete, and maintain Citrix resources in each subscription. The service principal account is created through an application registration within the Azure AD tenant. The creation of the service principal account can be created automatically by Citrix or manually by an Azure AD global administrator.

The creation of the service principal object can be accomplished automatically by Citrix if the user running the Citrix Host Connection Wizard has contributor permissions on the subscription. During the host connection setup, the Wizard requests all the required permissions, including contributor permissions on the subscription, and keeps that acceptance for future connections.

Security-sensitive environments do not allow service principals to have contributor permissions at a subscription level. Citrix provides an alternative solution referred to as a Narrow Scope service principal. An Azure AD global administrator needs to manually create an application registration. Then a subscription administrator manually grants the service principal account appropriate permissions. Narrow-scoped service principals do not have contributor permissions on the entire subscription. Their permissions are scoped to just the resource groups, networks, and images that are required to create and manage the Machine Catalogs.

Here are the questions that you need to answer regarding the service principal account:

Should I use a subscription-scope service principal account?

• Requires Azure AD global administrator permissions

• Contributor role for the entire subscription is created automatically and Azure will prompt for permission approval at the initial connection

• Use when information security allows a service principal account to be granted contributor permissions on the entire subscription and Citrix administrators have contributor access to the subscription

• Accounts used for authentication during the host connection creation must be at least co-administrators on the subscription and a member of the Azure Active Directory

• Recommended when subscriptions are dedicated to Citrix resources or the environment will contain many resource groups

• Use when a simple management experience is wanted

• Use when Citrix Studio is used to manage the environment more than PowerShell

• Preferred during proof-of-concept deployments

Should I use a narrow-scope service principal?

• The narrow-scope service principal is created manually by an Azure AD global administrator

• Before running the machine catalog Add Machines wizard, the target resource group must be precreated and granted these permissions:

  • Pre-Created Resource Group: Virtual machine contributor, Storage account contributor, and Disk snapshot contributor
  • Virtual Network: Virtual machine contributor
  • Storage Account: Virtual machine contributor

• Recommended when the number of resource groups is manageable either through the Azure console or through automation

• Recommended for higher-security environments where permissions are tightly controlled and fine-grained access control is prevalent

• Recommended when subscriptions cannot be dedicated to Citrix resources and are hosting other services

• Recommended when Azure administrators have different subscription permissions depending on their role

• For larger environments, consider using build scripts or ARM templates to pre-create resource groups and grant the required permissions

Should I use custom roles for the service principal?

• Citrix recommends the use of custom roles for setting permissions for the service principal when more than one subscription will be used

• Microsoft recommends setting the role permissions at the management group level through the Azure policy

Links to Other Resources

Azure subscription and service limits, quotas, and constraints

Citrix TIPs: Azure Subscription Sizing

Citrix TIPs: Citrix on Azure - Enterprise-Scale Landing Zones - Part 2

Enterprise Agreement enrollment and Azure Active Directory tenants

Management group and subscription organization

Throttling requests for subscription and tenant limits

What are Azure management groups?

Azure Virtual Desktop pricing

Application and service principal objects in Azure Active Directory

How to Grant Citrix Virtual Apps and Desktops Access to Your Azure Subscription

Manually Granting Citrix Cloud Access to Your Azure Subscription

Policies provide the basis to configure and fine-tune Citrix Virtual Apps and Desktops environments. Policies allow organizations to control settings based on various combinations of users, devices, or connection types and include settings for:

• Connections
• Security
• Bandwidth

When making policy decisions, consider both Microsoft and Citrix policies to include all user experience, security, and optimization settings. This article focuses on Citrix policies only. For a list of all Citrix-related policy settings, refer to the Citrix Policy Settings Reference.

Decision: Preferred Policy Engine

Organizations can configure Citrix policies either via Citrix Studio, Web Studio, or through Active Directory group policy. Active Directory policies use Citrix ADMX files, which extend group policy and provide advanced filtering mechanisms.

Using Active Directory group policy allows organizations to manage both Windows policies and Citrix policies in the same location, and minimizes the administrative tools required for policy management. Group policies are automatically replicated across domain controllers, protecting the information, and simplifying policy application.

Use the Citrix administrative consoles if Citrix administrators do not have access to Active Directory policies. Select the method which is most appropriate for the organization's needs and use that method consistently. Using a single method avoids confusion with multiple Citrix policy locations.

It is important to understand how the aggregation of policies, known as policy precedence, flows, to understand how a resultant set of policies is created. With Active Directory and Citrix policies, the precedence is as follows:

Policies from each level aggregate into a final policy that is applied to the user or computer. In most enterprise deployments, Citrix administrators do not have permissions to change policies outside their specific OUs, which are typically the highest level for precedence. When exceptions are required, use the "block inheritance" and "no override" settings to manage which policy settings apply from higher up the OU tree. Block inheritance stops settings from higher-level OUs (lower precedence) from being incorporated into the policy. However, when configuring a higher-level OU policy with no override, the block inheritance setting is not applied. For this reason, care must be taken in policy planning. Use available tools such as the "Active Directory Resultant Set of Policy" or the "Citrix Group Policy Modeling Wizard" to validate the observed outcomes with the expected outcomes.

Note:

Some Citrix policy settings, if used, need to be configured through Active Directory group policy. For example, Delivery Controllers and Delivery Controller registration port, are required for registration of the VDAs.

Decision: Policy Integration

When configuring policies, organizations often require both Active Directory policies and Citrix policies to create a fully configured environment. With the use of both policy sets, the resultant set of policies can become confusing to determine. Sometimes, particularly concerning Windows Remote Desktop Services (RDS) and Citrix policies, similar functionality can be configured in two different locations. For example, it is possible to enable client drive mapping in a Citrix policy and disable client drive mapping in an RDS policy. The ability to use the desired feature can be dependent upon the combination of RDS and Citrix policy. It is essential to understand that Citrix policies build upon functionality available in Remote Desktop Services. If the required feature is explicitly disabled in the RDS policy, the Citrix policy is not able to affect a configuration.

To avoid this confusion, Citrix recommends configuring RDS policies only where required, and there is no corresponding policy in the Citrix Virtual Apps and Desktops configuration. Only configure RDS policies if the configuration is needed for RDS use within the organization. Configuring policies at the highest common denominator simplifies the process of understanding the resultant set of policies and troubleshooting policy configurations.

Decision: Policy Scope

Once policies are created, apply the policies to groups of users, computers, or both, based on the required outcome. Policy filtering allows policies to be applied to the required user or computer groups. With Active Directory-based policies, a crucial decision is whether to apply a policy to computers or users within the Site, domain, or organizational units (OUs). Active Directory policies are broken down into user configuration and computer configuration. By default, the settings within the user configuration apply to users who reside within the OU at logon. Settings within the computer configuration are applied to the computer at system startup and affect all users who log on to the system. One challenge of policy association with Active Directory and Citrix deployments revolves around three core areas:

• Citrix environment-specific computer policies
  Citrix servers and virtual desktops often have computer policies that are created and deployed specifically for the environment. Applying these policies is easily accomplished by creating separate OU structures for the servers and the virtual desktops. Specific policies can then be created and confidently applied only to the computers within the OU and below and nothing else. Based on requirements, divide virtual desktops and servers within the OU structure based on server roles, geographical locations, or business units.
• Citrix specific user policies
  When creating policies for Citrix Virtual Apps and Desktops, several policies specific to user experience and security are applied based on the user's connection. However, the user's account might be located anywhere within the Active Directory structure, creating difficulty with simply applying user configuration based policies. It is not desirable to apply the Citrix specific configurations at the domain level as the settings would apply to every system to which any user logs on. Applying the user configuration settings at the OU where the Citrix servers or virtual desktops are located also doesn't work. The user accounts are not located within that OU, therefore the settings not applied to the users. The solution is to apply a loopback policy. A loopback policy is a computer configuration policy that forces the computer to apply the assigned user configuration policy of the OU to any user who logs on to the server or virtual desktop. The user's location within Active Directory does not affect the applied settings in a loopback policy. Loopback processing can be applied to either merge or replace mode. Using replace mode overwrites the entire user Group Policy Object (GPO) with the policy from the Citrix server or virtual desktop OU. Merge mode combines the user GPO with the GPO from the Citrix server or desktop OU. As the computer GPOs are processed after the user GPOs when merge mode is configured, the Citrix related OU settings have precedence. When using merge mode, Citrix related OU settings apply in the event of a conflict. For more information, refer to the Microsoft support article Loopback processing of Group Policy.
• Active Directory policy filtering
  In more advanced cases, there can be a need to apply a policy setting to a small subset of users, such as Citrix administrators. In this case, loopback processing only applies to a subset of users, not all users who log on to the system. Use Active Directory policy filtering to specify specific users or groups of users to which the policy is applied. A policy can be created for a specific function. Also, a policy filter can be set to apply that policy only to a group of users. Policy filtering is accomplished using the security properties of each target policy.

Citrix policies created using Citrix Studio have specific filter settings available, used to address policy-filtering situations that are not available when using group policy. Apply Citrix policies using any combination of the following filters:

Note:

Policies in a Citrix Virtual Apps and Desktops environment provide a merged view of settings that apply at the user and computer level. In the previous table, the Scope column identifies whether the specified filter applies to user settings, computer settings, or both.

Decision: Baseline Policy

A baseline policy contains all common elements required to deliver a high-definition experience to most users within the organization. A baseline policy creates the foundation for user access and any exceptions that are needed to address specific access requirements for groups of users. To create the simplest policy structure possible, configure the policy settings in the baseline to be comprehensive enough to accommodate as many use cases as possible. Set the priority of the baseline policy to lowest priority, for example, 99. A priority number of "1" is the highest priority. Enable all Citrix policy settings in the baseline configuration, even if those settings use the default value. Configuring these settings defines desired behavior and avoids confusion if the default settings change. Use Citrix policy templates to configure Citrix policies to effectively manage the end-user experience within an environment. Citrix Policy templates are a solid initial starting point for a baseline policy. Templates consist of pre-configured settings that optimize performance for specific environments or network conditions. The built-in templates included in Citrix Virtual Apps and Desktops are shown in the following table:

For more information on Citrix policy templates, refer to Citrix Docs - Policy Templates.

Include Windows policies in a baseline policy configuration. Windows policies reflect settings that optimize the user experience and remove features that are not required or desired in a Citrix Virtual Apps and Desktops environment. One common feature turned off in these environments is Windows Update. In virtualized environments, mainly where desktops and servers are streamed and non-persistent, Windows Update creates processing and network overhead. Changes made by the update process do not persist after a restart of the virtual desktop or application server. Organizations often use Windows Software Update Service (WSUS) to control Windows Updates. In these cases, updates are applied to the master disk and made available by the IT department on a scheduled basis.

In addition to the preceding considerations, an organization's final baseline policy includes settings to address the organization's requirements. These requirements can be related to security, common network conditions, or to manage user devices or user profiles.

Design Decision: Administrative Delegation

Prevent unauthorized access by limiting the number of users who can access the policies. Leaving security too relaxed can lead to the exfiltration of the configuration details of the Citrix Virtual Apps and Desktops deployment. The method to restrict access depends on the engine used to configure the policies. When using Citrix Studio as the policy engine, assign roles to groups to delegate administrative access. For more information about scopes and roles, refer to the Delegated Administration documentation.

• Use the built-in administrative roles
  Add Active Directory groups to the respective role to delegate the required level of control.

  • Full Administrator grants read and write access on all objects in the Citrix Virtual Apps and Desktops Site. Pay special attention when assigning the "Full Administrator" role. Besides policies, the "Full Administrator" role grants read and write access to all other objects within the entire Site as well.
  • Read Only Administrator provides read-only permissions on objects within the assigned scope in a Citrix Virtual Apps and Desktops Site. Assigning a group to the "Read Only Administrator" roles grants read-only access to all policies regardless of the assigned scope.

• Create a custom administrative role
  For more granular control over access to policies, create custom roles. A custom role enables administrators to assign specific tasks to a group of administrators. Assign the "Manage Policies" or "View Policies" definition to delegate the appropriate permissions. As policies are not part of a specific scope, the scope assigned to the administrator does not affect access to the policies. Add Active Directory groups as Administrators and assign the custom role to delegate access.

When configuring Citrix policies using Active Directory group policies, administrators delegate access using the Group Policy Management Console. A single GPO can contain multiple Citrix policies. The granularity of the assigned permissions depends on the design of the GPO structure. Read or write access is granted to a user or group on a per-GPO basis. Access granted on GPO level grants permissions to all Citrix policies configured in that GPO.

Policy Design Recommendations

Based on experience from the field, Citrix developed leading practices related to Citrix policy design. The leading practices put together the design decisions taken from the previous chapters.

Baseline Policy

Leave the unfiltered policy empty and set it to disabled. Configure the unfiltered policy to have the lowest priority possible. The higher the priority number (for example, a priority of 99), the lower the priority. Create baseline computer and user policies named according to the company's naming convention. Ensure that the baseline policies apply to most users and computers which connect to the Citrix Virtual Apps and Desktops environment. Configure all settings in the baseline policy, even if these settings use the default value.

Policy Layering

Create exceptions to the baseline policy based on the requirements of the end-users. Assign the policy exceptions based on the appropriate filter. Set the priority for the exception policies to be higher than the baseline policy. Create as few policies as possible and consolidate settings where possible. Defining too many policies can lead to complexity and unexpected outcomes.

Example:

In a Citrix Virtual Apps and Desktops deployment, users are not allowed to access the local drives on their endpoint devices inside the Citrix session. Active Directory group membership grants access to local drives. To achieve this behavior, set the "Client drive redirection" setting to "Prohibited" in the baseline policy. Create a policy with a higher priority and set the "Client drive redirection" setting to "Allowed" in the new policy. Add an Active Directory group in the assignments of the new policy. Only users who are a member of the Active Directory group have access to local drives. The default behavior is to deny access to local drives for all other users.

Policy Management

Do not mix-and-match policy engines. Choose one policy engine and configure all Citrix policies using that engine. For example, when using Active Directory group policies, do not use Citrix Studio to create other Citrix policies.

Document all policies, policy settings, and exceptions. Either use a company-internal documentation format or use the policy's description field to keep track of changes. When using the description field, use a standardized form. For example, include the date, author, and description of the changes: 2020-04-17 - FvdP: Disabled Client Drive Mapping according to request SR422344.

Citrix Virtual Apps and Desktops supports various printing solutions. It is essential to understand the available technologies and their benefits and limitations to plan and successfully implement the proper printing solution.

Decision: Printer Provisioning

The process of creating printers at the start of a Citrix Virtual Apps and Desktops session is called printer provisioning. Multiple approaches are available:

• User Added
  Allowing users to add printers manually gives them the flexibility to select printers by convenience. The drawback of manually adding network-based printers is that it requires the users to know the network name or path of the printers. There is a chance that the native print driver is not available in the Operating System, and the Citrix Universal Print Driver is not compatible. The user needs to seek administrative assistance in these cases. Manually adding printers is best suited in the following situations:
  • Users roam between different locations using the same client device (for example, using a laptop or tablet).
  • Users work at assigned stations or areas whose printer assignments rarely change.
  • Users have personal desktops with sufficient rights to install necessary printer drivers.
• Auto Created
  Auto-creation is a form of dynamic provisioning that attempts to create available printers on the client device during the login process. Locally attached printers, and network-based printers, are included when printers are auto-created. Automatically creating all client printers can increase the session login time as it enumerates each printer during the login process.
• Session-Based
  Session printers are a set of network-based printers that are assigned to users through a Citrix policy at the start of each session.
  • The policy filters proximity-based session printers on the endpoint device's IP subnet. Citrix recommends using proximity printing in the following situations:
    • Users roam between different locations using the same endpoint device (for example, using a laptop or tablet).
    • When using thin clients that can't connect to network-based printers directly.
  • Session printers can be assigned using the "Session Printer" or the "Printer Assignments" policy. Use the "Session Printer" policy to set default printers for a site, Active Directory group, or OU. Use the "Printer Assignments" setting to assign a large group of printers to multiple users. If both policies are enabled and configured, the session printers merge into a single list.

• Universal Printer
  The Citrix Universal Printer is a generic printer object. The VDA auto-creates the printer at the start of a session and does not represent a printing device. When using the Citrix Universal Printer, it is not required to enumerate the available client printers during logon. Skipping enumeration of printers can significantly reduce resource usage and decrease user logon times. By default, the Citrix Universal Printer prints to the client's default printer. Administrators can modify this behavior to allow the user to select any of their compatible local or network-based printers. The Citrix Universal Printer is best suited for the following scenarios:

  • The user requires access to multiple printers, both local and network-based, which can vary with each session.
  • The user's login performance is a priority, and the Citrix policy "Wait for printers to be created" is enabled.
  • The user is working from a Windows-based device or thin client.

  Note:

  Other options for provisioning printers into a Citrix session are available, such as:

  • Active Directory group policy
  • "Follow-me" centralized print queue solutions
  • Other third-party print management solutions

Decision: Printer Drivers

Managing print drivers in Citrix Virtual Apps and Desktops can be complicated, especially in large environments with hundreds of printers. In Citrix Virtual Apps and Desktops, several methods are available to assist with print driver management.

• User Installed
  When adding a printer in a Citrix Virtual Apps and Desktops session, it can happen that the native print driver is not available. The user can manually install the missing printer drivers. Many different print drivers can potentially install on various resources creating inconsistencies within the environment. Troubleshooting printing problems and maintenance of print drivers become very challenging since every hosted resource can have different sets of print drivers installed. To ensure consistency and simplify support and troubleshooting, Citrix does not recommend user-installed drivers.
• Automatic Installation When connecting a printer within a Citrix Virtual Apps and Desktops session, the VDA checks whether the required print driver is already present in the operating system. If the print driver is not available, the native print driver, if one exists, is installed automatically. If users roam between multiple endpoints and locations, inconsistencies can occur across sessions since users can access a different VDA every time they connect. When this type of scenario occurs, troubleshooting printing problems and maintenance of print drivers can become challenging. Every VDA can have a different set of print drivers installed. To ensure consistency and simplify support and troubleshooting, Citrix does not recommend using automatically installed drivers.
• Universal Print Driver
  The Citrix Universal Printer Driver (UPD) is a device-independent print driver designed to support most printers. The Citrix UPD simplifies administration by reducing the number of drivers required on the master image. For auto-created client printers, the driver records the output of the application and sends it, without any modification, to the endpoint device. The endpoint uses local, device-specific drivers to finish printing the job to the printer. The UPD can be used in conjunction with the Citrix Universal Print Server to extend this functionality to network printers.

Decision: Printer Routing

Print jobs can route along different paths: through a client device or a print server.

• Client Device Routing
  Client devices with locally attached printers route print jobs directly from the client device to the printer.

• Windows Print Server Routing
  By default, print jobs sent to auto-created network-based printers route from the user's session to the print server. However, the print job takes a fallback route through the client device when any of the following conditions are true:

  • The session can't contact the print server
  • The print server is on a different domain without a trust established
  • The native print driver is not available within the user's session
• Citrix Universal Print Server Routing
  Print job routing follows the same process as Windows Print Server Routing. The only difference is that Windows uses the Universal Print Driver between the user's session and the Citrix Universal Print Server.

The specifics with print job routing depend on the printer provisioning method. Auto-created and user-added printers can route print jobs based on the following diagrams:

However, the print job routing changes slightly if the VDA provisions printers as session printers. The jobs can no longer route through the user's endpoint device and route from the session to the print server.

Citrix bases the recommended option on the network location of the endpoint device, the user's session, and the print server.

• Client Device Routing
  • Use for locally attached printer implementations.
  • Use if a Windows endpoint device and printer are on the same high-speed, low-latency network as the Windows Print Server.
• Windows Print Server Routing
  • Use if the printer is on the same high-speed, low-latency network as the Windows Print Server and user session.
• Citrix Universal Print Server Routing
  • Use if non-Windows endpoint device and printer are on the same high-speed, low-latency network as the Windows Print Server.

Decision: Print Server Redundancy

Configure redundancy when managing network-based printers with a Citrix Universal or Windows print server. Redundancy eliminates the print server being a single point of failure. Define the Citrix Universal Print Server redundancy within a Citrix Policy.

Experience from the field

A print media company uses thin clients and Windows-based workstations at the company headquarters. The company placed network-based printers throughout the building (one per floor). Windows print servers reside in the data center and manage the network printers. The company hosts the Citrix Virtual Apps and Desktops solution in the data center as well.

A regional office has numerous Windows, Linux, and Mac endpoints with network-attached printers. A remote branch office has a few Windows workstations with locally attached printers.

The print media company applied three different print strategies:

• Headquarters
  Headquarter users use a Citrix Universal Print Server for printing within the Citrix Virtual Apps and Desktops session. The Windows-based workstations do not require native print drivers. A session printer policy is configured per floor, which connects the floor printer as the default printer. The policies are filtered based on the subnet of the thin client for proximity printing.

  Network staff implemented Quality of Service (QoS) policies. The QoS policies prioritize inbound, and outbound network traffic on ports TCP 1494, and TCP 2598 over all other network traffic. The QoS policies prevent large print jobs from impacting HDX user sessions.

• Regional Office
  The regional office deployed a Universal Print Server. The print job uses the Universal Print Driver and is compressed and delivered from the user's session to the Universal Print Server, across the WAN. The print job then routes to the network-attached printer in the office.
• Branch Office
  Since all branch users work on Windows-based workstations, branch office users use auto-created client printers with the Citrix Universal Printer Driver. Since the print job routes over ICA, the print data is compressed, which saves bandwidth. The Citrix Universal Printer Driver guarantees the ability to use all client-connected printers within the user's HDX session without concern about the printer model.

The key to a good plan is identifying what systems are critical for the business and setting the appropriate mitigation in place.

Here are the questions that you need to answer about Business Continuity and Disaster Recovery:

What Citrix components use Availability Sets or Availability Zones?

• Not all regions support Availability Zones, if they are available in your region, prefer Availability Zones over Availability Sets

• Citrix recommends using the Availability Set or Availability Zones for the following Citrix components:

  • Cloud connector
  • Citrix Gateway
  • StoreFront
  • VDA Hosts

What information do I need to back up?

• Backup anything that is important. Set the backup vault frequency to match the RTO and RPO requirements for the server and set the retention based on corporate data retention policies

• Backups are not automatically copied to the paired region. This task must be done manually or scripted

• Persistent desktops must be backed up

• Golden image machines are to be backed up

What regions should I place my resources in?

• Azure regions are to be selected primarily based on data sovereignty, governance, and compliance

• Place Citrix resources in Azure regions that are close to your users

• Azure regions are paired for disaster recovery/business continuity. Take note of the regional pairs and select the pairs that work best for your users and business continuity plan

• Depending on where your users and data centers are located, one regional pair is a better choice than another regional pair

• When planning for disaster recovery, use the paired region for backups and failover configurations to increase the probability of a successful recovery in the unlikely event of a regional failure

What information should I store in the paired region?

• Identify key servers (recovery tier 0) that absolutely must be up 100% of the time, such as a domain controller, since they are good candidates for replication

• Any servers that are using Azure Site Recovery (ASR) must be pointing to the paired sister region for

• Identify key images/backups for servers that have to be brought online quickly and do not have an automated build available

• Schedule PowerShell jobs to replicate the key images, backups, and snapshots across to the other region

• Regularly export the ARM templates for your configurations and store them in the paired region. This task can be scripted.

What is appropriate to use Azure Site Recovery Replication?

• Any core infrastructure that is considered Recovery Tier 0

• Keep at least one domain controller at the recovery site so the domain passwords are current. That allows the users doing the restore to authenticate to the ASR environment

• Keep at least one Citrix Gateways for remote access into the system

Links to Other Resources

Business continuity and disaster recovery (BCDR): Azure Paired Regions

Enterprise-scale business continuity and disaster recovery

Design Decision: Disaster Recovery Planning

The article covers guidance and best practices for using Citrix User Profile Manager to manage user profiles on Azure Files as the back-end storage location. The primary objective of this article is to provide you the necessary information to determine the best way to deploy User Profile Manager with Azure Files.

The article discusses the Citrix and Microsoft components, describe the test methodology, and report the test results. The article also provides an analysis of the findings and guidance around key deployment decisions. Finally, additional information around securing, protecting, and migrating your user profile data is provided.

Azure Files

Azure Files is a secure, publicly hosted Server Message Block (SMB) or Network File System (NFS) file share with low latency access. Azure Files supports both Active Directory integration and NTFS file-level permissions and is accessible from Windows, Mac and Linux clients.

Azure Files can be used for various enterprise purposes, including

• File Servers. Azure Files can directly be mounted from all popular operating systems across the world. Azure Files can also be deployed in a hybrid scenario, where data can be replicated to on-premises Windows servers via Azure File Sync.

• Application migration. Move both the user applications and their data to the cloud simultaneously. Alternatively you can move the user data first and run in a hybrid configuration until the applications can be moved into the cloud.

• Simplifying cloud development. Use Azure files to store shared application settings, diagnostic logs, metrics, or developer utilities.

• Containerization. Use Azure Files for persistent storage of containers or as a shared storage file system.

With a shared access capable, fully managed, resilient file share, Azure Files is an excellent place to store user profile data for your Citrix users. Azure Files uses SMB 3.0 so all the data transferred is protected by encryption while in-transit.

Performance Tiers

Azure Files has four different performance levels with different costs and performance metrics. The two tiers, transaction optimized and premium, are evaluated and recommended for storing user profile data

Transaction optimized (formerly known as Standard) performance offers up to 100 TB of storage in a single file share with a maximum throughput of 300 MiB/sec. The transaction performance level is designed to support workloads that do not need high-performance. When using the transaction optimized option for user profile workloads, be sure to enable the "large file shares" setting. The maximum IOPS supported for a transaction file share with "large file shares" enabled is 10,000 IOPS. This is true for a 10 TiB share and a 100 TiB share.

Transaction storage accounts without "large file shares" enabled (\< 5 TiB), support multiple redundancy options. These options include locally redundant, zone redundant, and geo-redundant with read-only and hot or cold tiering available. When support for "large file shares" is enabled, only the locally redundant storage and zone redundant storage options are available.

Premium performance offers up to 100 TiB of storage in a single file share with 100,000 IOPS and throughput based on the provisioned size of the file share. A premium file share is provided a base ingress of 40 MiB/sec + (0.04 * provisioned GiB) while the base egress rate is 60 MiB/sec + (0.06 * provisioned GiB). For a 10 TiB share, the throughput equates to an ingress rate of 450 MiB/sec and an egress rate of 675 MiB/sec. The premium performance level is designed to support I/O intensive workloads while delivering latency of under 10 milliseconds.

The IOPS and throughput are provisioned based on the size of the file share. Be sure to provision a size that covers the IOPS and throughput your users need. For a premium file share, the base IOPS allocated is 400 + 1 IOPS for each GiB of provisioned space, with a maximum of 100,000 IOPS. A premium share is allowed to burst up to greater of 4000 IOPS or 3 times of base IOPS allocated. A 10 TiB files share would be allocated 10,240 IOPS. With the premium file share, burst mode allows up to 3 times the allocated IOPS to be used. However burst mode requires the file share be running at less than the allocated IOPS for a period of time to regenerate. In other words, you cannot run in burst mode all the time. If there are enough credits available, the shares can burst up to 30,720 IOPS for up to a maximum of 60 minutes of duration.

Both performance tiers are acceptable options for storing user profile data. Selecting between the performance tiers depends primarily on the user workload. The goal is always to provide the end user with their required performance and data protection at the lowest cost. For user profiles where response time is the highest priority, premium file shares are the best candidates. For more information on the performance of the different tiers, visit Microsoft's storage performance webpage. To determine what performance tier is best starts with monitoring key metrics.

Monitoring

You can use Azure Monitor to view the key metrics on your Azure File shares and make adjustments. For storage, closely monitor throughput, transactions, and latency metrics. To enable these metrics, in the Azure portal select the storage account and then select Metrics from the Storage Account blade. The key metrics we used to monitor the performance of Azure Files during the test are as follows:

Ingress (Sum): The aggregated sum of inbound data in bytes to Azure Files during the time period, with a minimum granularity of one minute. To determine the average ingress throughput per second, you would need to take the sum of the ingress data for a minute and divide it by 60. To add this metric, click Add metric and set the scope to the storage account name. Set the namespace to Account, then choose Ingress as the metric and Sum as the aggregation type.

Egress (Sum): The aggregated sum of the outbound data in bytes from Azure Files during the time period, with a minimum granularity of one minute. To determine the average egress throughput per second, you would need to take the sum of the egress data for a minute and divide it by 60. To add this metric, click Add metric and set the scope to the storage account name. Set the namespace to Account then choose Egress as the metric and Sum as the aggregation type.

Transactions (Sum): The number of requests, both successful and failed, made to Azure Files, during the time period, with a minimum granularity of one minute. This metric is loosely analogous to input/output operations per second (IOPS). To determine the average IOPS during a one-minute time period, you would take the Sum of Transactions for that minute and divide it by 60. To add this metric, click Add metric and set the scope to the storage account name. Set the namespace to Account, then choose Transactions as the metric and Sum as the aggregation type.

Transactions Success with Throttling (Sum): The aggregated number of requests that failed due to provisioned limits on the first attempt but were successful after retries. This metric is an API filter on Response Type and is only valid for the premium tier where resources are provisioned. If this number is significant when compared to the sum of transactions, increase your file share size. To add this filter, you must first add the Transactions metric. Then click Add filter, choose the Response type property, operator of = and select one of the filters in the values field. This filter is not visible as a selection unless you are seeing throttled transactions. For premium shares, you can also look at specific response types (SuccessWithShareIopsThrottling, SuccessWithShareEgressThrottling, SuccessWithShareIngressThrottling) to determine whether IOPS or throughput is throttled. For more information on the metrics dimensions, check the Microsoft website.

Success Server Latency (Avg): The average amount of time that Azure Files required to complete the request. This value does not contain any network latency. If this value is increasing, the Azure Files infrastructure is working at maximum capacity and falling behind. To add this metric, click Add metric and set the scope to the storage account name. Set the namespace to Account, then choose Success Server Latency as the metric and Avg as the aggregation.

Success E2E Latency (Avg): The average end-to-end latency required to complete a successful transaction with Azure Files. This value includes the network latency between the client and Azure Files. If this number is increasing when compared to the success server latency value, investigate the network. To add this metric, click Add metric and set the scope to the storage account name. Set the namespace to Account, then choose Success E2E Latency as the metric and Avg as the aggregation.

Citrix Profile Management

The use of Citrix Profile Management (CPM) greatly enhanced the end-user experience during our testing. Citrix Profile Management is designed to remove profile bloat and significantly speed logon times, while reducing profile corruption. Citrix Profile Management is a feature of Citrix DaaS.

Key Features

The key features of CPM that provide a great end-user experience when Azure Files is used for the back-end storage are as follows:

Profile Streaming: Files and folders contained in a profile are fetched from the user store to the local computer only when users access the files after they have logged on. Profile streaming significantly reduces the logon time by reducing the amount of data that is copied down to the local profile at user logon. This setting is enabled by default and decreases the amount of transactions and egress data leaving Azure Files, reducing costs, and improving logon time.

Active Write-back: Files and folders that are modified can be synchronized to the user store in the middle of a session, before logoff. Usually, these mid-session writes occur about every 5 minutes. Enabling this setting provides Azure Files with a more consistent stream of write traffic, instead of having all the writes occur at logoff. This frequent write-back resolves the "last writer wins" issue when users are accessing their profile from multiple devices simultaneously.

Large File Handling: To improve logon performance and to process large-size files, a symbolic link is created instead of copying files in this list. You must configure the paths where the files are stored but you can use wildcards. This setting is a highly recommended when using the transactional tier and recommended for the premium tier.

Profile Management Configuration

For the testing, each of the Citrix hosts had the Multi-session OS Virtual Delivery Agent (VDA) 2006 installed. During installation, on the Additional Components screen, both the Citrix User Profile Manager and Citrix User Profile Manager WMI Plug-in were enabled.

The following GPO settings for Profile Management were configured and the GPO was assigned to all the Citrix hosts being used for the tests.

Monitoring

Monitoring the performance can be done through the Citrix Director. The key metric to watch for performance is the profile load time. The profile load time provides the number of seconds it takes for the user's profile to load into the Citrix session. Profile load time includes the amount of time it takes to copy the user's profile down from the profile storage, in this case from Azure Files. The user profile load time has a significant impact on the user experience. Profile load times of greater than 30 seconds usually result in a poor user experience. The profile load time is available through the Citrix Director. From Citrix Cloud Virtual Apps and Desktops Service, select Monitor >> Trends >> Logon Performance as shown in the following screenshot:

The profile load time is available by hovering over the time period being monitored or viewed in the table below the chart.

Test methodology

The primary goal was to assess the Azure File shares under a load and determine for a specific workload how the transaction optimized and premium tiers would perform with CPM configured on the Citrix hosts. The following settings were kept static during the test runs

The following test variables were decided upon after assessing performance during a wide variety of test runs of different configurations.

To simulate a 1000 users with a few resources, we designed a test that used a custom PowerShell script. The script randomly updated several files stored either in the user's redirected folder (87.5%) or in the user's local profile (12.5%). This script when run by a single-user would be updating between 20 and 100 files (depending on the number of files per hour to update) in the session. The script also updated the 4 GB OST file stored in the local profile folder.

To control the launch rate and provide random wait times during the session, we used LoginVSI for the test framework. The random waits helped prevent predictable spikes in network traffic caused by users logging on simultaneously and helps simulate a better workload. The user would logon and access the required files before logging off. To exclude logoff traffic from the results, all users were logged on and the files accessed before any user logged off. The graphs show the metrics from the first one hour of the test while users were logging on.

Test Environment Setup

For the 100,000 files to be tested, we built an environment that supported 1,000 user sessions logging on within a 60-minute window. To avoid the test infrastructure from becoming a bottleneck, the Citrix host servers were deliberately oversized to move the performance issues to the Azure Files share.

We had two peered virtual networks configured in the Azure US West 2 region. One virtual network contained the LoginVSI infrastructure used to manage the sessions and report on progress. The other virtual networks contained the 30 Citrix hosts running on a Standard FS16_v2 instance with 1 TB Premium SSD drive. The Citrix servers hosted the sessions that required user profiles to be loaded from and written to Azure Files. We enabled and configured Citrix Profile Management to use the Azure Files SMB share as the path store.

Two Azure storage accounts were created, one with a transaction optimized tier SMB share and one with a premium tier SMB share. The "large file shares" setting was enabled on the transaction tier. Both storage accounts used Locally Redundant Storage (LRS). The default user profile had 400 files ranging in size from 85 KB to 5225 KB plus a single 4 GB file use for large-file testing. Total user profile size was approximately 4.7 GB.

Citrix Cloud was used to manage the machine catalogs and the delivery groups. A local StoreFront server was used for enumeration. The 30 Citrix hosts were running Windows Server 2016 with the Citrix Multi-session OS Virtual Delivery Agent (VDA) 2006. The Citrix StoreFront server and the Citrix Cloud Connectors were on the subnet with the LoginVSI infrastructure, the Citrix hosts resided on a different subnet.

Test Runs

Based on the variables identified earlier, the following test matrix was run, and the performance data was gathered from Azure Monitor and Citrix Director on each run.

After the warm-up test run, each test run consisted of the following steps

1. For premium tier, wait 2 hours to allow the burst IOPS to regenerate

2. Update the Computer GPO assigned to the Citrix host servers to set the user's profile location to the correct file share (transaction or premium)

3. Restart the Citrix Servers

4. Set the PowerShell script to update the correct number of files (20, 40, 60, 80, 100)

5. Configure the test run to launch all 1000 user sessions within 60 minutes

6. Launch the test

7. Record the start date/time

8. Each session ran at least 60 minutes before logging off

9. Record the stop date/time after all sessions are logged off

10. Gather Azure Monitor data for key metrics

11. Gather Citrix Director data for Logon Performance

Test Results

The test results are displayed by the key metrics discussed earlier: Transactions (IOPS), Throughput (egress and ingress), Latency, and Profile Load Time. Note these tests were run before SMB Multichannel was released. Azure Files' SMB Multichannel benefits performance when used with Profile Services.

Transactions

Transactions are recorded as part of the Azure Files metrics. The preceding charts show a rather consistent amount of IOPS being used across all test runs. Regardless of type of file share, the average IOPS per second are under about 3300. The consistent results are expected since the test was only varying the number of files touched between the two file share types. The files themselves were the same for every user profile.

Since both file shares support 10,000 IOPS in the configuration for the test, the IOPS required by our workload is well within the limits. This suggests that Azure Files can support 3X this workload before IOPS became a bottleneck.

Throughput

Throughput is available as part of the Azure Files metrics. The following charts show the egress and ingress traffic for the different tiers. The egress traffic is higher than the ingress traffic because Citrix Profile Manager is scanning the full profile at logon but only changing a portion of the files. If profile streaming was not enabled as part of Citrix Profile Manager, the entire profile would be read and copied down to the local host. As the number of files updated during the run increases, the egress throughput also increases. This behavior is because the entire file is streamed down to the local host, changed, and written back to the share.

The throughput for the premium share is well within the provisioned amounts of 675MiB/sec egress and 450MiB/sec ingress. The premium share is able to handle approximately 15 times the workload as our test before throughput starts to become an issue.

The transaction file share limit is higher at 300 MiB/sec so theoretically, the transactional file share can handle 5x the workload before the throughput becomes a bottleneck.

These results are writing to the 4 GiB OST file, but that file is not read down to the local host because Large File Handling is enabled. The writes to the OST files are made directly on the Azure Files share, much like the folder redirection writes.

Latency

Latency is available as part of the Azure Files metrics. The following chart shows the Success Server Latency metric from the premium and transaction optimized test runs. Premium tier file shares do best with fewer large files, such as user profile technologies like FSLogix and Citrix Profile Manager VHD container. Using container files improves the performance of Azure Files because the number of file open/read/write/close requests are reduced.

Focusing on the premium file share, we see that latency remains relatively steady at around 4 milliseconds (ms). Reviewing the transaction optimized file share, latency stays consistent across the different test runs. The latency stays in the 7-10 ms range, with only occasional spikes outside of that range. These longer latencies translate into longer profile load times as the later users logon. These are discussed further in the Profile Load Time section.

Profile Load Time

Profile load time is part of the Citrix Director trend analysis. The following table provides a brief overview of the average profile load times based on the run data collected from Citrix Director.

The following charts show a moving average over the previous minute for the profile load time for the users during the test runs.

As expected, the premium tier was provided the best load times over all. The interesting thing is that the Premium-80K run had the best load times on average followed by the Premium-40K. While the Premium-20K, which had the lightest load, is in the middle with the Premium-100K. The Premium-60K run had the worst load times. These numbers show the variability within the performance tier, even though it can consistently provide profile load times in under 10 seconds.

The transaction file share seemed to fair well, with profile load times averaging under 20 seconds for the users with a slight variability in the load times. Some users did have longer load times, but the overall average for the standard file share was about 15.84. This load time is just over double the overall premium load time average of 7.25 seconds.

Analysis of the Results

Azure Files does an excellent job of managing end user profiles when configured correctly. The results of the testing indicate with our workload, the premium file share seems to provide the best overall consistent performance at different levels. This performance comes with the added cost of premium tier pricing. While the premium file share does perform well, it would probably perform better using profile container technologies.

Citrix Profile Manager Configuration Options

Citrix Profile Manager is best used when users have multiple profiles open from different devices to protect against the "last write wins" scenario. This feature keeps the profiles consistent between logins when users have multiple sessions open. However, when using Azure Files with Citrix, a few other features are recommended to improve the user experience.

Always enable Folder Redirection when possible. As the chart shows, enabling folder redirection prevents a significant amount of data from being copied from Azure Files to the Citrix host. Folder redirection was designed to reduce the amount of network traffic at logon/logoff and this greatly improves the user logon experience and lower cost of deployment. Less data being copied to the Citrix host means smaller (and less expensive) storage attached to the Citrix host for storing the user profiles. When Citrix hosts reside in Azure, the latency for connecting to Azure Files is minimal and the user experience is not impacted. When Citrix hosts are on-premises, the redirection saves on the chargeable egress traffic leaving Azure.

Enabling the Large File Handling feature of Citrix Profile Manager is a fantastic way to improve the logon experience for users. The improved logon time is achieved by updating large files directly on the profile store instead of copying them down at logon. Since Microsoft does not recommend storing PST and OST files in a redirected folder, use large file handling to provide the same benefits as folder redirection. Citrix Profile Manager creates a symbolic link to the file, so when it is opened or updated, the file operations are redirected to the user store in Azure Files. This feature allows PST and OST files to remain in the user's profile on a remote file share while being treated as local files. The preceding chart shows the difference in egress traffic between enabling and disabling large file support with the transaction and premium file shares.

The later versions of CPM enable profile streaming by default on Citrix servers. This technology prevents the entire user profile from being downloaded to the Citrix host at logon. Instead, only a list of the files residing on the profile are enumerated and that list is available to the operating system. When a file is requested, then it is fetched from the user store and brought to the Citrix host. This process reduces the number of file copies that happen at logon and improves the user logon experience.

Selecting A Performance Tier

The results support using a transaction optimized file share when your throughput and IOPS requirements are below the supported levels of 300 MiB/sec and 10,000 IOPS respectively. Creating multiple file shares across different storage accounts to host the user's profile data, distributes the load.

NOTE: The recommended best practice is to have only one transaction optimized file share per storage account. This recommendation is because two or more transaction optimized file shares under normal use can exceed the maximum limits on a single storage account. For more information see Scalability and performance targets

If using the higher performance premium tier for consistent performance and lower latency, do some benchmarking before you settle on a final size. The best way to benchmark is to run the most intensive benchmark tests with a 100 TiB premium file share and monitor the transactions, throughput, and latency metrics. Use that information to determine the amount of provisioned storage for the required performance level.

NOTE: If the file share has a significant number of small files being updated on a frequent basis, Azure Files may have delays reading and writing the file due to excessive metadata. This delay can also happen if there is a throttling observed on the file shares. If you are experiencing throttling, the Success Server Latency metric on the Azure Files share shows a steady increase. Once that metric exceeds 15 milliseconds, the user experience noticeably degrades.

In the end, selecting a performance tier comes down to determining your expected response time and redundancy requirements. If the user response time needs to be under 10 ms, the premium tier file shares provide a consistent response time. This response time, in our testing configuration, translates into a profile load time of under 10 seconds. If your users accept a response time of over 10 ms then you can use the more cost-efficient transaction optimized file share. To provide the best user experience use multiple storage accounts to distribute the load.

Use the data provided in this article to determine the optimal configuration for your organization and test the performance of Azure Files with your own workloads.

Securing Access via Private Endpoint and RBAC Permissions

Azure Files provides two main types of endpoints for accessing Azure file shares:

• Public endpoints, which have a public IP address and can be accessed from anywhere in the world.

• Private endpoints, which exist within a virtual network and have a private IP address from within the address space of that virtual network.

Considering the file share is used for the user profile and data, we recommend that it is only accessible from within the Azure VNET via private endpoints. For detailed instructions, use the following link.

Once the private endpoint is configured, validate accessibility from a VM within the Azure VNET as outlined in the document.

Next step is to lock down the permissions via RBAC. Follow the guidance in Set up Azure Files storage for User personalization layers and Citrix Profile Management.

Protecting Data

Once the user data is stored in Azure Files, you need to protect from backup and loss. This section provides guidance for the two built-in features: Soft Delete and Azure Backup, that offer data protection, in addition to the storage account redundancy options that are available when the storage account is created.

Soft Delete

Soft delete is an Azure feature that allows you to recover files accidentally deleted by a user without the need to go through a complex restore process. Soft delete can be enabled in the file service section of the storage account. Along with enabling soft delete, you can set the number of days the deleted file will be available through the soft delete feature.

Azure Backup

When using premium tier or when large file shares are enabled on a transaction optimized, the data replication is limited to Locally Redundant Storage and Zone Redundant Storage. Without large files shares you can also use Geo Redundant Storage, but this limits the share to 5 TiB, 60 MiB/s, and 1000 IOPS. Azure Backup also provides a reliable way to protect your data via Azure File snapshots. You can use Azure Backup to set up retention schedules through the configuration options in the Azure Backup vault.

Since Azure Backup is using snapshots to protect your data, do not delete those snapshots or you may lose your recovery points and be unable to restore. To prevent against accidental deletion, Azure Backup locks the storage account. If you delete that lock and the share is deleted, all the backups and snapshots are also deleted and all your data are lost.

In addition to these, many third-party vendors also offer solutions that can be used to backup your Azure Files data.

Migrating Files

Getting your user data files migrated into Azure Files can be accomplished through several different methods. Before selecting a migration strategy, you need to determine the amount of data, the target number of files shares, and the target structure for the user data store.

Which method works best for you depends on where the data currently resides within your network. No enterprise is the same and the process to migrate files varies greatly between enterprises. Several methods exist to move the files; usually, you end up employing more than one method to achieve the desired migration.

• Azure File Sync: Any Windows server running Windows Server 2012 R2 or later can have the Azure File Sync agent installed. The agent will then upload the user data files from the Windows serve to Azure Files. The agent can typically move about 1 TB every two days if the outbound network is not throttled. With Azure File Sync the NTFS permissions, access control lists (ACLs) and file metadata are preserved.

• Storage Migration Service: Storage Migration Service (SMS) is designed specifically to help you migrate data from Windows Servers into Azure. The service can inventory existing servers, transfer data and even assume the identity of the source servers to make cutover easier on end-users.

• Microsoft Data Box Disk: Microsoft Data Box Disk is an 8 TB SSD flash drive that can be stacked up to 5x for a total of 40 TB. The Data Box Disk supports AES-128 encryption and relies on file copy utilities such as Robocopy to transfer the data.

• Microsoft Data Box: Microsoft Data Box is an offline file transfer solution that uses common file transfer utilities such as Robocopy to move data securely from your servers to the Data Box. The Data Box is then shipped to Microsoft and uploaded directly to Azure Files. Microsoft Data boxes come in 2 sizes, 100 TB and 1 PB. Both devices store the data encrypted (AES-256) to protect it while in transit between the on-premises data center and the Azure data center. Microsoft Data Box supports the SMB and NFS NAS protocols.

• Data Box Gateway: A virtual appliance running on either Hyper-V or VMware that acts as a storage proxy server backed by Azure Files. Local users access it over SMB or NFS protocols and the gateway accepts the data and transfers it to Azure Files over the internet. The Data Box Gateway provides for continuous data flow from the on-premises servers to Azure Files.

• Storage Explorer: The latest version of Storage Explorer uses AZCOPY to speed the file transfers significantly. Storage Explorer can even provide you the AZCOPY command strings if you feel inclined to automate the data transfer.

• AZCOPY: An Azure command-line utility that allows you to move data between storage locations, including on-premises file servers and storage accounts within Azure.

• RoboCopy: Reliable file copy utility that preserves all the files attributes, permissions, and ACLs during the transfer.

In addition to these file transfer/copy solutions, many third-party vendors also offer solutions that can be used to migrate your data into Azure Files. For more information on the best migration path see Migrate to Azure file shares.

Conclusion

Using Azure Files with Citrix deployments in Azure is highly recommended. Azure Files provides that high-availability of user profile data without the complicated infrastructure. Azure Files can also be used with on-premises deployments if the latency and networking performance is acceptable for users.

Citrix Profile Manager provides several benefits when used with Azure Files. These benefits include the use of profile streaming and large file handling to reduce the amount of data copied from the user profile store to the Citrix host.

In most cases, the use of the transaction optimized tier for Azure Files is sufficient. Especially, if you can distribute the load across multiple shares and target workloads that keep the number of files updated per hour to around 100,000 for a single share. For low latency, high response time requirements, where users need latency to be consistently under 10 ms, look at using the premium tier of Azure Files.

References

Content from the following documents has been referenced in this article:

• Azure Files scalability and performance targets

• Enable and create large file shares

• Planning for an Azure Files deployment

• Monitoring Azure Files

• Azure Files monitoring data reference

• About Azure file share backup

• Enable soft delete on Azure file shares

• Migrate to Azure file shares

• Get started with AzCopy

• Storage Migration Service overview

• Azure Files Introduction

• Planning for an Azure Files deployment

• Citrix Product Documentation - Profile Management policy descriptions and defaults

• Citrix Product Documentation - Profile Management

• Citrix Tech Zone - Set up Azure Files storage for User personalization layers and Citrix Profile Management

Citrix HDX is a broad set of technologies that provides end-users a high-definition experience for their hosted desktops, designed around the three technical principles of:

• Intelligent redirection

• Adaptive compression

• Data de-duplication

Citrix HDX decreases bandwidth consumption and optimizes the end-user experience. How much bandwidth is required for each user session is an age-old question and challenging to predict. How the user interacts with the desktop generates most of the traffic that determines the outcome of the end-user experience. Therefore, each user session with a virtual desktop is unique in its bandwidth requirements. Generally, the answer about the amount of bandwidth required for virtual desktop sessions is "it depends." However, through user workload testing, we can monitor bandwidth consumption to provide updated estimated ranges. This testing will allow Citrix admins to assess the HDX bandwidth requirements for several typical workloads for Citrix DaaS sites.

Test Setup

The testing environment used Citrix Virtual Apps and Desktop Service (CVADS) delivered via Citrix Cloud for all tests. Using the evergreen CVADS environment ensures that the latest fixes and technologies are used for the overall end-user experience. The following is the additional information about the components used during testing:

• Citrix Gateway Service was used for all access to the test environment.

• HDX connections were made from a Windows 10 Endpoint with Citrix Workspace App 2109 for Windows installed.

• A Windows 10 virtual machine with Citrix VDA 2106 and Citrix Optimizer v2.8.0.143 hosted on Citrix Hypervisor 8.2.

• Default Citrix policies were applied for each test iteration.

Test Details

The amount of data required and used by Citrix HDX depends on the end user's activity within the desktop. For example, a typical task worker who only uses Microsoft Office requires much less bandwidth than a user on Microsoft Teams video calls throughout the day. To provide any estimated multiple test scenarios must be run and reported. The details for each of the test scenarios run during the bandwidth testing are as follows.

Idle User

• The end-user is not actively working with the VDA. No active screen updates, keyboard or mouse movements are captured.

Microsoft Office

• The end-user works within Microsoft Word, Microsoft Excel, and Microsoft PowerPoint opening multiple documents, typing, copying/pasting graphics between documents, and switching between the files.

Web Browsing

• The end-user launches both Google Chrome and Microsoft Edge web browsers. A graphically rich website is used for this test.

Microsoft Teams Meeting (Workspace for Windows Microsoft Teams Optimization)

• The end-user joins a Microsoft Teams Meeting with both audio and video enabled. During the meeting, the organizer shared a PowerPoint presentation along with their video. The Teams meeting test ran for 10 minutes.

YouTube Video Playback

• The end-user is watching a 1080p resolution, 30 FPS YouTube video movie trailer that lasts two minutes and thirty seconds. Two separate tests were performed, one with Browser Content Redirection (BCR) enabled, the other with BCR disabled.

Results

The following table provides the results of the average bandwidth captured for each of the testing scenarios.

The following table provides the average of the minimum and maximum average range captured each of the test scenarios.

Key Takeaways

Citrix continues to improve the HDX protocol to apply optimizations based on the end-user working scenarios intelligently. Each improvement brings further reductions in overall bandwidth consumption utilization to Citrix environments. Citrix bandwidth consumption depends on the use case, so there is no right or wrong answer on bandwidth consumed per session. However, we can offer bandwidth estimates to assist you with the bandwidth requirements for any virtual desktop project by continually testing different scenarios. Citrix HDX continues to provide a high-definition user experience for the typical workload scenario by default. Enabling the Citrix Microsoft Teams and Browser Content Redirection (BCR) optimizations decrease session bandwidth further, ensuring a great end-user experience.

• Use cloud workloads for burst capacity or for business continuity by tagging cloud workloads in the delivery group. Set Autoscale to power manage these tagged workloads. Use the selective power management feature to set the zone preference and failover to prefer on-premises workloads.

• Use the dynamic provisioning feature of Autoscale with low and high watermark machine counts. This approach reduces costs and still supports demand under high loads. This feature is especially useful for instance types with high monthly charges.

• If usage and capacity patterns can be predicted on a consistent basis, use the schedule-based scaling feature to manage available capacity. Configure the settings to start within a 30-minute window. When paired with the capacity buffer, the scheduler prevents users from waiting for capacity to come online.

• When enabling Autoscale, prefer smaller instances over larger instances. Smaller instances drain faster and go offline sooner.

Should I use Windows 10 Multisession or Windows Server OS?

• Windows 10 Multisession has about 10% lower density than the Windows Server operating systems

• Windows 10 Multisession allows users access to the Windows Store, which is not available on the server operating systems

• Azure Virtual Desktop (AVD) entitlement grants you the base VM price (Linux Pricing), saving a considerable amount over the normal Windows VM price

Links to Other Resources

Autoscale Design

Designing a desktop virtualization solution is simply a matter of following a proven process and aligning technical decisions with organizational and user requirements. Without the standardized and proven process, architects tend to randomly jump from topic to topic, which leads to confusion and mistakes. The recommended approach focuses on working through five distinct layers:

Users are segmented into logical groups, based on personalization, security, mobility, resiliency, and activity requirements. The top three layers are designed for each user group independently. These layers define the users’ resources and how users access their resources. Upon completion of these three layers, the foundational layers (control and hardware) are designed for the entire solution.

This process guides the design thinking in that decisions made higher up impact lower level design decisions.

Delivery Models

A Citrix Virtual Apps and Desktops solution can take on many delivery forms. The organization’s business objectives help select the right approach. Even though the infrastructure remains the same for all delivery models, the local IT team’s management scope changes.

• Citrix Virtual Apps and Desktops - This delivery model can be deployed entirely on-premises, entirely in the cloud or in a hybrid model. Even though components of the solution are using different hosting providers and hypervisors, the entire environment is a single solution using the same code and managed as a single entity. The local IT team continues to manage all aspects of the solution except for the hardware associated with the cloud-hosted resources. In many situations, organizations using an on-prem model extend the environment to the cloud when required to support contractors or seasonal workers.

• Citrix Desktops-as-a-Service (DaaS) - Breaks a typical deployment into multiple management domains. The access and control layer components are hosted and managed in the Citrix Cloud by Citrix while the resource layer components remains managed by the local IT team either as an on-premises, public cloud or hybrid cloud model. Citrix manages the hardware, sizing, and updates to the access and control components while the local IT team manages the resources. In addition, if the public cloud hosts the resources, the local IT team does not have to manage the resource hardware.

• Citrix DaaS Standard for Azure - Not only manages the access and control layer components (similar to the cloud service offering), but it also manages the hardware layer.

• Citrix Service Provider - A hosted service offering where third party organizations design, build, and manage a Citrix Virtual Apps and Desktops environment on behalf of the organization. The Citrix Service Provider program is an outsourced IT management service that can handle solution deployments, updates, and management.

Design Considerations

Selecting the correct delivery model is based on identifying how much of the solution does the local IT team want and need to manage. The local IT team can have complete ownership, no ownership or somewhere in the middle. The following diagram provides a high-level overview of how the different options compare.

Gateway URLs, Call back URLs, and GSLB URLs

StoreFront allows administrators to define multiple Gateways that can be used for Gateway passthrough authentication in a single Store. This feature is greatly beneficial as it minimizes the number of Stores that have to be configured in large, global deployments. We see that there are four key parameters for this integration to work successfully: Citrix Gateway URL, virtual server IP address, Callback URL, and GSLB URL, which are discussed in the following sections.

Gateway URLs

The first configuration screen in setting up a Gateway prompts the administrator to enter a friendly display name and the Gateway URL, which is the URL entered by users, as shown in the following:

StoreFront only allows Gateway passthrough from Gateways that are defined in its configuration. Citrix Gateway takes the URL entered into users’ web browsers or Workspace app (Receiver) clients and inserts it into an HTTP header (XCITRIXVIA). This information is passed back to StoreFront. StoreFront then attempts to match this value against one of the defined Gateway URLs. The Gateway URL field is mandatory and must match what is entered into a users’ web browser or Workspace app (Receiver) for Gateway passthrough to StoreFront to work.

Callback URLs and Virtual Server IP Addresses

Next, we cover some optional fields: Virtual server IP address and Callback URL, whose usage is related and are covered together. These fields show up on the Authentication Settings screen of the Gateway configuration wizard, as shown here:

The Callback URL is intended to be used by StoreFront to gather additional information about a user’s Gateway session. It is not strictly used for authentication. Instead, it queries for things like the name of the Gateway Session policy applied to the user’s session. These additional details can then be used as part of Citrix Virtual Apps and Desktops Access Control-based policy filters. It is also required in “password-less” authentication scenarios such as when Smart Card and SAML are in use to perform extra validation on the user’s Gateway session. For Global Site Load Balancing (GSLB) deployments, both the Callback URL and the vServer IP Address are used. These scenarios are discussed later in this article. If one of these scenarios is not in play, both the Callback URL and virtual server IP address fields are not required and can be left blank.

If a Callback URL is required and multiple Gateways are linked to a single Store, StoreFront needs a way of correctly identifying not just whether traffic is coming through a Gateway, but which Gateway virtual server the traffic is coming from to route the callback to the correct Gateway that holds the user’s session. StoreFront first does this action by matching on the Gateway URL, which it receives via the XCITRIXVIA HTTP header as previously covered. If there are multiple Gateways that have the same Gateway URL specified (which would occur in GSLB architectures where the same URL resolves to multiple individual Gateway virtual servers), StoreFront falls back to using an IP address to identify a Gateway, which is a unique value. StoreFront receives the virtual server VIP address from a Gateway via another HTTP header (XCITRIXVIAVIP). Once received, it then attempts to match against the value of the vServer IP Address field in one of its assigned Gateways. Assuming StoreFront can identify one Gateway based on a virtual server IP address match, then the callback completes successfully. Therefore, the virtual server IP address is only needed when a Callback URL is specified and there are multiple Gateways bound to a Store with the same Gateway URL defined.

GSLB URLs

Lastly, a parameter that is not shown in the GUI: the GSLB URL. StoreFront version 3.9 introduced the ability via PowerShell only to specify a GSLB URL parameter to a Gateway definition. This GSLB URL functions as an alternate source that StoreFront accepts authentication requests from for that same Gateway definition. This parameter is visible in the Get-STFRoamingGateway command output. The purpose of this parameter is to decrease the total number of Gateways that need to be defined for a single Store to simplify administration. Without it, a Gateway object must be defined for every Gateway URL + unique Callback URL combination that can be used by users, which can add up quickly in an enterprise environment.

For example, three global Gateways behind a unified GSLB address (https://www.nsg.com): one in America, one in Europe, one in Asia, each with a unique Callback URL would require three defined Gateways. On top of that fact, these administrators want to be able to test authenticating at each Gateway individually. This is important in understanding if there are GSLB issues, resulting in alternate, unique names for each Gateway being defined. This instance means that a total of six Gateways have to be configured for the Store, looking like the following:

Table 1: Complete Gateway List

By instead applying the GSLB URL parameter, the number of Gateways defined can be cut in half, while still allowing users to connect via all GSLB and unique Gateway addresses as follows:

Table 2: Consolidated Gateway List with GSLB URL

Even though the parameter is called “GslbUrl,” it functions just as an alternate host name for that Gateway definition. It does not have to be a GSLB address, just another name by which that same Gateway virtual server can be accessed. Another use case can be split DNS with external/internal Gateway addresses that are routed to the same virtual server.

Note, this parameter is not recognized by Workspace app (Receiver) and therefore usually, the URLs in the earlier example would be flipped so that Workspace app (Receiver) clients can continue to use the GSLB address and the per-Gateway URLs can be used when connecting through web browsers:

Table 3: GSLB URL Adjusted for Receiver and Workspace app

Design Takeaways

To summarize the previous sections:

• A gateway URL is always needed and must match what is entered into web browsers or Workspace app (Receiver) clients
• Callback URL is only needed if SmartAccess Citrix Virtual Apps and Desktops policies, password-less authentication methods (Smart Cards, SAML, and so on), or GSLB URLs are used
• A virtual server IP address is only needed if a Callback URL specified and there are multiple Gateways bound to the Store that have the same Gateway URL specified
• GSLB URL is a new parameter that was added in StoreFront 3.9. The URL simplifies Gateway integration with StoreFront when a single Gateway virtual server can be accessed via multiple URLs
• Workspace app (Receiver) does not read the GSLB URL parameter, so the Gateway URL is always been the URL that is used by those clients and the GSLB URL is an alternate URL that can be used by web browser-based connections

Selecting a "Default Appliance"

When an administrator enables Remote Access for a Store, a “default appliance” must be defined as shown in the screenshot.

For web browser-based access, the “default appliance” setting has no impact. For Workspace app (Receiver)-based access, this setting is downloaded to Receiver on connection to the Store as part of the Store configuration. Gateway is used thereafter by default. If all defined Gateways share the Gateway URL, then again, occurrence has no impact (Receiver just uses that Gateway definition to pull the URL to query for authentication, so in GSLB configurations, that query is GSLB routed). As noted earlier, the Workspace app (Receiver) does not read the “GSLB URL” parameter, so the Gateway defined as the default appliance must have an appropriate Gateway URL defined, as shown in “Table 3: GSLB URL Adjusted for Receiver and Workspace app” in the previous section.

If the Gateways bound to the Store have different Gateway URLs, then whichever one is defined as the default is us by all Receiver clients. This occurrence is problematic if you have defined different Gateways to be used by different sets of users. For instance, one Gateway configured with LDAP+RADIUS authentication for users with tokens and one Gateway configured with Smart Card authentication. If the user enters the Smart Card Gateway URL into the Workspace app (Receiver), but StoreFront has the LDAP+RADIUS Gateway defined as the default one, after the Workspace app (Receiver) connects to StoreFront and caches the configuration, all future authentication requests will be sent to the LDAP+RADIUS Gateway, despite what the user originally entered. The only way around this issue is a separate Store or Server Group.

Design Takeaways

• Store with Remote Access enabled has a default Gateway that defines which Gateway URL is used by the Workspace app (Receiver) clients
• To use multiple Gateway URLs and Workspace app (Receiver)-initiated access, separate Stores, or StoreFront server groups must be defined

Optimal HDX Routing

Outside of performing authentication, another reason Gateways can be defined in StoreFront is for Optimal HDX Routing. This setting assigns a Gateway per Citrix Virtual Apps and Desktops Site or Zone. The purpose of the setting is to reroute the ICA connection through a Gateway that can be different from the user’s original authentication point (such as through a Gateway that is closer to the VDA hosting the user’s session). If this “optimal” Gateway is not otherwise performing authentication, it only needs STA servers bound for the session proxy and can be set up in StoreFront as an “HDX Routing only” Gateway type, which eliminates all of the Authentication settings.

When assigning that Gateway to a Site (via Manage Delivery Controllers) or Zone (via Manage Zones) in the Store settings shown here, there is an optional External only checkbox.

That setting means that the “optimal” Gateway will only be used for ICA sessions originating “externally,” meaning sessions that use Gateway passthrough as the authentication type (which StoreFront assumes means that the user is originating outside the corporate network). The setting will not apply to users who authenticate directly at StoreFront (which StoreFront assumes means that the user is inside the corporate network). If the box is cleared, then all ICA sessions destined for that Site or Zone will be routed through the defined Gateway, whether the user is external or internal. There is no “internal only” checkbox. That means that the defined Gateway URL needs to be reachable from all possible user locations, which can be challenging without split DNS. This is another case where separate Stores (since this is a Store-level setting) may be required for complex architecture scenarios with Optimal HDX Routing.

Design Takeaways

To summarize:

• The gateway used for Optimal HDX Routing only requires STA servers bound
• Gateways used for Optimal HDX Routing can either be “external only” or “external and internal,” but not “internal only”
• Separate Stores or Servers Groups are required to define separate internal and external Gateway URLs for Optimal HDX Routing

Beacons

Beacons are defined separately from the Gateways in the StoreFront configuration and are enabled automatically when you Enable Remote Access for a Store and configure the first Gateway. Beacons are website addresses that to help the Workspace app (Receiver) identify whether the endpoint client is inside or outside the corporate network and seamlessly route the access request to either the StoreFront base URL, if the client is determined to be “internal,” or the default Gateway address, if the client is determined to be “external” without prompting the user for more input. To do this, the Workspace app (Receiver) will first query the internal beacon address (assuming the external, internet-facing address can always be reachable) and then fall back to the external beacon addresses only if the internal fails. If it can reach the internal URL (gets an HTTP 200 response), then the client is assumed to be on the corporate network and will be directed to the StoreFront base URL.

Beacons are not used at all if a user is connecting through a web browser. This is because a user provides input by typing a URL into the browser bar and therefore is directing the browser to a specific address. With the Workspace app (Receiver), after initial configuration, the user never provides further input on what URL to use. The Workspace app (Receiver) has the base URL and any configured Gateway URLs cached as part of the configuration and needs to be able to intelligently choose on behalf of the user which URL to use when the user attempts to use the application.

There is no need to modify or tweak the beacon addresses unless the same Gateway URL and StoreFront base URL are used. In this case, the external and internal beacons would then be the same and the internal URL would be accessible externally, which defeats the purpose. Therefore, an administrator would need to choose to “Specify beacon address” for the internal beacon and enter a website that is only accessible from the corporate network. Otherwise, it is simply a good troubleshooting practice to understand how the beacon addresses are applied so that they are not examined when investigating issues with web browser-based access.

Design Takeaways

To summarize:

• Beacons are only used Workspace app (Receiver) clients
• Beacons are only be modified if the StoreFront base URL matches a Gateway URL (and is accessible from outside the corporate network)

References

Integrating Gateway and StoreFront

Configuring Optimal HDX Routing

StoreFront New Features

This article reviews how those settings can be implemented in an enterprise environment and how they integrate with other related settings, such as application keywords and subscriptions, to further control how user sessions are routed and resources are presented.

Overview

The configuration of multi-site settings is done through the “Manage Delivery Controllers” wizard in the StoreFront console, with various configurations detailed in product documentation. If more than one Site is configured for the Store, the Configure button under “User Mapping and Multi-Site Aggregation Configuration” becomes enabled as shown in the following.

Once selected, an administrator is prompted to configure user mappings and resource aggregation as shown in the following. The “Map users to Controllers” link prompts the administrator to configure a user group to which these settings are to be applied. The “Aggregate resources” link is where the actual multi-site aggregation settings are specified.

Multi-site aggregation cannot be configured until at least one user group has been defined to apply the settings to. The built-in “Everyone” group can be used if a single set of aggregation settings is to be applied to all users who connect to the Store. Once these settings are configured, if a user is not a member of one of the groups specified here, no applications or desktops are enumerated for that user. The next two sections review the available configurations in detail, starting with the “Aggregate resources” options.

Aggregation options

In the Aggregate resources section, an administrator is presented with all Sites that are enumerated and prompted to select which have overlapping resources and are aggregated, as shown in the following.

For the aggregated Sites, there are two more configurations that control how resources are enumerated and sessions are launched across those Sites. These settings apply to all Sites marked for aggregation:

• Controllers publish identical resources: This setting controls enumeration. If two Sites are marked as “identical,” StoreFront places the farms in the same “equivalent farm set,” which means that enumeration requests are load balanced (round robin) across the Sites as the resource sets are assumed to be equivalent, saving enumeration time. If the two Sites are not marked as “identical,” then StoreFront sends an XML enumeration request to all and de-duplicates the common application and desktops from the resulting resource sets.
• Load balance resources across controllers: This setting controls session launch. Launch requests are either load balanced (round robin) across the Sites or distributed in failover order, regardless of whether the Sites are “identical” or not. Session sharing takes precedence over a load balancing decision and note that load balancing does not account for load index. Therefore, if a user already has a session in Site B and launches another application or desktop, that session will also launch in Site B (assuming the application or desktop is available there).

Some use cases that are not handled well through the GUI are combinations of load balancing and failover or combinations of identical and non-identical sites. For instance, if there are two production sites that need to be load balanced and one DR site that is only being used if both production sites are down, the GUI cannot be used and the web.config file must be manually modified instead (refer to Citrix Docs for the proper format for this file).

Design Takeaways

To summarize the previous section:

• Only one of a set of “identical” Sites is used to enumerate applications and desktops when the Sites are aggregated through StoreFront
• Session launches can either be load balanced or failed over across aggregated Sites

User Farm Mapping

The “Map users to controllers” settings are commonly referred to as “User Farm Mapping” as they are used to control which Sites a given group of users are allowed to enumerate against, whether those Sites are aggregated or not. There are two primary use cases for this functionality:

1. Limiting Enumeration: even without multi-Site aggregation, assigning a group of users in StoreFront to a subset of Sites means that StoreFront will only send XML enumeration requests to those Sites when a user from that group authenticates. In a global deployment, this situation can have a significant impact on enumeration time as it prevents StoreFront from attempting to communicate with Sites that a given user does not need to be accessing anyway. For instance, these settings can be used to map US users to US-based Citrix Virtual Apps and Desktops Sites such that StoreFront can’t reach other globally dispersed sites when those users log in.
2. Assigning Aggregation Settings: if multi-Site aggregation is configured, it can be desirable to assign different configurations to different groups of users, such as different failover configurations or different combinations of Sites.

The “Map users to controllers” first prompts administrators to specify a user group and then the Sites that the user group is assigned to (although the dialog says “Controllers,” it is Sites as defined for the Store). There is a column that denotes whether the selected Sites have been previously configured for aggregation or not as shown in the following. Also, if the aggregated Sites were configured for failover (“Load balance resources across controllers” was left cleared), the order of the Sites can be specified here via the arrows on the right.

It is possible for a user to be part of multiple user group mappings. When StoreFront reads the list of configured user groups, it does not stop processing after finding a match for a user. All configured groups are processed and all returned Sites enumerated against. This scenario can be expected in the case of Citrix administrators, who typically have access to all applications and desktops and are members of multiple user groups to be able to test and reproduce reported user issues. This just means that administrators (or other users that are members of multiple groups) might experience different launch behavior that users in one of the groups because they have multiple configurations applied to them. Configuration-wise, the only consideration is if two user groups have access to the same set of Sites with different aggregation settings. In the context of an individual user, a Site can only belong to one aggregation group or an error is displayed. This is resolved by naming the aggregation groups the same between the two user group mappings, which is done by default through the GUI, but can be a consideration if the web.config file is being manually modified for a more advanced set of configurations as previously referenced.

Design Takeaways

To summarize the previous section:

• Assigning user groups to Sites, even without aggregation configured, can be used to help limit enumeration traffic and thus reduce the time to completion for this process
• Failover order for aggregated Sites is specified in the user group assignment wizard
• When multiple user group mappings are configured containing some of the same aggregated Sites and users belong to multiple groups, the aggregation groups must be named the same

Application Keywords

Another way to control resource display and launch is by using keywords in the application description field. Within Store settings, display can be filtered based on custom keywords, as documented in article CTX223451. There are also a few special, pre-defined keywords that have unique functions. Two of these are “primary” and “secondary,” which influence session launch across multiple Sites. When there are two instances of the same application published, the one with the keyword “primary” specified will always be preferred over the one with the keyword “secondary.” This setting overrides any Site aggregation settings covered in the previous sections, which means these settings are frequently used together.

For instance, with two Citrix Virtual Apps and Desktops Sites, Site A and Site B, almost all applications need to be launched out of Site A (based on the back-end application architecture) and failed over to Site B, but there are a couple of applications that are primarily hosted out of Site B. Multi-Site aggregation would be configured for all users in failover order with Site A listed first. The exceptional applications that are primarily hosted out of Site B would be configured with the keyword primary in Site B and secondary in Site A.

Design Takeaways

To summarize the previous section:

• The “primary” and “secondary” keywords can be used to further control application launch across multiple Sites and will take precedence over Site aggregation settings

Subscriptions

Subscriptions in StoreFront are how users are allowed to “favorite” applications and are stored in a local database on each StoreFront server, replicated automatically across a server group. By default, subscription records are stored in the format <SiteName>.<DisplayName>. This is because StoreFront will by default enumerate resources from all Citrix Virtual Apps and Desktops Sites and display them individually. If two resources happen to be published with the same name in different Sites, subscriptions to them are tracked individually. Therefore, we need some kind of Site tag to differentiate those subscriptions.

This changes with Site aggregation where identical resources across Sites are not displayed and tracked individually, but aggregated behind the same subscription shortcut. The Site tag is no longer meaningful because a single subscription record must cover the same resource from multiple Sites. Therefore, when Site aggregation is configured, the subscription records for aggregated resources are stored in the format <AggregationGroup>.<DisplayPath>\<DisplayName>.

This means that if StoreFront multi-Site aggregation is configured after the Sites were previously available individually (with subscriptions enabled), any user subscriptions for newly aggregated resources will disappear because of the change in subscription record format. The previous subscription record is no longer valid as the application is no longer tied to a Site, but an aggregation group. A workaround for this is to export the subscription database via PowerShell, replace all records for resources that will be aggregated with the proper format, and reimport the subscription database after the configuration of Site aggregation. Otherwise, users have to resubscribe to their applications.

Design takeaways

To summarize the previous sections:

• Multi-Site aggregation changes the format of user subscriptions in the subscription database, which must be considered if aggregation is implemented post go-live

References

Product Documentation: StoreFront Multi-Site Settings

StoreFront Keywords Usage

This guide assists with disaster recovery (DR) architecture planning and considerations for both on-premises and cloud deployments of Citrix Virtual Apps and Desktops.
DR is a significant topic in breadth of scope in and of itself. Citrix acknowledges this document isn't a comprehensive guide to an overall DR strategy. It does not consider all aspects of DR and sometimes takes a layman’s perspective on various DR concepts.

This guide is intended to address these considerations which have a significant impact on an organization’s Citrix architecture and provide architectural guidance related to them:

• Business Requirements
• Disaster Recovery versus High Availability
• Disaster Recovery Tier Classifications
• Disaster Recovery Options
• Disaster Recovery in the Cloud
• Operation Considerations

Business Requirements

Aligning to the “Business Layer” of Citrix design methodology, gathering precise business requirements and known constraints for service recovery is the starting point in developing any recovery plan for Citrix. This step directs the scope and requirements that inform the DR strategy (discussed later) most suitable to meet the business and functional requirements within the constraints determined by the company or IT department.

Here are a few examples of useful questions to discuss when doing DR planning. These key components are important for the identification of business requirements and potential constraints. This list is an example of common baseline questions, and organizations will have other questions to answer to align with their requirements. These questions are discussed in more detail in the Disaster Recovery Options section later in the document:

• Backup Strategy. Backup can support DR but DR is not a replacement for backups which will in fact be needed during DR either as part of the plan or as a hedge against corrupt replication data. What backup strategy is used today for servers? Including:

  • Backup frequency?
  • Retention period?
  • Offsite storage?
  • Backup testing Methodology?

• Recovery Time Objective (RTO). Will the Citrix platform be immediately available in the event of a disaster or brought online within a specific time period? (See Disaster Recovery Tier Classifications). RTO varies based on application or service criticality classification but generally, Citrix’s RTO must match or be superior to the lowest RTO of an app hosted on Citrix and Citrix’s dependencies (network, storage, compute, and so on) must similarly match or be superior to Citrix’s or downstream RTOs to apps can be jeopardized. It is worth including application back-ends that Citrix-hosted apps connect to into the discussion to align expectations.

• Recovery Point Objective (RPO). For RPO what degree of data loss is considered tolerable for the organization, which can vary depending on the classification of the applications in question? How old can the recovered data for the service be? 0 minutes? An hour? A month? Within the context of the Citrix infrastructure, this consideration can apply only to database changes and user data (profiles, folder redirection, and so on). As with RTO, it is worth considering the application back-ends for Citrix-hosted apps.

• Scope of Recovery. This consideration does not include just the Citrix infrastructure but can include user data or key application servers which Citrix-hosted application clients interface with. It is important to identify if there is going to be a disparity between time to recover the Citrix platform, and time to recover applications hosted on Citrix. The time delta can prolong an outage as only part of the solution is.

• Use Cases. Citrix platforms often service many use cases, each with different levels of business criticality. Does recovery cover all Citrix use cases? Or key use cases upon which the business’s operational success is fully dependent. The answer has a large impact on infrastructure scoping and capacity projections. Tiering and prioritization of use cases is recommended to compartmentalize the enablement of DR if it is a net new capability for the Citrix environment.

• Capabilities. Are there any key capabilities that do not need to be included in DR which can reduce cost? An example of this capability would be the use of persistent desktops versus non-persistent; some VDI use cases that can be served by Hosted Shared Desktops or even specific application solos. Organizations have at times indicated component redundancy (ADCs, Controllers, StoreFront, SQL, and so on) is not considered necessary in DR. Such decisions can be viewed as a risk if there is a prolonged outage of production.

• Existing DR. Does an existing DR strategy or plan exist for other Citrix environments and other infrastructure services? Does it recover into new routable subnets or into a “bubble” mirroring production networks? The answer can help give visibility to current DR approaches, tools, or precedence.

• Technology Capabilities. This consideration can't dictate the final recovery strategy for Citrix. However, it's important to understand:

  • Recovery: What storage replication technologies, VM recovery technologies (VMware SRM, Veeam, Zerto, Azure Site Recovery (ASR), and so on) are available within the organization? Some Citrix components alongside Citrix dependencies can use them.

  • Citrix: What Citrix technologies are in use for image management and access? Some components can't lend themselves well to certain recovery strategies. Non-persistent environments managed by MCS or PVS make for poor candidates for VM replication technologies because of their tight integration with storage and networking.

• Data Criticality. Are user profiles or user data considered critical to recovery? For persistent VDI if this data is unavailable when DR is invoked, would this cause a significant impact on productivity? Or can a non-persistent profile or VDI be used as a temporary solution? This decision can drive up costs and solutions complexity.

• Types of Disasters. This decision is fairly significant but also can be pre-established through a corporate BC or DR plan. This requirement often dictates recovery location. Is the strategy intended to accommodate for recovery of service at the application level only? Between hardware racks? Within a metro location? Between two geographies? Two countries? Within a cloud provider region or an entire cloud provider’s infrastructure (multi-region)? Between cloud providers (multi-cloud)?

• Client Users. Where are the users located who access the services in production? This decision can have implications as to where service is restored to, including corporate network connectivity which can require manual modifications during a DR event. In addition, the answer dictates access tier considerations.

• Network Bandwidth. How much traffic (ICA, application, file services) does each user session consume on average? This decision applies both to cloud and on-prem recovery.

• Fallback (or Failback). Does the organization have pre-existing plans for how to return to service in production once the DR situation has been resolved? How does the organization recover to a normal state? How is new data that can have been created in DR reconciled and combined with production?

Constraints

Constraints limit DR design options or their configurations. They come in many forms but can include:

• Regulatory or Corporate Policy. This decision can dictate what technologies can or can't be used for replication or recovery, how they are used, RTO, data sovereignty, or minimum distance between facilities.

• Infrastructure. Is there is directive on type of hardware to be used, network bandwidth available, and so on? These considerations can impact RPO considerations or can even present risks. For example:

  • Undersized firewalls or network pipes in DR can ultimately lead to outages as network dependencies can't handle the DR workload.
  • In the case of compute, undersized hypervisor hosts or use of different hypervisors entirely can impact options.
  • Regarding networking, if the recovery site happens to have more constrained network throughput capability versus production when servicing the WAN.
  • In cloud environments, especially when scaling into thousands of seats, this potential risk can quickly become a significant bottleneck because of component service limitations such as virtual firewalls, VPN gateways, and cloud-to-WAN uplinks (AWS Direct Connect, Azure ExpressRoute, Google Cloud Interconnect, and so on).
  • Also, in cloud environments, appropriate consideration is given to the parity of services available in cloud regions designated in the DR region. Lessons learned from the field tell us that although a public cloud region initially seem to appeal from a hosting cost perspective, important features such as Availability Zones, instance types, or particular storage features for example, are not available in all cloud regions. A good rule of thumb is to assess if the features and capabilities planned for use are available in all regions planned for deployment.

• Budget. DR solutions come with costs that can conflict with project budgets. More often than not, the shorter the RTO, the higher the cost.

• Geography. If a designated DR facility has been identified, considerations such as latency from production to DR facilities, in addition to users connecting to DR must be understood.

• Data Residency or Data Classification. This decision can limit options for locales and technologies or recovery methods.

• Cloud Recovery. Is there a mandate for infrastructure to be recovered in the cloud versus in an on-prem facility?

• Capacity. Does the DR location have sufficient capacity to handle the load required to meet the DR requirements? If recovering to the cloud, is the volume of workloads needing to recover realistic within the RTO if not pre-provisioned or covered with guaranteed capacity entitlements?

• Application Readiness. Do the applications hosted on Citrix that have back-end dependencies have a DR plan and how do the RTOs line up with Citrix’s target RTO? Citrix can be designed for rapid recovery but if core applications do not have a recovery plan or one with an extended RTO, this risk likely impacts the usefulness of the Citrix platform in DR.

• Network Security. Does the organization have security policies that might dictate what traffic segments require encryption in transit? Does this consideration vary depending on the network link being traversed? The answer might require in DR scenarios the use of SSL VDA, ICA Proxy, GRE, or IPsec VPN encapsulation of ICA traffic if network flows for DR are different from production.

Misconceptions on Disaster Recovery (DR) Design

One of the most common misconceptions for Citrix DR designs is that a single control plane spanning two data centers makes up DR. It does not.
A single Citrix Site or PVS Farm spanning data centers, even ones near, includes a high availability design but not a DR design.
Some organizations choose this path as a business decision, valuing simplified management over DR capability. StoreFront Server Groups spanning data centers are supported only between well-connected (low latency) data centers. Similarly, PVS has documented latency maximums to consider when deploying across data centers as outlined in CTX220651.

The following diagram is an example of a multi-data center HA Citrix architecture that is not however, a DR platform because of the previously mentioned rationale. This reference architecture is used by organizations who treat two physically separate data center facilities as a single logical data center because of their close proximity to each other and low latency, high bandwidth interconnectivity. A DR plan and an environment for Citrix would still be recommended as this architecture would be unlikely to satisfy the needs of either DR or HA.

In addition to the HA conceptual reference above, HA architectures based on zones are available for organizations who want to provide cross-geo redundancy but do not require the platform to be fully recoverable. The zones concept allows for various intra-site redundancy capabilities which can solve challenges for multi-location deployments.

In a Site architecture that uses the zone preference feature, identical Citrix resources are deployed across several zones and aggregated into a single Delivery Group. Zone preference (with the ability to fail back to other zones for a given resource) can be controlled based on the application, the user’s home zone, or the user's location. Refer to Zone Preference Internals for more information on this subject. The VDA Registration auto-update feature allows VDAs to maintain a locally cached up-to-date list of all controllers within a Site. This function allows VDAs within the satellite zone to fail over registration to VDAs to the primary zone in the event their local Delivery Controllers or Cloud Connectors become unavailable in their local zone. StoreFront Server Groups are present in each zone location (or at least in two zones), as is recommended for users to continue accessing resources from their local access tier in the event of the primary zone being unavailable.

In contrast to these supported HA options, the following diagram illustrates unsupported or ill-advised component architectures spanning geographically distant data centers or cloud regions. These diagrams provide neither effective HA nor DR because of the distance and latency between components. Platform stability issues can also result because of latency concerns in such a deployment. Furthermore, the stretching of administrative boundaries between sites does not align to DR principals. We have seen similar conceptual designs by organizations in the past.

Another common misconception is the depth of consideration that a Citrix DR solution can include. Citrix Virtual Apps and Desktops at its core is primarily a presentation and delivery platform. The Citrix control plane depends on other components (networking, SQL, Active Directory, DNS, RDS CALs, DHCP, and so on) whose own availability and recovery design meet the DR requirements of the Citrix platform. Any shared administrative domain of technologies on which Citrix is dependent can impact the efficacy of the Citrix DR solution.

Of similar importance and occasionally not fully accounted for by organizations are recovery considerations for file services (user data and business data on shares, and so on) and application back-ends with which Citrix-hosted applications and desktops interface. Referencing the earlier point on application readiness, a Citrix DR platform can be designed to recover in short timespans. However, if these core use case dependencies do not have a recovery plan with RTO similar to Citrix or no recovery plan at all, this plan can affect Citrix successfully failing over in DR as expected but users being unable to do their job functions as these dependencies remain unavailable. Take for instance a hospital who hosts their core EMR application on Citrix. A DR event occurs, and Citrix fails over to an “always on” Citrix platform in DR and users are connecting, but the EMR application is recovered via more manual means which can take hours or days to complete. Clinical staff would likely be forced to use offline processes (pen and paper) during this time. Such an outcome can't be congruent to the overall expectation of the business for recovery time or user experience.

The next section will discuss DR versus HA in more detail.

Disaster Recovery (DR) versus High Availability (HA)

Understanding HA versus DR is critical in aligning with organizational needs and meeting recovery objectives. HA is not synonymous with DR, but DR can use HA. This guide interprets HA and DR as follows:

• HA is regarded as providing fault tolerance for a system (service, application, and so on) or component. A system can fail over to another system with minimal disruption to a user. The solution can be as simple as a clustered or load-balanced application to a more complex active or standby “hot” or “always on” infrastructure that mirrors the production configurations and is always available. Failover tends to be automated in these configurations and can also be termed a “geo-redundant” deployment.
• DR is concerned primarily with the recovery of service (due to significant app-level failure or catastrophic physical failure of the data center) to an alternate data center (or cloud platform). DR tends to involve automated or manual recovery processes. Steps and procedures are documented to orchestrate recovery. It is not concerned with redundancy or fault tolerance of the components and is generally a broader-scope strategy that withstands multiple types of failures.

While HA tends to be embedded into design specifications and solutions deployment, DR is largely concerned with the orchestration planning of IT personnel and infrastructure resources to invoke recovery of the service.

Can HA include DR? Yes. For enterprise mission-critical IT services, this concept is common. Take for instance the second example in the earlier HA description, coupled with appropriate recovery of other non-Citrix components related to the solution, which would be regarded as a highly available DR solution wherein a service (Citrix) fails over to an opposing data center. That architecture can be implemented as Active-Passive or Active-Active in various multi-site iterations such as Active-Passive for all users, Active-Active with users preferring one data center over another, or Active-Active with users being load balanced between two data centers without preference. It is important to note that when designing such solutions, standby capacity must be considered, accounted for, and load monitored on an ongoing basis to ensure that capacity remains available to accommodate DR if it is required.

It is also essential that DR components be kept up to date with production to maintain the integrity of the solution. This activity is often overlooked by organizations who design and deploy such a solution with the best of intentions, then start consuming more platform resources in production, and forget to increase available capacity to keep the DR integrity of the solution.

In the context of Citrix, spanning Citrix administrative domains (Citrix Site, PVS Farm, and so on) between two data centers such as two geo-localized facilities as per published guidance would not make up DR and for some components such as StoreFront Server Groups. Similarly, because of PVS’s database latency limitations per published guidance, such a deployment is not recommended. Supportability constraints between geos also extend to Citrix Site and Zone design, because of latency maximums between satellite controllers and the databases per published guidance.

As many Citrix components share dependencies such as databases, stretching administrative boundaries between the two data centers would not protect against several key failure scenarios. If databases became corrupt the failure domain would impact application services at both facilities. To regard an HA Citrix solution as being sufficient for DR, we recommend that the second facility does not share key dependencies or administrative boundaries. For instance, create separate Sites, Farms, and Server Groups for each data center in the solution. By enabling a recovery platform to be as independent as possible we reduce the impact of component-level failures from affecting both production and DR environments. This consideration also extends beyond Citrix, and using different service accounts, file services, DNS, NTP, hypervisor management, certificate authorities (FAS dependency), and authentication services (AD, RADIUS, and so on) between production and DR environments is also recommended to reduce single points of failure.

Disaster Recovery Tier Classifications

Tier classifications for DR are an important aspect of an organization's DR strategy as they provide clarity into application or service criticality which in turn dictates the RTO (and thus costs for accomplishing that level of recovery). Alternatively, some organizations establish supported RTO (backed by technology solutions or architectural models) and assign tier classifications to them. Generally, the shorter the RTO, the higher the DR solution cost. Being able to break down various inter-dependencies into different classifications (based on business criticality and RTO) can help optimize cost-sensitive DR cases.

The following is a set of sample DR tier classifications to serve as a reference when assessing the Citrix infrastructure services, its dependencies, and critical applications or use cases (associated with VDA workloads) hosted on Citrix. DR tiers are outlined in order of recovery priority with 0 being the most critical. Organizations are encouraged to apply or develop a DR tiering classification aligning with their own recovery objectives and classification needs.

Disaster Recovery Tier 0 (Example)

Tier 0 - Recovery Objectives

Recovery Time Objective: 0

Recovery Point Objective: 0

Tier 0 - Classification

Core IT Infrastructure

• Network and security infrastructure
• Network links
• Hypervisors and dependencies (compute, storage)

Core IT Services

• Active Directory
• DNS
• DHCP
• File Services
• RDS Licensing
• CA Services (if using FAS)
• SQL Servers (on-prem WEM or Citrix Virtual Apps and Desktops, PVS, Session Recording)
• Critical End User Services (Citrix)

Tier 0 - Notes

This classification is largely for core infrastructure components. These components are always available in the DR location as they are dependencies for other tiers, and not in an isolated network segment. They need to be provisioned and maintained alongside their production equivalents. If Citrix is hosting critical applications, it’s likely regarded as a Tier 0 platform. In such scenarios, the Citrix infrastructure is deployed “hot” in DR.

Disaster Recovery Tier 1 (Example)

Tier 1 - Recovery Objectives

Recovery Time Objective: 15 Minutes

Recovery Point Objective: 1 Hour

Tier 1 - Classification

Critical Applications

• Websites and web apps
• ERPs and CRM applications
• Line-of-business applications

Citrix Use Cases

• Management
• Call Centers
• Customer Service or Sales
• IT Engineering and Support

Tier 1 - Notes

Applications or virtual desktops that the business depends on to carry on core business activities would typically be contained in this tier. They would likely also employ a similar “hot standby” DR architecture as Tier 0 or benefit from an automated replication and failover solution. If provisioning into the cloud, considerations discussed later, need to be accounted for as they can impact RTO targets.

Disaster Recovery Tier 2 (Example)

Tier 2 - Recovery Objectives

Recovery Time Objective: 4 Hours

Recovery Point Objective: 1 Day

Tier 2 - Classification

Non-critical Applications

Non-critical Citrix use cases

User Data that does not impact the functionality of Tier 1 or Tier 0 apps

Tier 2 - Notes

Applications or use cases which are key to business operations, but whose short-term unavailability is unlikely to cause serious financial, reputation, or operational impacts. These applications are either recovered from backups or recovered as the lowest priority by automated recovery tools.

Disaster Recovery Tier 3 (Example)

Tier 3 - Recovery Objectives

Recovery Time Objective: 5 Days

Recovery Point Objective: 1 Week

Tier 3 - Classification

Infrequently Used Applications

Tier 3 - Notes

Applications with negligible outage impact are unavailable for up to one week. These applications are likely recovered from backups.

Disaster Recovery Tier 4 (Example)

Tier 4 - Recovery Objectives

Recovery Time Objective: 30 Days

Recovery Point Objective: 24 Hours or None

Tier 4 - Classification

Non-production environments

Tier 4 - Notes

Applications, infrastructure, and VDIs whose outages also have a negligible impact on business operations can be restored over time. Depending on their nature, they can also have an extended RPO or not at all. These RPOs can be recovered from backups or built brand new in DR and are regarded as the last tier to be recovered.

Why is Tier classification important for Citrix DR planning?

Tier classification is important for Citrix for the following reasons:

• How critical is the Citrix infrastructure to business operations? If Citrix is considered essential, but its host application is regarded as necessary, the Citrix infrastructure would become classified as critical.
• The use cases or core business applications that Citrix hosts. Do they have different DR tier classifications?

For the first question, many enterprise Citrix deployments tend to be classified as Tier 0 because of the delivery of critical apps or desktops; the “always on” tier like network, Active Directory, DNS, and hypervisor infrastructure. This circumstance is not always the case however for every Citrix use case. Treating every Citrix use case as Tier 0 when some can fall into Tier 1 or higher can impact the overall cost and complexity of the DR process.

The second question stresses classification per Citrix use case and lends its importance most significantly in cloud environments which is discussed later in more detail. In the cloud, there are significant cost considerations between accommodating for “always on” application or desktop platforms, versus applications or desktops regarded as being less business critical. Such considerations can also influence application or use case isolation (silos) in production to take advantage of deployment flexibility in a DR platform.

When establishing a DR design for Citrix, bringing the discussion beyond the scope of Citrix itself helps set expectations for business units. For example, a Citrix environment is considered an “always on” service and is made highly available in an alternate data center. Yet, the critical back-ends that Citrix’s hosted applications depend upon will remain unavailable for several hours as the recovery is invoked. This gap creates a recovery time disparity between the two platforms and can provide a misleading user experience during the recovery. Citrix is available immediately, but critical applications are not functional. Setting expectations at the outset offers appropriate visibility to all stakeholders on the recovery experience. In some situations, a customer can want to keep Citrix on hot standby (always on) in the opposing facility but manually control the failover of the access tier to avoid misunderstandings on platform availability.

Disaster Recovery Options

In this section, common Citrix recovery strategies are outlined including their pros and cons, and key considerations. Other recovery capabilities or variations of the following themes for Citrix are possible. This section is focusing on some of the most common. In addition, this section illustrates how responses to the core questions indicated early on influence the DR design.

Correlating to several of the earlier DR questions, the following question topics have Citrix DR design implications as follows:

• Backup Strategy and Recovery Time Objective (RTO). If Citrix infrastructure or applications served from Citrix are considered mission critical, it is likely Citrix must employ an “always on” model where hot standby or Active-Active Citrix infrastructure is present at two or more data centers. This architecture would result in multiple independent Citrix control planes being created at each data center (separate Citrix Sites, PVS farms if applicable, StoreFront Server Groups, and so on). Spanning the control planes for the Citrix infrastructure does not make up DR (ex. spanning a Site across data centers or using satellite zones); doing so if the business is expecting a DR platform for Citrix, that counters to that mandate.
• Scope of Recovery. If Citrix is deployed in a hot standby (always on) capacity in DR but the core business application back-ends take, for example, 8 hours to recover, it can't make sense to employ an automated access tier failover. Manual failover of the access tier can be more appropriate.
• Use Cases. If only critical workloads or core applications must be rapidly recovered in Citrix to maintain business operations in a DR scenario, this scenario can likely reduce DR costs from a capacity perspective. If multiple use cases of differing priorities need recovery, the classification of importance per use case contrasted to business impact per the recovery tiering strategy discussed earlier can't reduce capacity costs but will provide IT staff a more focused set of recovery priority orders.
• Capabilities. If certain components such as HDX Insight, and Session Recording are not considered critical to DR platform operation, their omission in the DR environment removes some complexity and maintenance overhead. Similarly, if many use cases in a DR scenario can subsist on a simpler and more generic Citrix delivery option in DR, this can also reduce complexity and costs. For example, using a Hosted Shared Desktop instead of Pooled VDI or Persistent VDI if technically feasible (in the context of rapid and cost-effective service recovery in a DR scenario), or aggregating more use cases into one, provided they are not detrimental to business operations.

  

• Existing DR. If the existing DR strategy of the organization, for example, recovers Citrix and other application infrastructure using data replication and orchestration tools, most Citrix components can be suitable for this model. If the size of the platform and reliance on single-image management technology is a production platform requirement, then often such technologies can't be suitable; a hybrid approach of a hot standby Citrix platform and perhaps replication of master images can be more appropriate.
• Data Criticality. If profiles are considered critical for recovery in DR, appropriate replication technology can be necessary. Many organizations are less concerned with profiles in DR scenarios and accept them being created again. This consideration will also apply to user data accessed in Citrix (folder redirection, mapped drives); RPO and RTO for this data can be a business decision. Furthermore, if many persistent VDIs are considered important enough to remain intact in DR (versus requiring users to reinstall their software, and so on) then these VMs must be considered for replication, which can incur extra costs to support recovery.
• Types of Disasters. The extent to which a customer wants to protect against failure can dictate various architectural changes. If the customer only wants HA of the Citrix infrastructure within the data center or cloud region, this type of disaster can merely require ensuring management components are both redundant and operating on opposing infrastructures. As an example, a type of StoreFront server pair uses VMware anti-affinity rules to operate on different hosts in a cluster, or on different clusters entirely within the data center, or perhaps as part of different availability sets. This situation seldom requires creating redundant control planes entirely (multiple Citrix Sites and StoreFront Server Groups for example). However, if DR is intended to encompass multiple data centers regardless of region, then redundant control planes using local dependencies at each data center (AD, DNS, SQL, hypervisor, and so on) is more appropriate. If the customer is global and employs multiple data centers to service Citrix (or plans to) with respective application back-ends local to these data centers, then it is more likely that a geo-localized Active-Active HA-DR architecture is more appropriate. This architecture provides users with the most optimal user experience, when possible, by using geo-localized Citrix infrastructures which can if need be, failover to a backup data center in a secondary preference order.
• Client Users. Beyond relation to the earlier point about user, app, and data localization considerations, some client user networks can be relatively locked down with security devices (proxy, firewall, and so on) which can restrict outbound communication to the Internet or even the WAN. It is important to consider if this state applies to the client networks and ensures that new IPs for Citrix services (such as StoreFront VIP and VDA IPs, or Citrix Gateway IP) are accounted for in their local security configurations to ensure that further recovery delays do not occur due to local LAN security restrictions when invoking DR. From a preparedness perspective, in the event of DR, will client access change in some fashion? In some DR scenarios, for example, an organization can assume that the WAN is unavailable and all users must access Citrix resources via the Internet. Such steps would need documentation in the DR and readiness plans, with prerequisites established for end-users (regarding supported Citrix client details, and assumptions on corporate or BYOD device access) to remove barriers for users returning to service, further reducing the burden on the support desk.
• Network Bandwidth. The amount of network bandwidth used about VDA traffic (ICA, applications, file services) must be factored into DR facility network sizing and firewalls. This consideration is especially important in cloud environments where limits on VPN gateways and virtual firewall capacity exist. Monitoring production traffic from VDAs to determine an average value for sizing is important to effectively size networking. Where network constraints exist, organizations must use different network configurations to accommodate the expected DR traffic load if and when invoked.
• Fallback (or Failback). If user data has changed in DR, or if VDA images while in DR were updated, the organization must plan for failback to propagate these changes back to production if the production environment proves recoverable. For user data, it can be as simple as reversing the replication order and recovering; similarly, for Citrix infrastructure if not using single image management technologies. If using MCS or PVS, master images or vDisks can be manually replicated back to production, and VDAs updated accordingly.

The following list outlines various common recovery options for Citrix. Adaptations of each exist in the field (organizations sometimes opt to implement only portions of these options to limit DR to critical use cases or use a mix of options to balance costs among use case tiers), however for the sake of comparison, we are outlining basic versions of each. The options are organized starting with the simplest (often higher RTO and lower cost) through the more advanced (often lower RTO but higher cost). The ideal option for a given organization is to align to recovery time for hosted applications or use cases, in addition to the IT skills, budget, and infrastructure available.

In addition, although possible to accomplish, many options indicate that Citrix technologies integrated with network and storage such as NetScaler and single-image management technologies are not suited for methods other than “always on” recovery models. It is not that it is technically impossible to accomplish, but the level of complexity involved in accomplishing them can make recovery riskier and more prone to human error.

Recovery Option - Recover from Offline Backup

In this option, organizations rely solely on traditional backup solutions to restore the Citrix platform to service in another location. There is no standby infrastructure to fail over to. Multiple, often manual tasks are required in the recovery plan to restore the core services Citrix depends on and later restore Citrix itself. The outage time of Citrix is dictated by the speed at which components can be restored in the DR location. This model is not ideal for advanced Citrix deployments and is best suited for smaller, simpler infrastructures and organizations that can withstand an extended period of downtime.

Pros and Cons

Pros:

• Lower maintenance costs compared to replication or hot-standby solutions

Cons:

• High downtime impact
• Larger, more detailed recovery plan (DR orchestration) documentation
• Extended recovery time
• Relies on integrity and age of backups
• A higher degree of human error if manual reconfiguration is necessary (networking, and so on)
• Unsuitable for Pooled, Fast Clone MCS or PVS
• Unsuitable for NetScaler VPX because of networking (and needs to be rebuilt using backups of nsconfig directory and ns.conf files)
• Considerations must be given to sufficiently extending backup retention scope to accommodate other modern-day threats such as time bombing and ransomware

Use Case and Assumptions

Useful for less mature IT organizations and organizations with limited IT operations budgets and can allow for extended outages to recover core business services. Assumes backups and the recovery process are tested regularly to ensure that the backups are good and the recovery process is well understood by those who have to do it during a real event.

Recovery Option - Recover via Replication

This option is similar to the prior option but uses replication instead of backup from which to restore infrastructure. Replication can reduce some aspects of manual tasks in the recovery sequence and potentially allow RTO to be reduced (and thus restore service faster) but remains reliant on many manual tasks. It is worth noting that replication is not a replacement for backup. Corrupt or compromised data still gets replicated to DR and is thus not useful in such scenarios. Backups must still be considered as a fallback. As with the prior option, this model is not ideal for advanced Citrix deployments and is best suited for smaller, simpler infrastructures and organizations that can withstand an extended period of downtime.

Pros and Cons

Pros:

• Replication is likely automated and aligns with RTO and RPOs
• Likely uses less complex technologies as compared to automated recovery solutions

Cons:

• Relies on administrative intervention
• Larger, more detailed recovery plan (DR orchestration) documentation
• A higher degree of human error if manual reconfiguration is necessary (networking, and so on)
• Unsuitable for Pooled, Fast Clone MCS, or PVS. Recreation of Machine Catalogs is factored into the projected RTO. However, by creating dummy Machine Catalogs in DR or scaling out VDA instances in DR and performing an “Update Catalog” action applying a replicated master image, this RTO can be shortened
• Unsuitable for NetScaler VPX due to networking and thus better suited to employ hot standby NetScaler and assume the cost of doing so

Use Case and Assumptions

Useful for less mature IT organizations and organizations with limited IT operations budgets. This solution relies on storage replication technologies from the SAN vendor or the hypervisor vendor (vSphere Replication, Nutanix Replication, and so on) to replicate VMs to another facility over the WAN.

Recovery Option - Replication with Automated Recovery

In this option, automated recovery is introduced for several components of the solution to restore the Citrix infrastructure and its dependencies in the DR location. Technologies that orchestrate automated recovery further reduce manual steps and minimize human intervention (and potential human error), and can further improve RTO by streamlining the recovery of service. It is expected that intervention is still required to fail over to the DR location by way of DNS record modifications. It is worth noting that replication is not a replacement for backup. Corrupt or compromised data still gets replicated to DR and is thus not useful in such scenarios. Backups are still be considered as a fallback. While this model is more mature, it remains unsuitable for use with advanced Citrix configurations or for organizations that can't tolerate much downtime.

Pros and Cons

Pros:

• Lower maintenance costs compared to hot-standby solutions
• Replication is likely automated and aligns to RTO and RPOs
• Recovery plans tend to be automated
• Less administrative intervention and human error

Cons:

• Relies on more advanced technologies such as VMware SRM, Veeam, Zerto, Nutanix Disaster Recovery Orchestration, XenServer Disaster Recovery, and Azure Site Recovery (ASR) to orchestrate recovery and modify network parameters (unless network segments are stretched)
• Unsuitable for Pooled, Fast Clone MCS or PVS. Recreation of Machine Catalogs must be factored into the projected RTO. However, by creating dummy Machine Catalogs in DR or scaling out VDA instances in DR and performing an “Update Catalog” action applying a replicated master image, this RTO can be shortened
• Unsuitable for NetScaler VPX due to networking and thus better suited to employ hot standby NetScaler

Use Case and Assumptions

Useful for enterprise organizations with appropriate resources and budget for DR facilities. This solution relies on the same storage replication of the previous option but includes DR orchestration technologies to recover VMs in a particular order, adjust NIC configurations (if needed), and so on.

Recovery Option – Hot Standby (Active-Passive) with Manual Failover

We now start to look at options that use hot standby infrastructure. Although more costly, the advantages of an appropriately designed hot standby model drastically reduce the number of steps required to orchestrate recovery to the DR location, reduce the risk of technology and human errors, and can significantly recovery time. These models do need maintaining separate independent Citrix infrastructures which create more overhead (ongoing maintenance and testing) but accommodate more advanced Citrix deployments that use single image management technologies. In this option, failover relies on administrator intervention to redirect traffic to the alternate location through DNS changes, reducing the cost and complexity of automated failover of the access tier. This option would best suit organizations with sufficient budget to accommodate standby Citrix infrastructure, rely on advanced Citrix configurations, and require low RTO but manually controlled recovery.

Pros and Cons

Pros:

• Short recovery time as the platform is "always on"
• Supports storage and network-dependent components such as NetScaler, MCS, PVS
• Lower recovery plan (DR orchestration) documentation

Cons:

• Relies on administrative intervention to fail over URL or direct users to back up the URL (albeit manual orchestration is sometimes intentional)
• Higher costs due to having "hot" hardware in DR sitting on standby
• Higher administrative overhead to keep standby platform's configurations and updates in sync with production

Use Case and Assumptions

Useful for enterprise organizations with appropriate resources and budget for DR facilities. Can use a hot standby “fully scaled” platform or a “scale on-demand” platform. The latter can be appealing for cloud recovery to reduce operating costs, with caveats.

At the time of failover, administrators update the DNS entry for one or more access URLs to point to one or more DR IPs for Citrix Gateway and StoreFront, or users are advised by formal communication to begin using a “backup” or “DR” URL.

This manual option can be useful for scenarios where application back-ends can require longer recovery time but would add confusion for users if Citrix was fully available and applications were not.

This model assumes a mature IT organization and enough WAN and compute infrastructure are available to support failover.

Recovery Option – Hot Standby (Active-Passive) with Automated Failover

Similar to the prior hot standby option, this option includes automated failover typically via DNS technologies that detect failures in production and redirect traffic to the DR location. The automated nature does put further emphasis on the IT teams to ensure that configurations are duly maintained in DR and application and data dependencies remain accessible from DR during regular operation to avoid impact to users who might end up in the DR site as a result of an isolated temporary service disruption in production. Capacity management and monitoring (ensuring DR can accommodate the same number of users on Citrix as production) become increasingly critical due to the automated failover capability. This option would best suit organizations with sufficient budget to accommodate standby Citrix infrastructure, rely on advanced Citrix configurations, and require RTO near zero.

Pros and Cons

Pros:

• Short recovery time as the platform is "always on"
• Supports storage and network-dependent components such as NetScaler, MCS, PVS
• Minimal recovery plan (DR orchestration) documentation
• Easier for end-users as URLs fail-over
• Supports EPA Scans at Citrix Gateway with GSLB

Cons:

• Higher costs due to having “hot” hardware in DR sitting on standby and NetScaler licensing. Cost burden can be reduced to an extent if the standby capacity is used by development or other less critical non-production workloads which can be powered down during a DR event to prioritize production recovery
• Higher access tier complexity
• Higher administrative overhead to keep standby platform’s configurations and updates in sync with production

Use Case and Assumptions

A common configuration with enterprise organizations that allows for an automated failover to the DR site via NetScaler GSLB (Advanced or higher needed). This model assumes a mature IT organization and enough WAN and compute infrastructure to support failover. This model also assumes that application and user data dependencies are in alignment with the latest active site versions/updates and recoverable at the DR facility in a similarly automated manner to reduce prolonged impact of service to the end user and confusion due to partial solution functionality.

Recovery Option – Active-Active with Automated Failover

This option builds off the former but during regular operation, both production and DR locations are in active use. Capacity can be managed and monitored tightly, as in such a model it is easier to overlook when either location has surpassed a threshold (say 40–45% usage) to ensure that either location can accommodate the full user load during a DR event. As both environments are used actively, routine validation of most components is built in. However, components that require failover to the DR location (apps, storage arrays, and so on) still undergo routine periodic failover testing. As the model uses two or more active locations, consideration must be given to performance tolerances and variances between locations, especially if the locations span significant geographic distances. Some performance deviations can be minimized in this model by homing users to one location over another (localizing the VDAs) during regular operation and failing over to other VDAs if their local location is unavailable. This model is the most complex, most robust, and accommodates the most advanced Citrix deployments. It is ideal for organizations who need an RTO of zero and desire a model that sees only 50% of users failover during a DR event versus 100%.

Pros and Cons

Pros:

• Short recovery time as the platform is "always on"
• Supports storage and network-dependent components such as NetScaler, MCS, PVS
• Minimal recovery plan (DR orchestration) documentation
• Seamless to end-users
• Supports EPA Scans at Citrix Gateway with GSLB (NetScaler 13.0+ firmware releases from 2022 onward can accommodate EPA scans in Active-Active GSLB configurations, previously functional only in Active-Passive configurations)

Cons:

• Higher costs due to having “hot” hardware in DR sitting on standby and NetScaler licensing. Cost burden can be reduced to an extent if the standby capacity is used by development or other less critical non-production workloads which can be powered down during a DR event to prioritize production recovery
• Highest access tier complexity
• Higher administrative overhead to keep standby platform’s configurations and updates in sync with production
• Relies on administrators to monitor and adjust resource and hardware capacity at all data centers to ensure that as the platform grows, DR capacity integrity is not impacted

Use Case and Assumptions

A more advanced but common configuration with enterprise organizations that allows access tier URLs to operate in an Active-Active manner via NetScaler GSLB (NetScaler Advanced or higher needed). This functionality is useful in environments with local data center proximity to each other, or in situations where data centers can be remote but with the means to pin users to preferred data centers (often driven by advanced StoreFront configurations and GSLB to a lesser extent) for multi-site scenarios.

This model assumes a mature IT organization and enough WAN and compute infrastructure to support failover. This model also assumes that application and user data dependencies are in alignment with the latest site versions/updates and recoverable at the DR facility in a similarly automated manner to reduce prolonged impact of service to the end-user and confusion due to partial solution functionality.

Disaster Recovery in Public Cloud

Disaster Recovery involving on-prem to cloud platforms or cloud-to-cloud comes with its own set of challenges or considerations which often do not present themselves in on-prem recovery scenarios.

The following key considerations can be addressed during DR design planning to avoid missteps that can render the DR plan applying cloud infrastructure either invalid, cost prohibitive, or unable to meet target capacity in the event of DR.

Consideration – Network Throughput

Impact Areas

Availability Performance Cost

Details

Organizations can underestimate and thus undersize the throughput juncture points in their cloud solutions including virtual firewalls, VPN gateways, and WAN uplinks (AWS Direct Connect, Azure Express Route, GCP Cloud Interconnect, and so on) which can have a significant deleterious effect on performance if the Citrix platform is to recover in the Cloud and be accessible over the WAN. Also, be mindful of VPN Gateway limits if in use with a cloud provider. For example, AWS Transit Gateways presently have a maximum limit of 1.25 Gbps. This limit can require scaling out gateways or perhaps using multiple VPCs if these are critical to the cloud architecture. Azure and GCP VPN Gateways will have higher limits. Many virtual firewalls have licensed limits on the throughput that they can process or maximums even at their highest limit. This constraint can require scaling out the number of firewalls and load-balancing them in some manner.

Recommendations

When establishing throughput sizing calculations, assume the full DR capacity load. Capture the following data per concurrent user:

• Estimated ICA session bandwidth
• Estimated application communication bandwidth per session if they traverse security boundaries
• Estimated bandwidth for file services per session if they traverse security boundaries

For the previous metrics, it can be useful to gather data on current traffic patterns to and from VDAs in production. It is also important to consider what other data flows unrelated to Citrix are also anticipated to be using these network paths. Be sure to engage the network and security teams in planning Citrix DR to ensure any bandwidth estimates traversing security zones and network segments is understood and can be accommodated.

Consideration – Authentication Services

Impact Areas

Availability Performance Security

Details

Authentication services are critical to the operation of Citrix platforms. Most continue to rely on Microsoft Active Directory Domain Services (AD DS). If using Citrix Federated Authentication Service (FAS), Microsoft Certificate Authorities (CAs) or supported third-party CAs are also critical to the ongoing operation of the platform. That is, if they are down, Citrix is down.

Some organizations continue to resist placing authentication services infrastructure in public clouds, often citing security concerns. As a result, authentication and brokering performance are impeded when backhauling to writeable on-premises Domain Controllers (DCs) during regular operation, and the availability and reachability of DCs and CAs in a DR scenario must not be overlooked.

Recommendations

Implement sufficient compensating controls to address the concerns of the security team and place writeable DCs and CAs (if using FAS) near the Citrix infrastructure in the cloud. If non-negotiable, ensure that recovery is planned for, and reachability validated for these servers if the production data center fails or is unreachable.

Consideration – Windows Desktop OS Licensing

Impact Areas

Cost

Details

There are potentially complex licensing considerations for Microsoft Desktop OS instances running on different cloud platforms. Microsoft revised their cloud licensing rights in August 2019 which can affect VDI cost implications in some deployment scenarios. Microsoft licensing for Office 365 / Microsoft 365 apps has also been a changing landscape about support on public clouds and OS types.

Recommendations

Licensing of Microsoft products is a continuously evolving matter. Refer to Microsoft and the cloud provider (if relevant) for the current details on licensing and supportability statements when determining the use case architecture. If there is a potential cost challenge for VDI in the DR solution, consider supplementing with Hosted Shared Desktops (augmentation of RDS CALs can be required), if feasible, as they can provide greater flexibility at a lower cost of operation. Restrictions or future support changes require reevaluating the platform used for DR entirely.

Consideration – Timing of VDA Scale Out (Before or During DR)

Impact Areas

Cost Availability

Details

Organizations are drawn to the cloud by the appeal of paying for capacity only when it is needed. This solution can reduce DR costs dramatically by not paying for reserved infrastructure whether it is used or not.

However, at large scales, a cloud provider can't commit to an SLA of powering on hundreds or thousands of VMs at once. This solution becomes particularly challenging if the VDA footprint for DR is anticipated to run into hundreds or thousands of instances. Cloud providers tend to maintain bulk capacity for various instance sizes on hand; however, this provider can vary from moment to moment. If a disaster affecting a geographical area occurs, there can be significant contention from other tenants also requesting on-demand capacity.

In the case of Azure, do not confuse reserved instances with reserved (on-demand) capacity. If choosing reserved instances to act as a hedge against resource availability in a DR region on demand, it would be prudent to deploy them as Microsoft does not guarantee the capacity is available on demand. With on-demand capacity, Microsoft sets aside capacity outright (at pay-as-you-go rates). Ultimately, it would be prudent to assume that only running VDAs are guaranteed to be available.

Key questions that need to be answered that can influence decisions:

• Is always 100% DR capacity required?
• What types of disasters are being planned for? If a regional disaster is one, would it be wise to use a single cloud region or a cloud region close to production?
• Do we host critical workloads only (that is, only a subset of production)?
• Is it viable to scale out at the time of DR? If so, have the use cases been prioritized by DR Tiers to better understand the varying RTOs of each use case to support a phased scale out of capacity?
• Can we scale up the OS instances supporting app or hosted shared desktop use cases to save on provisioning time and power off these instances to conserve operating costs?

Recommendations

Citrix recommends first engaging with your cloud provider to determine the viability of powering on the anticipated capacity within the RTO timeframe and if it can be met with on-demand instances or not. To hedge against capacity availability limitations for VDAs in a DR scenario, Citrix recommends provisioning VDAs across as many availability zones as possible.

At large scales, it can be worthwhile provisioning across various cloud regions, and adjusting the architecture accordingly. Some cloud providers have suggested employing varying sizes of VM instance types to further mitigate VM exhaustion.

Some organizations want to diversify vendor risk and adopt a multi-cloud deployment wherein use cases are provisioned across two different cloud providers to hedge against an entire cloud platform failing (DNS, routing, breach, and so on).
This scenario does increase complexity and has not performance or feature parity between clouds, but these potentially minor caveats are eclipsed by the perceived value of platform diversification. Where possible, it would be prudent to pre-provision VDA instances and update them regularly (keeping them online or offline would be at the discretion and risk tolerance of the customer). Provisioning is a resource and time-intensive process and scaling out VDA DR capacity on-demand can be expedited to a degree by pre-provisioning.

If the organization has little appetite for capacity availability risk, it can require applying reserved or dedicated compute capacity and budgeting accordingly to guarantee resource availability. It is possible to combine on-demand and reserved/dedicated models by referencing the DR recovery tier wherein certain use cases can require VDAs to be immediately available, while others can have the flexibility to be recovered over a longer RTO of several days or weeks if they are less critical to sustaining the business.

Consideration – Application and User Data

Impact Areas

Cost Availability Performance

Details

The location of user data and application back-ends can have a notable impact on the performance and sometimes, availability of the Citrix DR environment. Some customer scenarios use a multi-data center DR approach where not all application back-ends or user data such as home and departmental drives can't be recovered to the cloud alongside Citrix. This gap can result in unanticipated latency which can impact the performance or even functionality of the applications. From a throughput perspective, this gap can add further strain to available network and security appliance bandwidth.

Recommendations

Where possible, keep application and user data local to the Citrix platform in DR to maintain performance as optimal as possible by reducing latency and bandwidth demands across the WAN. Determine what actually needs to be recoverable in DR (FSLogix Office Containers might not be critical, and some use cases work fine with a new profile being created) and ensure that application packages and containers are available in the DR site. Determining the most appropriate and cost-effective replication solution is critical. This topic is discussed further later in this guide.

Disaster Recovery Planning for Citrix Cloud

There are several notable differences between on-prem or “traditional” deployment of Citrix Virtual Apps and Desktops, versus Citrix DaaS provided by Citrix Cloud regarding DR planning:

• Citrix manages most control components for the partner/customer, removing significant DR requirements for the Citrix Site and its components from their responsibility.
• The deployment of a DR environment for Citrix resources merely requires a customer to deploy Citrix Cloud Connectors in the recovery “Resource Location”, and optionally StoreFront and NetScaler (for Citrix Gateway).
• Citrix Cloud's unique service architecture is geographically redundant and resilient by design.
• Access tier DR is not required if using Citrix Workspace and Citrix Gateway services.

Beyond these core differences, many of the DR considerations from prior sections still require partner/customer planning, as they retain responsibility for the Citrix VDAs, user data, application back-ends, and Citrix access tier if Citrix Gateway Service and Citrix Workspace service are not used from Citrix Cloud.

This section covers key topics to assist organizations in defining an appropriate DR strategy for Citrix Cloud.

Citrix DaaS Simplifies Disaster Recovery

The following is a typical conceptual diagram outlining the conceptual architecture of Citrix DaaS, in addition to the separation of responsibility for Citrix-managed components and partner/customer-managed components. Not illustrated here are the WEM, Analytics, and Citrix Gateway “Services” which are elective Citrix Cloud components related to Citrix DaaS which would fall under “Managed by Citrix”.

As illustrated in the diagram, a significant portion of Control components requiring recovery decisions fall under Citrix's management scope. Being a cloud-based service, the Citrix DaaS architecture is highly resilient within the Citrix Cloud region. It is part of Citrix Cloud's “Secret Sauce” and is considered within Citrix Cloud's SLAs.

Service availability management responsibilities are as follows:

• Citrix. Control Plane and access “services” if in use (Citrix Workspace, Citrix Gateway Service).
• Customer. Resource Location components including Cloud Connectors, VDAs, application back-ends, infrastructure dependencies (AD, DNS, and so on), and access tier (StoreFront, NetScaler) if not using Citrix Cloud access tier.

Organizations gain the following advantages regarding disaster recovery on Citrix DaaS:

• Lower administrative burden due to fewer components to manage, and fewer independent configurations to replicate and maintain between locations.
• Reduced chances of human error and configuration discrepancies between Citrix deployments due to the centralized configuration of the “Citrix Site” in the Cloud.
• Streamlined operations due to simplification in resource management for production and disaster recovery deployments due to fewer Citrix Sites and components to be configured and maintained, no access tier to manage between locations (optional), and less complex disaster recovery logic for Citrix resources.
• Reduced operational costs with fewer server components to deploy and maintain and gain a single pane of glass view of environment trends across deployments through the centralization of the monitoring database.

Citrix DaaS Disaster Recovery Considerations

Although many components for recovery planning are removed from the organization’s scope of management, organizations remain accountable the planning and management of DR and high availability (optional) for the components located within the Resource Location.

The most significant difference in how we address availability comes down to how we interpret and configure Resource Locations. Within Citrix DaaS itself, Resource Locations are presented as Zones. With Zone Preference we can manage failover between Resource Locations based on the logic we specify. Using Zone Preference within a traditionally deployed Citrix Virtual Apps and Desktops Site would be considered a high-availability design but not a valid DR design. In the context of Citrix Cloud, this option is a valid DR solution.

Most of the Disaster Recovery Options discussed earlier applying to Citrix DaaS so there are numerous options to fit organizational recovery goals and budgets.

When planning DR for Citrix Cloud's Citrix DaaS service, several key guiding principles from an infrastructure planning perspective need to be understood:

• Resource Locations. Production and DR locations are set up as independent Resource Locations in Citrix Cloud.
• Cloud Connectors. Each Resource Location must have a minimum of two Cloud Connectors deployed. For clarity, Cloud Connectors are not a component that must be “recovered” manually or automatically during a DR event. They must be deemed “hot standby” components and kept online within each location always.

• Customer-Managed Access Controllers (Optional). Organizations can elect to deploy their own NetScaler for Citrix Gateway and StoreFront servers and not consume Citrix Workspace or Citrix Gateway Service for several reasons. These can include:

  • Custom authentication flows
  • Increased branding capabilities
  • Greater HDX traffic routing flexibility
  • Greater insight into HDX network performance (HDX Insight)
  • Auditing of ICA connections and integration into SIEM platforms
  • Ability to continue to operate if the Cloud Connector's connection to Citrix Cloud is severed, by using the Local Host Cache function of the Cloud Connectors with StoreFront

  As with the Cloud Connectors, it is recommended to keep StoreFront and Citrix Gateway components deployed as “hot standby” in the recovery location and not recovering them during a DR event.

Operation Considerations

Maintaining the DR platform is essential to maintaining its integrity to avoid unforeseen issues when the platform is required. The following guidelines are recommended regarding operating and maintaining the DR Citrix environment:

• The Citrix DR Platform is Not Non-Prod. Organizations with a “hot-standby” environment can be tempted to cut corners and treat DR as a test platform. This treatment negatively impacts the integrity of the solution. In fact, DR would likely be the last platform to promote changes to, to ensure that its utility is not affected in the event maintenance went catastrophically wrong in a manner that was not exhibited in non-prod environments.
• Patching and Maintenance. Routine maintenance in lockstep with production when using “hot standby” Citrix platforms is essential to maintaining a functional DR platform. Keeping DR in sync with production regarding OS, Citrix product, and application patches are important. To mitigate risk, allowing several days to a week between patching production and patching DR is suggested.
• Periodic Testing. Regardless of whether DR involves replication of production to a recovery facility or the use of “hot standby” environments, it is important to periodically test (once or twice a year ideally) the DR plan to ensure that the teams are familiar with recovery processes and any flaws in the platform or workflows are identified and remediated. Practice makes perfect. A plan can work on paper but testing reveals the expectations for RTO and RPO objectives are unachievable, an order of operation needs adjustment, or an oversight must be accounted for. For example, the actual time to sync and failover storage might result in an unexpected amount of data loss, requiring adjustment to expectations or replication configuration.
• Capacity Management. True for both Active-Passive and Active-Active “always on” Citrix environments, capacity or use case changes in production must be considered for DR as well. Especially when Active-Active models are used, it is easy for resource utilization to increase beyond say, a 50% steady-state utilization threshold of resources at each environment, only for a DR event to occur and the surviving platform becoming overwhelmed and either performing poorly or failing due to the overload. Capacity can be reviewed monthly or quarterly.

Data Recovery Considerations

In this section, we will briefly cover several concepts and considerations related to dealing with user profiles, VDA images, and applications (App-V, App Layers, MSIX, and so on) in multi-location and DR scenarios.

Replication versus Backup

Both functions of data availability and integrity are important factors when considering the recovery of critical data sets such as core infrastructure, images, user data, and apps. While replication features typically offered in public cloud solutions provide convenient mechanisms to ensure the resilience of critical data, replication does not address data corruption or DR scenarios where even replicated data has been compromised. Replication is not a replacement for backups, and where financially feasible, both must be used. Replication can assist in the rapid recovery of data, while backups can assist in recovering from data loss or corruption (to the extent dictated by the retention policy and backup interval). Whether on-premises or in the cloud, numerous backup options are available.

While on the topic, it is prudent to consider backup retention policies carefully in the current cybersecurity landscape due to ransomware attacks now employing “time bombing” where data sets are compromised but appear normal while the ransomware remains dormant until a specific date. By playing the “long game”, cybercriminals can render backups with shorter retention periods useless upon recovery. Extending retention periods increase costs but improve the probability of recovering critical data.

What Do I Recover? Classifying Data Criticality

While much of this article has focused on availability and recovery options for Citrix infrastructure, consideration is also be given to the availability of various data sets to align business expectations with cost. For example, if user profiles are unavailable in the DR location, will this merely be an inconvenience or a significant impediment to business operations? The answer to this question will directly impact the complexity and costs of the infrastructure components. The following is a list of considerations of common data associated with a Citrix platform:

• User Profiles. Includes Citrix Profile Management (CPM) profiles or containers, User Layers, or FSLogix containers. Organizations must determine if they must be recovered in the DR, and their RTO and RPO. For some use cases, the cost to recover profiles in DR is not worth if creating profiles is merely an inconvenience with 10-15 minutes of extra user burden to set up new profiles in DR. For other use cases, it is non-negotiable. Office Containers can not be worth replicating since they are caches of online data. This differentiation within the user profile/user data discussion allow a reduction in storage and replication costs by electing to not replicate these containers (and makes the case to separate FSLogix profile containers from Office containers).
• Images / Layers. In most cases, there is no real decision to be had on whether to recover images or not (master images, PVS vDisks, App Layering layers) but how they are to be recovered. Are they replicated automatically? Or are they maintained independently between production and DR?
• Applications. Here we specifically are referring to containerized applications (elastic layers, MSIX, App-V). As with images, the decision tends to be more about “how” we recover and not “if”.

Replication Options

Whether in a public cloud or on-premises, multiple replication technologies are available to choose from. This section covers many common options but is not exhaustive. Note that in many scenarios where replication is used, data loss to some degree is inevitable, unless the DR location allows synchronous replication. Otherwise, usually, recovery options have an RPO consideration, often the lowest being 15 minutes (depending on platform/tool/scenario). The more frequent the replication interval, the higher the cost, typically.

User profiles (of all forms), User Layers, apps, and images might be able to withstand a day’s worth of data loss for many organizations, reducing the amount of cross-site traffic and bandwidth consumption. In some use cases, user profile data loss is inconsequential to the business in the context of DR and replication costs. Regardless of the option chosen, both IT and the business must be aware of expectations in terms of potential data loss during an unplanned DR event. Planned DR events allow the opportunity to complete a full replication of data before failing over.

In public clouds, native file services solutions such as Azure NetApp Files, Azure Files, and AWS FSx are commonplace for storing user data and often have built-in replication capabilities within a region or to other regions. If planning for production and DR within a cloud platform and replicating data to other regions, be mindful when selecting regions to use as to what their region pairs are, to ensure that the data can be replicated to the region you desire. Azure maintains a list of cross-replication region pairs here. Azure NetApp Files also has its own cross-replication region pair list here. If deploying in regions that are not paired in Azure data can recover to a region that generates more latency to your VDAs.

In some instances, this can be overcome by not relying on such technologies and using app-level replication capabilities such as FSLogix (if used) Cloud Cache for example. AWS FSx in contrast does not restrict replication via region pairs, allowing greater choice. Be aware of cloud replication options that will not retain SMB connections. Azure for example if using geo-redundant storage (GRS) will not maintain SMB handles or leases, which can impact users depending on the failure scenario and require a logout/login to remount the shares affected for apps, layers, or profiles.

Third-party options, notably NetApp Cloud Volumes OnTap (CVO) are also available on major public clouds (including GCP) and are popular for organizations experienced with NetApp and that desire a similar management and capability experience in clouds.

As with the Azure GRS example, be aware that many technologies that involve file share recovery between data centers or public cloud regions will not retain SMB connections. This can result in the need for users to log out and in upon failover to remount the file shares for technologies that do not automatically retry access and also require manual recovery of the share/volume as well. This is not be noticeable in a full DR scenario where users are logging into a different VDA than production to recover productivity.

On-premises, Citrix Professional Services has long recommended organizations use native filer platforms (that is, IP network file services provided by SANs or NAS devices) where available based on their robust feature sets and perceived lower overhead as compared to Windows-based file server solutions. This is especially important when planning storage replication functions for a DR site.

If hardware/appliance filer solutions are unavailable, Windows-based solutions are a fallback (be sure they are tuned per Microsoft guidance and duly backed up), with various options available but are likely to be more limited in their performance and capacity scalability than built-in solutions:

• Standalone File Server. A basic Windows file server for hosting Citrix data needs including profile and app containers, user profiles, Microsoft Folder Redirection, and so on provided the environment is not 24/7 and can withstand outages.
  • Pros: Simple to set up and maintain
  • Cons: No HA, no replication ability unless layering on another technology such as DFS-R, Storage Replica, or VMware Site Recovery Manager (SRM), Veeam, robocopy, and so on, reboots (planned or unplanned) will interrupt user profiles, user data (such as home drives and Microsoft Folder Redirection), mounted app packages, Personal vDisk access (likely resulting in crashed PVS targets)
• Windows File Cluster. A mainstay technology in Windows environments and relies on Windows Failover Clustering. Intended for creating high availability for file shares within a local data center.
  • Pros: Robust and mature solution ideal for Citrix data needs for various profile and container types, allows failover between nodes without interrupting SMB sessions of clients
  • Cons: Not intended as a DR solution, more of an HA solution. Can be more complex to set up and maintain depending on the shared storage architecture used, not ideal for spanning data centers unless being recovered using a VM-level recovery orchestration tool such as VMware SRM, or using DFS-R or even robocopy to replicate changes between two clusters
• Distributed File System (DFS). DFS Replication (DFS-R) and DFS Namespace (DFS-N) are two of the oldest file server replication and availability options in Windows, initially used to power Active Directory replication functions. They can be configured independently of each other but are often configured together. Its prevalence within long-lived Microsoft-based organizations makes it a familiar solution to many admins. However, it has limited supported applicability to Citrix environments.
• Pros: Familiarity to admins, easy to set up, can span data centers, not reliant on complex cluster configurations (but can be layered over top of them)
• Cons: DFS has many limitations and is not ideal for most Citrix-related data replication use cases. DFS-N and DFS-R are unsupported by Microsoft in combination with each other for various common user data and profile scenarios but can be supported in active-passive (one-way replication) scenarios with manual failover (one that would require users to log out and log back in to re-establish file server access), not suitable for hosting app packages/containers unless acceptable for users to log out and back in
• Storage Replica. Storage Replica enables replication of volumes between servers or clusters in multi-data center scenarios. It can be configured in three forms including stretch clusters, cluster-to-cluster, and server-to-server. Both asynchronous and synchronous replication are possible.
  • Pros: Supports automatic failover when used with Storage Spaces Direct to form a stretch cluster. May be suitable (subject to testing failover performance) for user profiles, user data, profile containers, app containers, and layers. If not using stretch clusters, may be suitable for FSLogix containers only
  • Cons: If not using stretch clusters, failover must be performed manually and will sever SMB sessions requiring a logout/login to re-establish access to user profiles and data. Not suitable for hosting app packages/containers if not using stretch clusters
• Scale-Out File Server. Scale-Out File Servers are ideal for use cases with larger files such as VM disks, or app data that is not particularly write intensive.
  • Pros: SMB continuous availability (reduces downtime when failing over between nodes), ideal for VHDs, app containers, layers, profile containers, can span data centers
  • Cons: Not recommended for user profiles (roaming profiles, CPM), Microsoft Folder Redirection per Microsoft

Depending on the data set, it may not be critical to keep the same namespace path between production and DR data centers. Keeping independent paths for user profiles, app package shares, Elastic or User Layers, profile containers, and so on may reduce complexity. Evaluate implications on a per-dataset type basis, including considerations for reverse replication and failback if not operating in an active-active scenario. In active-active scenarios, homing users to one regional data center over another would be ideal, to avoid mounting apps and data containers between sites and adding latency.

Recovery Considerations for App Layering

For organizations running App Layering, review the Citrix reference architecture for App Layering there are several components where backups are appropriate, including the appliance, Elastic Layer Shares, and User Layer Shares.

• Appliance backups ensure that all the layers are available even if the appliance is somehow destroyed. These backups ensure that all the layers are available even if the appliance is somehow destroyed or corrupted.
• Elastic Layers are mounted from an SMB share at logon. It is critical to ensure that the file server/services used for Elastic Layers are highly available by using appropriate clustering. Using native hardware (or cloud) file share solutions or Windows-based solutions that support continuous SMB availability is important. Using multiple DFS-R targets is not sufficient for this use case because if a target fails, the mounted VHD files can't be remapped to another target until another logon happens. Similarly, Storage Replica in several configurations is not suitable for similar reasons as discussed earlier.
  • For DR, there are two models that can be used to address Elastic Layers
    • Replication Model
    • Dual Appliance Model
• User Layers are more complicated from a DR perspective. These writable VHDs are mounted by Windows at logon and locked by the Windows desktop.
  • As with Elastic Layers, native hardware or cloud file share solutions are optimal due to limitations of common Windows-based methods such as DFS-R or robocopy. Understand that User Layers tend to be larger, with blocks changing frequently, and replication methods that replicate at the block level versus copying the entire file when changed, will be superior otherwise organizations may see untenable amounts of User Layer replication between sites.
  • Due to the difficulty of replicating large disks or the inherent cost implications, organizations most often opt to provide DR desktops for users without a replicated User Layer.

Recovery Considerations for Profiles

Recovery of profiles requires a suitable storage replication and access strategy regardless of whether the profiles are located in on-premises data centers or in the public cloud. This section is intended to build upon the prior considerations for recovery discussed earlier on a per-profile solution basis. For each use case, be sure to ask “Does the profile in fact need to be recovered in a DR scenario?” If it does not, costs and complexity can likely be greatly reduced.

General Recommendations:

• Confirm if the profile data must be replicated at all, or if only a portion of it does. For example, if using FSLogix profile containers, you may want to split it into a profile and an Office container (potentially on different shares for granular replication control if relying on storage replication) to provide flexibility in replicating the profile containers but not Office containers (which during a DR event can be created on demand).
• Lead with using purpose-built “native” file services (filer) platforms on-premises or in the cloud. Numerous Windows options are available but are typically less robust (both in features and scalability), have more overhead, and have more caveats. Local redundancy is strongly recommended in addition to cross-site redundancy to avoid some of the common pitfalls of failover scenarios that can interrupt SMB connections.
• If replicating VHDs (which tend to be large files) with a high rate of change such as User Layers and profile containers, consider replication solutions that can replicate at the block level and do not trigger a full copy of the file on change otherwise storage providers and network throughput can become overwhelmed.
• Consider the RPO interval. The shorter it is, the higher the cost and load on storage and network systems. An asynchronous replication option that balances data loss tolerance and replication interval is optimal.
• Design the solution so that profiles are local to the VDAs in either location to improve performance.
• If using CPM and planning to use DFS, refer to the CPM DR documentation for DFS considerations.
• Subject to testing, consider Cloud Cache to simplify replication and recovery as the feature handles replication of profile data at the application level, and allows administrators to set a preferred list of container target shares (one per file server storage provider) on a per-data center basis (typically via GPO / OU structures). Other considerations for Cloud Cache are provided by Microsoft and must be referenced in planning. Cloud Cache does have drawbacks and certain scenarios can result in outages, so organizations are encouraged to review the potential risks to understand if the rewards of handling replication and recovery of profiles at the application level outweigh them.
• Plan for failback which may entail reversal of replication from DR to production.

Recovery Considerations for Application Packages

When we refer to application packages, we refer to App-V, MSIX, Elastic Layers, and similar application packages collectively. These are characteristically read-intensive but not write-intensive and as compared to profiles, are not updated as frequently. The following are some general recommendations for this data type:

• Lead with using purpose-built “native” file services (filer) platforms on-premises or in the cloud.
• Consider maintaining separate file shares for the applications (that is, not using the same path or namespace) and pointing the VDAs to use their local file share. Keep the file shares in sync manually or via robocopy.

Recovery Considerations for Images

Master images and PVS vDisks must also be considered in the DR recovery strategy. Master images if lost, result in considerable effort to rebuild. App Layering falls into this classification but has been covered already. The following are some general recommendations to consider:

• Replicate master images and PVS vDisks between data centers or cloud regions if relying on the same image between production and DR. Consider manual replication to avoid the possibility of a corrupt image being pushed to DR.
• Back up master images and PVS vDisks (including version files and manifest files), ideally to an offsite location rather than production.
• Use Azure Compute Gallery (formerly Shared Image Gallery) if using Azure, to replicate images across regions, subscriptions, and tenants, providing a shared image repository.

Summary

We have covered a fair bit of ground on the topic of disaster recovery in Citrix. The following points summarize the core messages and takeaways of this guide for architects and customers alike to consider when planning their Citrix recovery strategy:

• Understanding the business requirements and technological capabilities of a customer environment influences the disaster recovery strategy required for Citrix. The recovery strategy chosen must align to business objectives.
• High availability is not synonymous with disaster recovery. However, disaster recovery can include high availability.
• Sharing administrative boundaries between physical locations does not make up a disaster recovery solution.
• Developing a disaster recovery tiering classification for an organization’s technology portfolio provides visibility, flexibility, and potential cost savings when developing a recovery strategy.
• RTO requirements for the Citrix infrastructure and the applications that are hosted on Citrix, is the most significant influencing factor in defining the disaster recovery design. A shorter RTO often requires a higher solution cost.
• Disaster recovery architectures for Citrix which do not employ a “hot” disaster recovery platform present limits on the technologies a customer can use, such as Citrix MCS, NetScaler, and Provisioning.
• A Disaster recovery strategy for Citrix must consider the recovery and recovery time of user data and application back-ends. Citrix can be engineered to recover rapidly, however, users can nonetheless be unable to work if application and data dependencies are not available in a similar timeframe.
• Disaster recovery in cloud environments has a different set of considerations not present in on-premises environments which organizations must review to avoid unforeseen bottlenecks or operational impacts when invoking disaster recovery in cloud environments.
• It is imperative that disaster recovery components are kept up-to-date to match production updates and configurations to maintain the integrity of the platform.
• Platforms that use an active-active configuration for Citrix between locations must be mindful of capacity monitoring to ensure in the event of a disaster, sufficient resources are available to support the projected load at one or more surviving locations.
• Organizations must test their disaster recovery plan for Citrix periodically to validate its operation and ability to serve its core purpose.
• Citrix DaaS significantly simplifies many aspects of the DR architecture and allows Resource Location redundancy via Zone Preference configurations.
• Plan and choose your replication options wisely for user data, application packages, layers, vDisks, and profile containers to ensure that their limitations are known, and their capabilities align with business requirements.
• Do not confuse replication with backup. They go hand-in-hand and give deeper consideration to back up retention because of the prevalence of ransomware “time bombing” backups.

Sources

The goal of this article is to assist you with planning your own implementation. To make this task easier, we would like to provide you with source diagrams that you can adapt for your own needs: design-decisions_cvad-disaster-recovery.vsdx

Evaluating which is the best application delivery method is an activity as old as Citrix and has grown more complex as more application delivery technologies have been developed. Although it is a question asked frequently, the answer isn't always straightforward. Circumstances like different user demands, various application types, and new or changing delivery technologies can strongly impact the evaluation.

This article is meant to be a guideline to help you identify the best application delivery method for the use case at hand based on its requirements. As the ecosystem of applications today has dramatically changed, it is expected to evolve even more in the upcoming years with the incorporation of SaaS-based applications. Hence, different aspects have to be considered during an evaluation process to identify the best delivery method. To simplify this complex process, decision tree diagrams have been created to guide you through the various scenarios. The diagrams are separated into the following segments:

1. Modern vs. Traditional
2. Endpoint vs. Citrix Virtual Apps and Desktops
3. Hosted Shared vs. VDI Desktop
4. Hosted Shared Desktop vs. Hosted Shared Application

The four segments depict different tiers of the application delivery methods and some of the outcomes of a segment lead to a subsequent flowchart. The following overview shows how the tiers relate to each other:

The first and second tiers are most relevant to Solution Architects and Application Business Owners as their outcome is a technology stack rather than a delivery method. Tier three and four are technically driven and related to Citrix Virtual Apps and Desktops delivery methods. Therefore, they are geared towards Engineers and Administrators.

Different requirements, needs, and circumstances lead to different outcomes and therefore a “one size fits all” method does not exist. There is no right or wrong in the evaluation process either, as every environment has its own unique characteristics. Companies with a large user base and complex change management process spread across many branches come to different conclusions than small businesses with one data center and a simple change management process.

While you can run the flowchart for every application, the diagrams are primarily meant to give you a general guidance on a delivery strategy and can also be used to challenge the current install base. The diagrams furthermore contain explanations to almost every decision and its implications and give recommendations for the particular use case.

Please note: The unique characteristics of your environment require all setups and combinations to be thoroughly tested before an implementation to avoid any unforeseen results.

For more resources on the different delivery methods, see Citrix Docs.

Modern vs. Traditional Overview

Modern: For this article, we consider web-based applications, delivered as a Software as a Service (SaaS), as modern. These applications are typically hosted in a cloud computing environment. Web applications located in an on-premises data center can also be considered as modern, as long as the code execution is done on the web server and no client components are needed (except a web browser).

Traditional: Traditional means an application is installed directly on the user’s endpoint and / or Citrix Virtual Apps and Desktops workload. This type is also referred to as classic application. Calling them legacy application would not be accurate since most of the applications today still have to be installed and are not available as SaaS applications.

From a technical perspective, SaaS applications are preferred. The code is run on a web server hosted in a cloud environment, which typically causes less resource usage on the client / front-end side. In addition, scalability and maintenance of the back-end system is no longer your concern as it is taken care of by the application provider. In this model, the application is also kept in an "evergreen" state without major impacts to your environment. On the client / front-end only a browser is needed to access the application. Hence, little to no maintenance work related to the application is needed here either. This setup also enables you to use any device of your choosing since there is no dependency on the operating system. Citrix Workspace is the ideal platform to deliver and manage SaaS applications in a secure manner. Features and solutions like Secure Private Access, and Security Analytics provide a unified and therefore best user experience with the highest security possible.

However, there are reasons why SaaS applications cannot be used. For instance, if technical, legal and / or security requirements can’t be met, a traditional approach needs to be considered. In such a scenario, it’s best to identify the exact reasons why the usage of SaaS is not possible. Once identified, it's recommended to clarify if a partial integration or a transition in stages is possible, to benefit from the advantages offered by SaaS technologies.

Endpoint vs. Citrix Virtual Apps and Desktops Overview

Endpoint: Installation on the physical client device.

Citrix Virtual Apps and Desktops: Application virtualization via Citrix Virtual Apps and Desktops, where the applications are installed on a Hosted Shared server or VDI Desktop. The exact Citrix Virtual Apps and Desktops delivery method will be determined in the subsequent segments.

Device Diversity

The ever-increasing number of digital natives joining the workforce compels companies to expand their endpoint portfolio with non-Windows devices as well. Also, SaaS applications are further enabling users to access applications regardless of the type of device and operating system in use. The demand to allow devices which are not Windows-based has drastically increased over the last couple of years. To allow a bring-your-own-device (BYOD) or chose-your-own-device (CYOD) approach, Citrix Virtual Apps and Desktops can be used to deliver Windows-based applications to non-windows devices as well.

Security

Moving applications to Citrix Virtual Apps and Desktops lowers the client footprint and allows a zero-trust architecture. Citrix virtualization and networking technologies provide robust methods for segmenting users, applications, and data while still providing a seamless user experience. This way, the network traffic can be streamlined. The endpoint to server network communication will be reduced to a minimum, which in turn reduces the exposure of your server network. The application data traffic between the front-end and back-end will solely reside within the confines of your server network.

Contractors

Usually contactors already own devices. Instead of handing out corporate devices, Citrix Virtual Apps and Desktops along with Citrix Gateway can be used to allow a secure access to applications, desktops, and other resources. This approach lowers the endpoint costs and maintenance efforts.

Time-to-Market

Installation of applications on numerous endpoints can be a tedious, time consuming and error-prone task, as the installation needs to be ran on every device. This circumstance especially applies to large enterprises, with thousands of devices spread across the globe. In such use cases, an application release can take weeks or even months until it has been distributed to every device. If issues occur, a rollback might be an even more complex and time-consuming endeavor.

Citrix Virtual Apps and Desktops allows you to centralize the application management. Application releases are independent of the client device, since the update is done on Hosted Shared servers or VDI Desktops in a corporate data center. Furthermore, it’s highly recommended to use Citrix Provisioning Services or Machine Creation Services to benefit from Citrix’s market leading image management capabilities. Both image management solutions allow a consistent installation baseline across all virtual machines and provide the fastest rollout and rollback methods. Releases can be rolled-out or rolled-back with a simple reboot of the virtual machine which reduces the time-to-market of new application deployments to a minimum.

Mobile Workforce

Mobile users are often traveling and need to access applications offline as well. While offline, editing documents or writing emails are the most common tasks performed. In such a scenario the application has to be installed on the endpoint. Most of today's business applications however, require a back-end connectivity to work. Which in turn means, that the mobile user has to be online to use the application. The Citrix HDX protocol enables mobile workers to access applications with great user experience even on a low bandwidth or high latency connection.

Hosted Shared vs. VDI Desktop Overview

Hosted Shared (multi-user): Hosted Shared systems are VDAs based on a Windows server operating system with the Remote Desktop Session Host role (formerly known as Terminal Server) installed. This type is referred to as Multi-session OS / Server OS VDAs and is shared among multiple users simultaneously.

VDI Desktop (single-user): In this article, VDI refers to Single-session OS / Desktop OS VDAs. This delivery type is based on a client operating system and used exclusively by a single user at a time.

Generally, Hosted Shared Desktops tend to be more cost effective as multiple users are hosted on a single machine. Nevertheless, there are use cases where a VDI Desktop is preferred, such as to support resource (CPU, memory, disk) intensive applications. Also, users who need administrative privileges to work, require a VDI due to security considerations and to have the ability to install and change the desktop according to their needs (without impacting others). There are also customers using VDI because the operational and processual synergies with other solutions outweigh the additional cost overhead.

Hosted Shared Desktop vs. Hosted Shared Application Overview

Hosted Shared Desktop: This method is a desktop published to multiple users on a single Multi-session OS.

Hosted Shared Application (multi use): With the Host Shared Application model (multi use), multiple applications are installed on the same server and shared among some users. It’s based on a Multi-session OS as well and sometimes referred as a siloed approach. In this model, applications are delivered virtually and displayed seamlessly in high definition on user devices.

Hosted Shared Application (single use): The only difference between multi and single use is, that Host Shared Application single use has only a single business application installed. This application can still be used by multiple users at once. Important: This type of solution can be avoided as much as possible, as it is inefficient from a resource (costs) and maintenance (efforts) point of view.

The approach in this segment is slightly different to the others. There are many different combinations how these three delivery methods can be utilized. Because of this, we tried to identify the optimal delivery method based on the challenges our customers and partners face the most. It is important to work closely with the appropriate Business Application Owners to understand the application's characteristics in detail and therefore to better assess which delivery models can be used.

From the operational perspective, placing as many applications as possible on a single image can often lead to a reduction of maintenance efforts. Fewer images mean less work. However, this requires that there are no technical conflicts between these applications. Sometimes changes on one application require testing all other applications on the image as well. Hence, it’s important to reflect on the change and release management process of every application in detail, to avoid organizational conflicts. Hosting the application on a file share (if at all possible) or via App-V (Shared Content Store) can even more simplify the release process, as changes can be applied without going through an imaging process. Both options can't be used for all use cases and require extra and appropriately sized infrastructure. Regardless, these methods should be at least considered as it can help to reduce the number of image changes.

Other factors such as security requirements and performance utilization can also have an impact on the decision-making process. Especially applications with unpredictable resource utilization and regular CPU-bursts have a negative impact on other applications and its users. Such bottlenecks have to be avoided at any cost, because then all users on that system suffer from a bad user experience. Workspace Environment Management can help to mitigate such performance bottlenecks. Applications, where bottlenecks can’t even be handled by Workspace Environment Management, can be placed on dedicated servers (Hosted Shared Application single use). This type of setup ensures that the necessary resources are available and avoids negative impact on other applications.

Summary

In this article, we have reflected the most common decision factors when choosing an application delivery method. This guide can help you to identify the optimal method for your own unique environment.

Sources

The goal of this article is to assist you with planning your own implementation. To make this task easier, we would like to provide you with source diagrams that you can adapt for your own needs: design-decisions_application-delivery-methods.vsdx

To meet different user requirements, the Citrix HDX protocol allows for different graphics modes to be configured. The purpose of this article is to outline the different HDX modes and how they are configured. It gives you a starting point from where you can configure your environment to best fit the needs of your users, your workload, and the current network conditions.

Important to note: This article is based on Citrix Virtual Apps and Desktops 1912 unless stated otherwise.

DISCLAIMER: Results may vary and it is highly recommended to run your own tests to see what works best for your use cases.

HDX Graphics Overview

Before diving into the specific graphics policies, let’s review how we categorize what you see on your HDX session screen and the underlying technologies that are used for presentation.

As we deliver graphic content for applications or desktops the HDX graphics encoding engine, Thinwire, analyzes the content on screen in real-time and dynamically categorizes the display data into three types:

• Text, Simple Images and Solid Colors
• Static Image Content
• Moving (or Fluid) Images

In this example, text or simple images are highlighted in blue, static images in orange and moving (or fluid) images in green.

Within Citrix Virtual Apps and Desktops, Thinwire can take different approaches for display analysis, compression, and delivery: Citrix adapts the use of industry-leading standards, Advanced Video Coding (AVC), also called H. 264, High-Efficiency Video Coding (HEVC), also known as H.265 and AOMedia Video 1 (AV1) for efficient delivery of high-quality video content in its “Full-Screen” and “Selective” codec implementations.

• Choosing to Configure Thinwire to not use the Video Codec or Configure Thinwire to use the Video Codec for Actively Changing Regions allows Thinwire to sense regions of transient content (fluid images or video) and encodes it based on set policy and capabilities detected on the endpoint. Thinwire encodes these “selected” (or transient) regions either as Adaptive JPEG or by employing a video codec (Hx264, H.265, or AV1). Adaptive JPEG and “Selective” H.264 / H.265 / AV1 are considered subfeatures as Thinwire is the core technology. The remaining, non-transient regions (encoded as JPEG and Run-Length Encoding (RLE)) are then combined to complete the in-session display.
• Choosing to Configure Thinwire to use the Codec For the Entire Screen tells Thinwire to treat the entire screen as transient content, except for text (by default), and encodes display data using H.264, H.265 or AV1 video codecs, depending on the server and client capabilities.
  Text is then overlaid onto the screen to provide a complete image. Newer video codecs like AV1 and H.265 can achieve higher compression over H.264 without compromising quality. However, AV1 and H.265 are expensive in processing and are only supported when used with select GPUs on the Virtual Delivery Agent (VDA).
  Neither AV1 nor H.265 can be used when CPU encoding is used. Also, AV1- or H. 265-compatible hardware, in the form of GPU or purpose-built thin client, is required to decode AV1 and H.265 display data on the client endpoint.
  Review vendor documentation to determine AV1 and H.265 supportability for your endpoint hardware.

AV1 is supported from CVAD 2305 (NVIDIA) or CVAD 2308 (Intel) and CWA 2311 and onward.

Compared to both H.264 and H.265, AV1 has better compression and image quality. Encoding and decoding AV1 is currently impossible to do without a compatible GPU. Citrix supports AV1 encoding for both NVIDIA and Intel GPUS.

The requirements for AV1 are as follows:

• VDA 2305 or higher for NVIDIA GPUS or
• VDA 2308 or higher for Intel GPUs
• And a compatible GPU for encoding, see below:
• NVIDIA Ada Lovelace-based GPU
• Intel ARC or Intel Data Center GPU Flex Series GPUs

Depending on the configure HDX mode, these categories are encoded by different means:

• Text and simple images are almost always encoded in a lossless fashion using Run-Length Encoding (RLE). Starting with Version 7.17, a Citrix proprietary RLE flavor called MDRLE is used which allows for a better compression rate CTX232041. Enabling the Optimize for 3D graphics workload disables the lossless text detection and transfer the content with H.264 / H.265, rather than RLE. You can visualize this policy through our Visio diagram later in this article.
• For static images with both Selective H.264 / H.265 / AV1 and Adaptive JPEG, JPEG is used for encoding while the video codec is used if Full-Screen H.264 / H.265 / AV1 has been chosen as the graphics mode. If JPEG is used, the quality of it can be configured with the Visual Quality setting. Check the attached Visio chart later in this article for more details.
• For moving images the video codec H.264 / H.265 / AV1 is used when configuring Full-Screen H.264 / H.265 / AV1 or Thinwire with Selective H.264. If Thinwire with Adaptive JPEG has been configured, JPEG is used with a quality that automatically adapts (hence the name) to conditions such as frame rate and available bandwidth.

To recap, Thinwire uses different technologies when configured as follows:

Configure Thinwire to not use the Video Codec

• Text: RLE
• Simple Images and Solid Colors: RLE
• Static images: JPEG
• Moving Images: Adaptive JPEG

Configure Thinwire to use the Video Codec for Actively Changing Regions

• Text: RLE
• Simple Images and Solid Colors: RLE
• Static images: JPEG
• Moving Images: H.264 / H.265 / AV1

Configure Thinwire to use the Codec For the Entire Screen

• Text: RLE (or H.264 / H.265 if Optimize for 3D graphics workload has been enabled)
• Simple Images and Solid Colors: H.264 / H.265 /AV1
• Static images: H.264 / H.265 / AV1
• Moving Images: H.264 / H.265 / AV1

In the next section, we will cover the policies to achieve the behavior already mentioned.

HDX Graphics Modes

The Use Video Codec for Compression policy is the central function to give your end-users an optimal experience by configuring display methods appropriate for different use cases.
We map the technologies outlined with configurable policy settings within Citrix Studio.

• For Actively Changing Regions = Thinwire with Selective H.264 / H.265 / AV1
• Do Not Use Video Codec (default fallback method) = Thinwire with Adaptive JPEG
• For the Entire Screen = Thinwire Full-Screen H.264 / H.265 /AV1
• Use When Preferred (policy default) = Thinwire with Selective H.264 / H.265 is used unless Optimize for 3D Graphics Workloads is also set, then Thinwire Full-Screen H.264 / H.265 / AV1 is used.

Capabilities of both the client endpoint and the Virtual Delivery Agent (VDA) are evaluated at session launch or session reconnect. If the client does not support H.264 / H.265 / AV1, the display method is Thinwire with Adaptive JPEG regardless of the policy set on the VDA.

Configure Thinwire to use the Video Codec for Actively Changing Regions

The For Actively Changing Regions graphics mode is our most balanced setting. As such, we recommend starting with this mode as you begin to baseline policies within your environment since it covers a wide user base (for example, Office workers with occasional video playback). At its core, this mode uses JPEG for still images, RLE for text, simple images, solid color blocks, and bitmap caching for areas of the screen that the VDA determines to be static. Thinwire continuously analyzes the screen for regions of fluid movement, such as multimedia, and selectively uses a video codec: H.264 / H.265 / AV1 to encode the fluid region.

As illustrated, the video codec usage (H.264, H.265 or AV1) is “Inactive” until regions of fluid movement are detected. The VDA then transitions to a video codec in the form of H.264, H.265 or AV1 to encode the selected region during fluid movement and returns to an “Inactive” state once the selected region no longer contains fluid content.

The usage of a video codec, whether it is H.264, H.265 or AV1, provides a richer experience than adaptive JPEG at the expense of CPU to compress regions of fluid movement. Network bandwidth will generally be less with a video codec compared to Adaptive JPEG for multimedia workload, and newer codecs have better compression and, therefore generally speaking, lower bandwidth usage. It is highly recommended to run your own tests with your specific use case (check the Tools section below).

Newer codecs produce higher quality images at the same bandwidth utilization and higher FPS on lower bandwidths, as AV1 uses lower bandwidth per frame when compared to H.264 / H.265.

Configure Thinwire to not use the Video Codec

The Do Not Use Video Codec offers maximum compatibility for client endpoints, including endpoints that do not support the decoding of H.264 / H.265 graphics.

In this graphics mode, Thinwire behaves similarly as when it is configured For Actively Changing Regions. The VDA analyzes the screen for regions of fluid movement. Rather than encoding with H.264 / H.265 / AV1, however, Thinwire encodes moving images as Adaptive JPEG to deliver high compatibility or the use of a video codec is not needed. The remaining regions are presented as JPEG for still images, and RLE for text and simple graphics to deliver quality imagery.

CPU processing for encoding moving images using Adaptive JPEG is typically lower than with Thinwire with H.264 for Full Screen or Actively Changing Regions. This mode is desired if server scalability is your priority. The trade-off is seen in terms of increased bandwidth and decreased moving image fidelity in WAN scenarios. This graphics mode is recommended for use cases in which moving images are minimal, such as in a call center or point of sale system. In which case, the bandwidth utilization in this mode would be similar in comparison to Thinwire configured with Video Codec for Actively Changing Regions.

The Do Not Use Video Codec policy setting is the default fallback method for the other two graphics modes (Use the Video Codec for Actively Changing Regions or For the Entire Screen).

Configure Thinwire to use the Codec For the Entire Screen

The For the Entire Screen graphics mode setting configures the VDA to encode all display data using a video codec, except for text. Text is encoded using RLE and is overlaid with the remainder of the screen. If Optimize for 3D Graphics Workloads is enabled the entire screen, all content including text, is encoded with a video codec (H.265, H.265 or AV1).

Configuring Thinwire to use the Video Codec for the Entire Screen is designed for the heavy multimedia use case, where larger regions of the screen are in motion. Higher compression and quality are achieved at the expense of CPU and server scalability.

On its own, this mode provides a good user experience when heavy multimedia, 3-D modeling, or CAD drawing applications are in use.
The CPU can quickly become a bottleneck, if undersized, resulting in poor performance and user experience under heavy multimedia conditions.
Consider GPU offload capabilities to supplement this graphics mode while using these application types.

By default YUV420 is used as color space. With Full-Screen H.264 and Full-Screen H.265, you can choose between YUV420 or YUV444:

As you can see, YUV444 results in a better quality but significantly impacts the bandwidth requirements. When using YUV 444 with H.264, this will also disable hardware decoding on the client side.

With the release of Citrix Workspace app 2305, we introduced H.265 4:4:4 support, enabling us to offload the decoding to a capable Client GPU, thus lowering CPU utilization and significantly improving performance.

Citrix internal tests showed a 4X increase in Frames per Second (FPS) compared to H.264 4:4:4 and a 2.5X reduction in size per frame compared to H.264, due to H265’s superior efficiency. As a customer, this means you get higher FPS at the same bandwidth utilization.

You can enable YUV444 for Full-Screen H.264 with the following settings:

• Visual Quality: Always lossless / Build to lossless
• Allow visually lossless compression: Enabled

YUV 444 with Full-Screen H.265 has the following extra requirements:

• VDA
• CVAD 2209 or newer
• NVIDIA Pascal GPU or newer

Policies:

• Use video codec: “Use when Preferred” or “For the Entire Screen”
• Optimize for 3D Graphics: “Enabled”
• Visual Quality: “Build-to-Lossless” or “Always Lossless”
• Allow Visually Lossless: “Enabled”
• Hardware Encoding: “Enabled”

Client:

• Citrix Workspace app for Windows 2305 or newer
• Enable H.265 support
• Client GPU that supports H.265 4:4:4 decoding:
• NVIDIA Turing or newer
• Intel 10th Gen or newer

Check the Visio chart in this article for more details.

Build to lossless

Build to lossless is a special Thinwire configuration that optimizes graphics delivery for interactivity and final image quality. You can enable this setting by setting the Visual quality policy to Build to lossless.

Build to lossless compresses the screen using H.264, H.265 or AV1 during screen activity and sharpens to pixel perfect (lossless) when activity stops. The lossy image quality adapts to available resources to maintain the best possible frame rate. The sharpening step is performed gradually, radiating outwards from the mouse cursor. This allows for an immediate response if the user begins screen activity shortly after sharpening starts. For example, selecting a model and rotating it.

H.264 Build to lossless offers all the advantages of using a video codec for the entire screen, including hardware acceleration, but with the added benefit of a final, guaranteed lossless screen. This is critical for 3D-type workloads that require a final pixel-perfect image. For example, manipulating medical imagery. Also, H.264 Build to lossless uses fewer resources than full screen H.264 4:4:4. As a result, using Build to lossless usually results in a higher frame rate than Visually lossless H.264 4:4:4.

HDX Graphics Configurations

As the Use Video Codec for Compression policy is a good starting point to baseline your configuration, more policies can be set to further customize your visual policies to fit your different workloads.
By customizing these supporting policy settings, you can opt to reduce quality in certain areas to reclaim resources and achieve higher scalability and save on bandwidth.
You can also opt to increase quality to support use cases requiring precise visualizations, as in the healthcare industry. The chart below outlines these settings (click image to view full size PDF):

design-decisions_hdx-graphics_overview.pdf

Furthermore, review our Use Cases Section below to discover how these additional policies (listed below) can reduce resource consumption, although at a slight reduction in quality (sometimes).

CPU or GPU

By default, all processing for encoding graphics occurs within the CPU on the VDA. AMD, Intel, and NVIDIA graphics cards are currently supported to offload encoding to the GPU before being sent to your endpoint for decoding.

Offloading graphics encoding to a GPU will free up resources on the CPU for other tasks, resulting in a better overall experience for the end-user.

Due to varying GPU feature support, visit Citrix Docs to review feature supportability for each vendor GPU when enabling the Use hardware encoding for video codec policy setting.

Use Cases

Once the settings details are known, the obvious next questions are: “What HDX mode is best for my use case?” or “Are there any configuration recommendations?” As usual, the answer is: It depends. Usually, a "one size fits all" approach may not be the best approach but rather different settings for different use cases. So, the first questions you have to ask yourself are: What challenges and use cases do I have? Are there are any intense graphics workloads, any multimedia requirements that I need to fulfill? How is the network connection of the users?

Usually, configuring Thinwire to use a video codec for actively changing regions is the best choice. Also, it is a good idea to explicitly configure the different settings to ensure that the same settings apply even after an update of your environment. As you can see in the following link, the default HDX mode used has changed over time HDX Graphics Encoder Configuration Overview – What Really Matters. Therefore, explicitly configure the HDX mode you would like to run. Generally, refrain from using "Use Video Codec for Compression: use when preferred" as this setting may have a different effect depending on the type of OS, Hardware, and VDA version you are running. Also avoid configuring any Citrix policies that are linked to legacy graphics mode. These settings are only supported on Windows Server 2008 R2 and Windows 7 and are left for compatibility reasons.

To give you an idea on how to start, we have created a few base line configurations for a few generic use cases below. Still, we recommend you run your own tests to ensure you have the best mode configured for your specific needs:

Low Bandwidth

This use case describes a user connecting through a connection with serious bandwidth constraints. The following baseline is a good start:

• Use Video Codec for Compression: Do not use video codec
• Visual Quality: Low
• Preferred color depth for simple graphics: 8 bit / 16 bit
• Extra Color Compression: Enabled
• Target Frame Rate: 15
• Target minimum frame rate: 10
• Moving Image Compression: Enabled
• HDX Adaptive Transport: Preferred

As you can see, even with a low bandwidth connection we oftentimes do not set the color depth to 8 bit but keep it at 16 bit. While 8 bit can lower the bandwidth requirement substantially, it also comes with a significantly lowered user experience. Therefore, 8 bit is only recommended for the most extreme cases where access won't be possible otherwise.

Call-Center / Point-of-Sales

This use case describes a call-center or point-of-sales workplace with no special multimedia requirements. The goal is to find a good mix between user-experience and user density:

• Use Video Codec for Compression: Do not use video codec
• Visual Quality: Medium
• Preferred color depth for simple graphics: 24 bit
• Extra Color Compression: Disabled
• Target Frame Rate: 20
• Target minimum frame rate: 10
• Moving Image Compression: Enabled
• HDX Adaptive Transport: Preferred

Task Worker

In the task worker use case, a user has some multimedia requirements such as watching videos online in addition to using a basic set of office applications:

• Use Video Codec for Compression: For actively changing regions
• Use hardware encoding for video codec: Enabled (where available)
• Visual Quality: Medium
• Preferred color depth for simple graphics: 24 bit
• Extra Color Compression: Disabled
• Target Frame Rate: 30
• Target minimum frame rate: 10
• Moving Image Compression: Enabled
• HDX Adaptive Transport: Preferred
• Hardware Acceleration for graphics: Enabled (configured for Citrix Workspace app if available)

3D workload

For 3D workloads such as CAD / CAE, the user experience is key, therefore the following settings are used:

• Use Video Codec for Compression: For actively changing regions
• Use hardware encoding for video codec: Enabled (where available)
• Visual Quality: Build to lossless
• Target Frame Rate: 30 (can be 60 if necessary)
• Target minimum frame rate: 10
• HDX Adaptive Transport: Preferred
• Hardware Acceleration for graphics: Enabled (configured for Citrix Workspace app if available)
• H265 Decoding for graphics: Enabled (configured for Citrix Workspace app if available)

Endpoint Device Consideration

Our goal is to support delivery of your Citrix Virtual Apps and Desktops to any device, anywhere.

On the surface this sounds appealing. However, this does not necessarily mean that all capabilities are present on all endpoints. For example, H.264 / H.265 / AV1 decoding support may be missing, or supported only within specific boundaries such as the max monitor resolution or max number of monitors.

We recommend reviewing the vendor documentation for your chosen endpoint to determine overall supportability for H.264 / H.265 / AV1.

Thin Clients

Most thin clients are purpose-built for targeted use cases, with specific software configurations that are optimized for their hardware platform. We encourage working with your vendor to evaluate a thin client test unit within your environment before purchase to ensure that the endpoint meets your organizational needs.

It is highly recommended to visit our Citrix Ready website during the research phase of your project to determine if the hardware under consideration has passed evaluation and is compatible with the features you desire.

The Citrix Ready program classifies thin clients based on qualified use case capabilities:

• HDX Ready – Supports Task Workers accessing basic office applications and light multimedia.
• HDX Premium – Supports similar workloads as HDX Ready endpoints Additionally, HDX Premium endpoints provide support for Unified Communications such as Skype for Business.
• HDX 3D Pro – Supports Power Users requiring high-end endpoint performance when accessing graphically intensive applications, such as CAD, Geographical Information System (GIS), and medical imaging related software H.264 / H.265 Codec support is required to pass qualification.

You can find the certification criteria for the features under each HDX level here.

Thick Clients

If you manage thick client endpoints in your environment, consider the following components when determining Codec Support (H.264 / H.265):

• Operating System: Some Linux distributions require other libraries to be installed.
• Browser: Citrix Workspace app for HTML5
• Citrix Receiver/ Workspace app version: The Feature Support Matrix can be found here.
• GPU Offload Capability
  • Requires extra configuration for Citrix Workspace app for Windows and Citrix Workspace app for Linux to enable Hardware Acceleration for Graphics with compatible GPU.

Tools

Are there any tools that help you configure your environment? Yes, there are quite a few to help you on your journey on finding the perfect configuration set for you:

HDX Monitor

HDX Monitor helps you to check what settings are actually in effect in a particular session. The latest version can be found here.

Graphics status indicator

The build-in graphics status indicator can be enabled through Citrix policy by enabling the Graphic status indicator setting and will show you the current settings in the Citrix session:

Key Takeaways

The Use Video Codec for Compression policy lets you choose between the different HDX graphics modes illustrated above. Each mode has its benefits and trade-offs in terms of resource consumption, whether it be CPU or network utilization. Resource consumption, particularly CPU, affects server scalability.

Other policies, such as Visual Quality, Target Framerate, and others can be customized to offset the resource consumption at the expense of minor visual quality, or increase quality where it is needed most. Customize these policies to fit the uses-cases within your own environment. Refer to the Visio Diagram to guide you through the process.

Endpoint selection is essential for compatibility with your selected graphics mode. The VDA configures Thinwire to not use the video codec as a fallback method, for endpoints without H.264 support.

Use our built-in tools (HDX Monitor and the Graphics Status Indicator) to evaluate if your policy settings have met your desired outcome.

Thinwire For Actively Changing Regions is oftentimes an appropriate starting point. However, knowing your use cases and configuring your environment accordingly is the best approach to deliver a rich experience to end-users.

Sources

The goal of this article is to assist you with planning your own implementation. To make this task easier, we would like to provide you with source diagrams that you can adapt for your own needs: design-decisions_hdx-graphics.vsdx

These improvements help ease our customer's migration to the Azure cloud and spin up any workload in a matter of minutes.

One of the new Citrix services includes the Image Portability Service (IPS).
This service provides a way to port images between your on-premises data center and your Azure environment. The service uses a temporary Virtual Machine (VM) to host the Compositing Engine (CE).

The CE has two modes: Prepare or Export:

• The Prepare mode converts virtual disk formats used for Citrix Workloads and updates the portable properties.
• The Export mode copies the prepared disk to the target cloud. A connector appliance controls the individual jobs, spawns the CE VM, and secures communications between Azure and your on-premises environment.

Azure supports a version of Citrix Provisioning Services (PVS) which allows you to stream a virtual disk to multiple VMs simultaneously.
With PVS, you can create 2500 identical VMs within a single subscription. Most of the administration and the functionality are the same as for an on-premises PVS environment, except for a few changes.

Azure does not allow the use of the Pre-boot Execution Environment (PXE), so changes were required for the streaming and boot process. This version of PVS includes a UEFI-based Boot Disk Manager (BDM) to create UEFI boot disks.
These boot disks require Azure Gen2-VMs and cannot support 32-bit operating systems. The Citrix broker is now responsible for all power management of the PVS VMs, though the PVS console can still power off the targets.
The Citrix Virtual Apps and Desktops Setup Wizard handles all the provisioning steps.

How can I use the image portability service to move my Citrix workloads?

• Deploy golden images on-premises, using either Machine Creation Services or PVS.

• The entire process for using IPS is completed through PowerShell commands from a remote workstation.

• The latest version of PowerShellGet should be installed on the remote workstation before beginning the process.

• Citrix connector appliances must be installed at each resource location where IPS is used.

• At the on-premises location, a Windows SMB File share is required for temporary data storage during export jobs. The file share should have enough free space to hold two copies of the disk to be exported.

• Automated publishing is only available with PVS on Azure deployments. Manual publishing to Azure is available for both PVS images and MCS images.

• The following machine catalog configurations have been tested with the IPS

  • Windows Server 2016, Windows Server 2019, or Windows 10 2004 or later
  • Source images provisioned by Citrix Provisioning 1912 or later
  • Citrix Virtual Apps and Desktops VDA 1912 or later

What are the limitations and requirements for deploying PVS in Azure?

• The Azure subscription must have the ReserveMacOnCreateNic feature enabled.

• At most 2500 VMs can be streamed in a single subscription.

• All provisioned VMs must be created in the same region as the hosting unit. VMs are automatically spread across all availability zones in the target region.

• Cross-region provisioning is not supported.

• Requires UEFI boot of Generation 2 Azure VMs. Generation 1 (BIOS-based) VMs are not supported. Boot images that are created using the PXE or ISO format are not supported.

• Supports only 64-bit versions of Windows 10 and Windows Server 2019 for streaming.

• Use the Citrix Image Portability Service to import an existing image.

• Active Directory support is required for machine naming using one of these methods:

  • Azure Active Directory Domain Services (AADDS) added to your Azure Active Directory (AAD) tenant
  • ExpressRoute connection to your on-premises Active Directory environment
  • Active Directory domain controllers installed in Azure and synchronized with your on-premises AD environment and the AAD tenant

• When using Azure Files Services for PVS vDisk storage, premium storage or Azure NetApp Services are required and must be in the same region as the PVS server.

• SQL Server or SQL Server Express VM are required. Currently, using any authentication method except Windows Integrated Authentication or using an Azure SQL database is not supported.

• The Golden VM must have the same disk and vGPU configurations.

• Set up a virtual network for streaming to targets and peer that network with the network used for the VM communication to Active Directory. Use the AD Domain Controller IP addresses for DNS servers.

• The PVS Server VM must have at least one vNIC on each virtual network where targets reside. The PVS server should also have at least 2 vCPUs and 8 GiB of memory.

What are the best practices for Image Management in Azure?

• Always make a copy or snapshot of your golden image and use the copy or snapshot for machine catalog images. This practice allows for easy image updates and provides protection against image corruption.

• Point existing image-based machines (PVS and MCS) to the cloud connectors by modifying the ListOfDDCs registry key on the golden images. The registry key can be found at HKLM\Software\Citrix\VirtualDeliveryAgent. After modifying the registry key, take a snapshot of the image. Update the machine catalog with the new snapshot when ready for the images to register with Citrix Cloud.

• Use the Citrix Group Policy (Computer Configuration > Citrix Policies > Controllers) to point the other Citrix workload servers to the FQDN of the cloud connector and set the Enable auto update of Controllers to “allowed”.

• Use Managed disks for the golden images, unless you are using App Layering.

• Be sure to keep copies of golden images is different regions.

• Automate the build of the Golden Image

  • PowerShell SDKs can be used for full automation of the Golden Image build

    • PowerShell v5.x
    • Azure PowerShell Module
    • Citrix Cloud Remote PowerShell SDK

  • Use Azure PowerShell to do the following:

    • Create a new Azure VM
    • Configure the Windows Firewall
    • Remove the Azure Public IP address (if present)
    • Automate domain join
    • Automate software installation such as the VDA
    • Seal the image
      • Reset Log files
      • Reset Office Licensing
      • Clear GPO cache
      • Remove user profiles used for the build process
    • Copy golden image to business continuity region

  • Install any other software manually that cannot be automated.

  • Use the Citrix Cloud Remote PowerShell SDK to do the following:

    • Update the Machine Catalog

  • Do not store passwords unencrypted in any scripts or change any stored passwords (such as the local admin password) immediately after building.

Should I do anything different from on-premises when managing MCS images in Azure?

• Use the Azure shared image gallery for storing MCS images to reduce the time required to create and hydrate the OS disks and improve application performance.

How do I manage images across geographic regions?

• Use Azure shared image gallery to provide multiple replicas in different regions.

How do I enable MCSIO in Azure?

• MCSIO can be enabled on MCS catalogs in Azure by using PowerShell to create a new Provisioning Scheme and Catalog
  • Use VDA version 1912 or later for the best results since not all earlier versions are supported.
  • Do not forget to pre-create a Resource group in Azure for the MCS catalog.
  • Install Citrix Remote PowerShell SDK to access Citrix Cloud configurations.
  • In Azure, the write-back cache is stored on non-persistent media.

Links to Other Resources

Automate the cloud! Citrix + Azure + MCS + PowerShell

Citrix Provisioning on Azure

Enable persist write-back cache for MCS pooled catalog in Azure

Image Portability Service

Microsoft Azure Resource Manager virtualization environments

SDKs and APIs

How do I migrate my Citrix VDA hosts to Azure?

• Azure Migrate integrates with Azure services and supports other third-party tools. Azure Migrate helps you complete the migration process from an on-premises data center to Azure.

Are there any utilities available to help migrate applications and application data to Azure?

• Azure Migrate includes tools to help discover and assess the current environment and plan the migration of applications, servers, and data to Azure.

How do I use Azure for capacity planning?

• Use the Azure Migrate Server Assessment for planning the migration when possible.

• Movere (Microsoft SaaS offering) is another possible Microsoft tool to assist with migration assessment. Movere is only available through the Microsoft Solution Assessment and Microsoft Cloud Economics Program.

How do I move my application data to Azure?

• Azure Migrate offers an Azure VMware Solution tool in Preview to help with migrating all of your servers, including the Citrix workloads.

• Use Azure VMware Solution (AVS) integration to provision your Citrix VDA workload just like other vSphere workloads in your on-premises data center

Azure VMware Solutions

Using Azure VMware Solution (AVS) provides the following advantages if you are currently use VMware virtualization in your on-premises data center:

• Cloud Migration Support: Easily migrate desktops and applications between VMware deployments, including replicating up to 500 VMs simultaneously.

• Familiar Administration: Administrators are familiar with the VMware interface and are comfortable with the administration tools and interfaces.

• Data center extension: Use the data center extension to provide burstable capacity and support remote locations. The data center extension helps during periods of high demand, disaster recovery or business continuity by taking advantage of the rapid elasticity provided within AVS.

Links to Other Resources

What is Azure VMware Solution?

Azure Migrate

Microsoft Azure Resource Manager virtualization environments

Migrate VMware VMs to Azure

Migrate Workspace Environment Manager

Migrating Citrix Virtual Apps and Desktops from on-premises to Citrix Cloud

Migrating Citrix Virtual Apps and Desktops from VMware vSphere to Microsoft Azure

Reference Architecture: Citrix DaaS - Azure

Select a VMware Migration Method

Selecting assessment and migration tools

Windows 10 Deployment Guide

Windows 11 Deployment Guide

• Machine Creation Services (MCS)
• Citrix Provisioning (PVS)

It is also important to mention that Citrix App Layering is out of scope for the current version of this document. Implementation of Citrix App Layering can influence many of these design decisions and we will include it in one of the future updates for this article.

This article is focused on design decisions and factors involving image provisioning. If you are interested in more general reference architecture for Citrix PVS or MCS, we highly recommend reading Citrix Virtual Apps and Desktops Image Management reference architecture.

Overview of Citrix Provisioning Services (PVS)

Citrix Provisioning is a software-based streaming technology that can deliver centralized, shared operating system image to multiple virtual or physical endpoints. For design purposes, it is important to understand that PVS is active component - it is actively involved in the daily operations of image management and delivery. As an advantage, Citrix PVS can reduce the operational and storage costs, since it acts as a software-based storage offloading solution. This advantage, however means that the environment needs to be properly designed and maintained. Citrix PVS requires dedicated set of streaming servers, database, and needs to be included in high-availability planning. Design of the PVS environment is mostly hypervisor agnostic and implementations are similar for different hypervisors.

Overview of Citrix Machine Creation Services (MCS)

Citrix Machine Creation Services is an orchestration component of Citrix Virtual Apps and Desktops that can provide single image management for shared or dedicated machines. MCS is passive component - in most deployments, MCS is involved only in the image build orchestration process (telling the hypervisor what and where to do) and not in daily operations and delivery of images. There are some exceptions to this rule, most notably for hypervisors that cannot automatically reset disks. From a design perspective, environments built using MCS inherits the behavior and characteristics of the hypervisor or cloud provider that is hosting workloads. Design of the MCS environment is therefore heavily influenced by a combination of hypervisor and storage used.

Provisioning Decision Factors

Each project and environment is unique and have different requirements and goals. For that reason, it is common that a good architect chooses different provisioning models for different projects and not exclusive prefer only one. It is common that different provisioning models are used even in a same environment - for example when providing a combination of dedicated and shared machines.

For this document, we are going to divide decision factors in two categories - factors where it is clear which provisioning model is preferred (or have to be used) and factors that are more open to interpretation and where personal preference / experience is playing much bigger role in decision making.

Explicit Decision Factors

Explicit decision factors cover scenarios where PVS or MCS is not only preferred, but often there is only one possible option.

Physical Machines

If your project involves provisioning to physical machines (typically classrooms or similar use cases), only Citrix PVS is supported (and functional). This model works well for standardized desktops such as those in lab and training environments, call centers, and “thin client” devices used to access virtual desktops.

Recommended model: PVS

Cloud Deployment

If you are planning to run virtual apps and desktops in one of the public infrastructures as a service (IaaS) environments like Microsoft Azure, Amazon Web Services or Google Cloud Platform, Citrix PVS is supported today in Microsoft Azure and Google Cloud Platform only. There are technical limitations that prevent some features of PVS from running in cloud environments - such as PXE and ISO boot of master and target VMs, 32-bit operating systems, and support of Gen1 VMs. If any of the above are required for your Citrix PVS implementation, it is still possible to run them in hosted virtualized environments.

Recommended model: MCS preferred

Persistent Desktops

When deploying virtual desktops in persistent mode, these are the most common approaches:

• Full clones with MCS
• Fast clones with MCS
• User layers
• Manual / ESD provisioning (SCCM)
• Hypervisor templates

While it is theoretically possible to provide dedicated desktops with Citrix PVS using private image mode, this approach is not recommended and does not provide any operational or performance benefits. Without the use of separate technology for persistent user layers, Citrix PVS is not a recommended model for persistent machines.

When using MCS for persistent machines, there are two possible approaches - using fast clones or full clones. While fast clones with MCS provide the benefit of small storage footprint and fast create and reset times (small delta disks), storage migration / backups / high-availability is more complicated in this deployment model. Since this is often a requirement for dedicated / persistent machines, full clones with MCS are the recommended approach to persistent desktops. You can read more about the difference between fast and full clones in article CTX224040.

Recommended model: MCS

License Entitlement

Citrix Provisioning is available under the Citrix Virtual Apps and Desktops entitlement, and is licensed per target device. Each provisioned target device checks out a single Citrix Cloud license, whether it is a data center running Windows Server, or a desktop running a Windows desktop version. For more information, read Products and License Models.

Recommended model: MCS

Variable Decision Factors

While the previous sets of decision factors are straightforward, this second set is much more flexible, open to interpretation and personal preference / experience with technology plays a much bigger role in decision making. Citrix gives you flexibility to choose the best solution for your needs and your decisions for the following factors might be different than our recommendations.

Technical Skills

This is an important factor to consider, especially if you are a Citrix partner and you are building a green field environment for a new customer. Consider the skills and capabilities of the team that is going to manage this environment - if a customer is new to Citrix technologies, is operating a static environment with minimum changes or if they have multiple roles and Citrix management is only a subset of their responsibilities, it might be a good idea to minimize the complexity of the environment and reduce the number of moving parts. In that case, MCS might be a better solution.

Recommended mode: MCS if technical skills are concern

Familiarity With Provisioning Model

Many of the Citrix customers and partners are familiar with Citrix PVS or MCS and have provisioned thousands of machines using this technology. This is the important factor when deciding which provisioning model to use - if you and the rest of your team is familiar with the procedures and technical aspects of one solution and that solution meets all your requirements and needs, use it for your new project. However, if you are a partner or build solution for third party, it is important to consider the learning curve and skill set of your customer.

Both PVS and MCS can support complex architectures and large environments - personal preference / experience is often the most important decision factor involved in choosing one.

Recommended mode: PVS or MCS, whichever model is more familiar

Complex Multi-Site Architecture

In certain type of environments, the ability to quickly replicate images across multiple storage repositories is critical. With PVS, this replication is simply and usually involves simple copy operation across different file shares. With enterprise MCS design, this requires replication of a master image itself together with automated image provisioning using CVAD SDK / PowerShell.

While it is possible to automate multi-site deployments with MCS, the PVS process is simpler and easier to use.

Recommended mode: PVS preferred

Requires Frequent Changes

One of the biggest advantages of Citrix PVS is the ability to almost instantly switch from one virtual disk (vDisk) to another and support for advanced versioning of virtual images. It is possible to achieve similar results with MCS using rolling catalogs and versioning on a master image level, however this process is simpler with PVS.

For environments that require frequent changes (multiple images changes every week), PVS might offer a more flexible, out of the box solution. There are more factors involved in this decision - for example how long does it take to update images using MCS in your environment and how many storage repository replications are required, but generally, you can expect PVS to be the more flexible image management solution.

Recommended mode: PVS preferred

Size of Environment

One factor that is not as important as many people believe is the scale of the target environment. Both PVS and MCS are enterprise ready solutions that can scale to tens of thousands of machines.

When scale is a potential decision factor is if you are designing a small and simplistic environment - unless there are some other factors involved (such as provisioning to physical machines), MCS is the preferred method for smaller environments (tens of machines).

Recommended mode: MCS for smaller environments, PVS/MCS for larger environments

Network Bottleneck

Citrix PVS is sensitive to a properly working network environment - whether it's proper routing/size of packets or stable network connection. Because it is using a hybrid UDP traffic, the impact of dropped packets can be substantial, as it requires a repeat of the whole sequence of packets. If network performance or stability is a concern, MCS (preferably not using NFS) might be a better approach.

Recommended mode: MCS if network stability is a concern

Requires Persistent Disk

There is a common requirement to keep some data persistent between reboots - for example event logs or configuration that needs to be restored after machine changes are deleted (for example unique machine identifiers used by anti-malware or software deployment tools to identify target machine).

With PVS and newer versions of MCS IO drivers (introduced in version 7.9), it is possible to store persistent data on write cache disk. This capability is possible with older versions of MCS, however it requires more scripting and automation skills. If you are not willing to automate this procedure or don't have the required skills, using an out-of-box functionality might be a better option. Be careful when redirecting data to a write cache disk - not properly planning and monitoring free capacity can lead to stability issues and needs to be carefully considered before implementation. Best candidates for redirections are smaller files with fixed size (e.g. log files with maximum size or small text files), it is not recommended to redirect large or unpredictable amount of data.

Recommended mode: PVS or new version of MCS IO preferred

Optimized Hypervisor and Storage

As mentioned at the beginning, PVS is mostly a hypervisor agnostic solution, while the performance, stability and flexibility of MCS has a strong dependency on the underlying hypervisor and storage.

If your underlying infrastructure is however optimized and designed to work properly with MCS, it is possible to achieve better results with MCS, as you are going to use hardware-acceleration instead of software acceleration.

The most notable candidate to mention here is the Nutanix implementation of Shadow Clones, which is optimized for MCS provisioning. Another good examples are hypervisors that are optimized for virtual desktop workloads - for example Citrix Hypervisor with support for In-memory Read Caching or IntelliCache.

Recommended mode: MCS if using hypervisor/storage optimized for MCS

Summary

In this article, we have discussed the most common decision factors when choosing a provisioning method for your Citrix Virtual Apps and Desktops environment. Both Citrix PVS and MCS are enterprise-ready solutions that offer great performance and flexibility.

Remote PC Access is an easy and effective way to allow users to access their office-based, physical Windows PC. Using any endpoint device, users can remain productive regardless of their location. However, organizations want to consider the following when implementing Remote PC Access.

Deployment Scenarios

There are multiple ways to connect a PC to a user, each applicable to different scenarios.

Office Workers

In many deployments, Remote PC Access gets deployed in an office worker scenario where one user is permanently assigned to one PC.

This is the most common deployment scenario. To implement this use case, the administrator can use the Remote PC Access machine catalog type.

Computing Labs

In certain situations, users need to share a set of computing resources, often found in computing labs at schools, colleges, and universities. Users are randomly assigned to an available physical PC.

This type of configuration uses an unmanaged, randomly assigned, single-session OS, which is configured as follows:

• In the Machine Catalog Setup Wizard, use Single-session OS

• Select Machines that are not power managed
• Select Another service or technology

• Select I want users to connect to a new (random) desktop each time they log on.

Because there are often more users than PCs in a computing lab, policies limiting session time are recommended.

Authentication

Users continue to authenticate to their office-based PC with their Active Directory credentials. However, as they are accessing over the Internet from outside the office premises, organizations typically require stronger levels of authentication than just user name and password.

Citrix Workspace supports different authentication options to be selected including Active Directory + Token and Azure Active Directory. Current Supported Authentication Options

If NetScaler Gateway is configured as the authentication option for Citrix Workspace, or if a customer chooses to use NetScaler Gateway + Citrix StoreFront as an alternative to Citrix Workspace, then the much wider selection of NetScaler Gateway authentication options becomes available.

Some of these options, like the time-based one-time password, require the user to initially register for a new token. As token registration requires the user to access their email to validate their identity, it may need to be completed before the user attempts to work remotely.

Session Security

Users can remotely access their work PC with an untrusted, personal device. Organizations can use integrated Citrix Virtual Apps and Desktops policies to protect against:

• Endpoint Risks: Key loggers secretly installed on the endpoint device can easily capture a user name and password. Anti-keylogging capabilities protect the organization from stolen credentials by obfuscating keystrokes.
• Inbound Risks: Untrusted endpoints can contain malware, spyware, and other dangerous content. Denying access to the endpoint device's drives prevents transmission of dangerous content to the corporate network.
• Outbound Risks: Organizations must maintain control over content. Allowing users to copy content to local, untrusted endpoint devices places extra risks on the organization. These capabilities can be denied by blocking access to the endpoint's drives, printers, clipboard, and anti-screen-capturing policies.

Infrastructure Sizing

Note: The following sizing recommendations are a good starting point, but each environment is unique, resulting in unique results. Monitor the infrastructure and size appropriately.

As users are accessing existing office PCs there is minimal extra infrastructure needed to support adding Remote PC Access, however it is important that the Control layer and Access layer infrastructure are sized and monitored correctly to ensure that they do not become a bottleneck.

Control Layer

The Virtual Delivery Agent (VDA) on each Office PC must register with Citrix Virtual Apps and Desktops. For on-premises deployments VDA registration happens directly with a Delivery Controller, for Citrix DaaS in Citrix Cloud this registration happens via a Citrix Cloud Connector.

Sizing of Delivery Controllers or Cloud Connectors for Remote PC Access workloads is similar to VDI workloads. Citrix Consulting recommends at least N+1 availability. Guidance for Cloud Connector scaling including conditions where Local Host Cache may be required is available here

• Scale and size considerations for Cloud Connectors
• Scale and size considerations for Local Host Cache

Access Layer

When a user establishes an HDX session to their office PC, the ICA traffic needs to be proxied to the VDA. ICA Proxy can be provided via NetScaler Gateway appliances or NetScaler Gateway Service.

When using an on-premises NetScaler for NetScaler Gateway, consult the data sheet for the particular model and refer to the SSL VPN/ICA proxy concurrent users line item as a starting point. If the ADC is handling other workloads, validate that current throughput and CPU usage are not approaching any upper limits.

Ensure that there is adequate available Internet bandwidth where the Gateway appliance is located to support the expected concurrent ICA sessions.

When using the Gateway Service from Citrix Cloud, the ICA traffic flows between the Resource Location (where the VDAs and Cloud Connectors are located) directly to the Gateway service. The traffic is either proxied by the Cloud Connectors (default) or can flow directly from the VDA, bypassing the Cloud Connectors if the conditions to use the rendezvous protocol can be met.

When Cloud Connectors are used to proxy ICA traffic to the Gateway Service then this can be a bottleneck and careful monitoring of CPU & memory on the Cloud Connector VMs is advised. For initial planning estimates a 4 vCPU Citrix Cloud Connector VM can handle a maximum of 1000 concurrent ICA Proxy sessions. Rendezvous protocol (when configured) enables the Virtual Delivery Agent installed on each physical PC to communicate directly with the Gateway Service instead of tunneling the session through the Cloud Connector.

When using Gateway Service, Citrix recommends using rendezvous protocol to mitigate the issue of Cloud Connectors being a bottleneck for ICA Proxy.

There are certain prerequisites to allow the rendezvous protocol to function, including:

• Citrix DaaS
• VDA version 1912 or higher
• An enabled HDX Policy
• DNS PTR Records for all VDAs
• Specific SSL Cipher Suite Order
• Direct (non-proxied) Internet connectivity from VDA to Gateway Service

Availability

If the office PC is not powered on with the VDA registered, the user’s session cannot be brokered. Citrix recommends putting in place processes to ensure the machines that users need to connect to are powered-on.

If available, modify the PC’s BIOS setting to automatically power on in the event of a power failure. Administrators can also configure an Active Directory Group Policy object to remove the “Shut Down” option from the Windows PC. This helps prevent the user from powering down the physical PC.

Remote PC Access also supports Wake on LAN operations to enable powering on Windows PCs that are currently powered off. The Wake on LAN feature is fully standalone and there is no need to have additional infrastructure for the Wake on LAN feature as of release 2009.

User Assignments

It is important that users are each brokered to their own office PC. Once the VDA has been installed and the catalog and delivery group defined, users are automatically assigned when they next logon locally to the PC. This is an effective method for assigning thousands of users.

By default, multiple users can be assigned to a desktop if they have all logged into the same physical PC, but this can be disabled via a registry edit on the Delivery Controllers.

The Citrix Virtual Apps and Desktops administrator can modify the assignments as needed within Citrix Studio or via PowerShell. To get started with using PowerShell for adding Remote PC Access VDAs to a Site and assigning users, Citrix Consulting produced a reference script which can be found on the Citrix GitHub page.

Virtual Delivery Agent

This section reviews key considerations for handling the Virtual Delivery Agent package.

Versions

Citrix Virtual Apps and Desktops administrators can use either the VDAWorkstationCoreSetup.exe package or the VDAWorkstationSetup.exe package with the /remotePC flag. The VDAWorkstationCoreSetup.exe package is smaller and only includes the core components required for Remote PC Access, but notably, in version 1912 and earlier, excludes the components required for content redirection (see the Microsoft Teams section for further guidance).

Windows 7 and 8.1

Though Windows 7 is no longer supported, many organizations still have legacy Windows 7 desktop machines. For deployment on Windows 7 and Windows 8.1, customers should use the XenDesktop 7.15 LTSR VDA.

Windows 10

For deployment on Windows 10, customers should use the Citrix Virtual Apps and Desktops 1912 LTSR VDA or a supported Current Release VDA. Citrix version compatibility for Windows 10 builds can be found in CTX224843.

Helpful Command-line Options

There are several VDA installer command-line options to consider when deploying Remote PC Access that can enable useful functionality.

/remotePC

Used with the full VDA package, VDAWorkstationSetup.exe, to install only the core components required for Remote PC Access.

/enable_hdx_ports

Opens ports in the Windows firewall required by the Cloud Connector and enabled features (except Windows Remote Assistance), if the Windows Firewall Service is detected, even if the firewall is not enabled.

/enable_hdx_udp_ports

Opens UDP ports in the Windows firewall that are required by HDX adaptive transport, if the Windows Firewall Service is detected, even if the firewall is not enabled.

To open the ports that the VDA uses to communicate with the Controller and enabled features, specify the /enable_hdx_ports option, in addition to the /enable_hdx_udp_ports option.

/enable_real_time_transport

Enables or disables use of UDP for audio packets (RealTime Audio Transport for audio). Enabling this feature can improve audio performance.

To open the ports that the VDA uses to communicate with the Controller and enabled features, specify the /enable_hdx_ports option, in addition to the /enable_real_time_transport option.

/includeadditional "Citrix User Profile Manager","Citrix User Profile Manager WMI Plug in"

In a Remote PC Access deployment, most implementations do not require profile management. However, Citrix User Profile Manager also captures performance metrics, which are useful for administrators to identify and fix performance-related issues. User Profile Manager does not have to be configured, it just needs to be deployed to capture metrics.

When installed, the Citrix User Profile Manager allows administrators to run reports on the user experience, session responsiveness, and insights into logon performance within Citrix Director and Citrix Analytics for Performance.

/logpath path

Log file location. The specified folder must exist as the installer does not create it. The default path is "%TEMP%\Citrix\XenDesktop Installer," but if the install is conducted via SCCM, then depending on the context, the log files can be in the system temp folder instead.

/optimize

Do NOT use this flag as it is intended mainly for MCS deployed machines.

Deployment

To deploy the Virtual Delivery Agent to thousands of physical PCs, automated processes are required.

Via Scripting

The install media for Citrix Virtual Apps and Desktops includes a deployment script (%InstallMedia%\Support\ADDeploy\InstallVDA.bat) that can be used via Active Directory Group Policy Objects.

The script can be used as a baseline for PowerShell scripts and Enterprise Software Deployments (ESD) tools. These approaches allow organizations to quickly deploy the agent to thousands of physical endpoints.

Via SCCM

If you are automating the VDA installation with an ESD tool such as SCCM or Altiris, creating separate packages for the prerequisites and VDA tends to work best. You can find more information about VDA deployment with ESD tools in the product documentation.

Microsoft Teams

If users access Microsoft Teams for voice and video calls, content redirection functionality is required to create a positive user experience.

For content redirection to be available when using VDA 1912 or older, it is required to deploy the VDA on the physical PCs using the single-session full VDA installer (standalone VDAWorkstationSetup.exe) with the /remotepc command line option.

For example: VDAWorkstationSetup.exe /quiet /remotepc /controllers “control.domain.com” /enable_hdx_ports /noresume /noreboot

If deploying VDA 2003 or newer, the single-session core VDA installer can be used instead (standalone VDAWorkstationCoreSetup.exe).

For example: VDAWorkstationCoreSetup.exe /quiet /controllers “control.domain.com” /enable_hdx_ports /noresume /noreboot

Common Network Ports

Similar to any other Citrix VDA, there are a handful of key network ports to be mindful of opening for the system to function. As a reminder, ICA traffic needs to reach the Remote PC Access from the NetScaler hosting the external NetScaler Gateway. A comprehensive list of ports can be found in Communication Ports Used by Citrix Technologies.

VDA Registration

Depending on the network topology, the subnet containing the Virtual Apps and Desktops Delivery Controllers might not allow communication to or from the physical PCs. To properly register with the Delivery Controller, the VDA on the PC must be able to communicate with the Delivery Controller in both directions using the following protocols:

• VDA to Controller: Kerberos
• Controller to VDA: Kerberos

If the VDA is unable to register with the controller, review the VDA registration article. If you are using Citrix Cloud, the Cloud Connectors take the place of the Delivery Controller.

Further Guidance

More design guidance including considerations and troubleshooting steps can be found in the Remote PC Access product documentation.

This article provides recommendations and guidance to estimate how many users or virtual machines (VMs) can be supported on a single physical host. This is commonly referred to as Citrix Virtual Apps and Desktops “single-server scalability” (SSS). In the context of Citrix Virtual Apps (Citrix Virtual Apps) or session virtualization, it is also commonly called “user density”. The idea is to find out how many users or VMs can be ran on a single piece of hardware running a major hypervisor such as a Citrix Hypervisor.

In this article, we cover several of the variables or factors that influence Citrix Virtual Apps and Desktops (Citrix Virtual Apps and Desktops) SSS. We provide recommendations and simple guidelines to quickly estimate SSS for a given environment. We conclude by providing a few sizing examples using real-world scenarios.

Warning!

This article contains guidance to estimate SSS. The guidance is high level and will not be specific to your unique situation or environment. The only way to truly understand Citrix Virtual Apps and Desktops SSS is to use a scalability or load testing tool such as Login VSI. Citrix recommends using this guidance and these simple rules to quickly estimate SSS. But Citrix recommends using Login VSI or the load testing tool of your choice to validate results, especially before purchasing hardware or making any financial decisions.

Scalability Factors

There are many factors, parameters, or variables that impact SSS. This is by no means an exhaustive list but the following are several of the major factors that impact SSS. While there are many more factors that influence performance and scalability such as antivirus and monitoring agents, graphics encoding, recent security vulnerabilities such as Spectre and L1 Terminal Fault, the factors detailed below typically have the greatest impact on Citrix Virtual Apps and Desktops SSS. Now let’s look at how to quickly estimate Citrix Virtual Apps and Desktops SSS using a simple formula.

Workload

One of the primary factors that impacts performance and scalability is the workload itself. Some workloads might involve task workers performing simple data entry tasks on a Citrix Virtual Apps server. Other workloads might involve developers compiling code or engineers manipulating 3D models via Citrix Virtual Desktops. These are commonly referred to as “light” and “heavy” workloads, respectively. And as you see later in this article, this type of workload variance can have a dramatic impact on SSS.

Hardware

The physical hardware that the workloads run on has a direct impact on SSS. It probably goes without saying but a newer server equipped with 28 cores and 1 TB of RAM will be able to support more users than an older piece of hardware with only 12 cores and 256 GB of RAM running a similar workload. And the CPU plays an especially important part in the Citrix Virtual Apps and Desktops SSS as you see later.

Activity Ratio

One of the often overlooked aspects of SSS is the activity ratio or how often users are working versus idle. Many scalability testing tools take a conservative approach and might use a fairly high activity ratio like 80% (which effectively means users are active or working 80% of the time and idle 20% of the time). However, we often see in the real world that activity ratios are closer to 40-60%. And this additional idle time can dramatically impact SSS and how much hardware one needs to purchase to support a Citrix Virtual Apps and Desktops environment.

CPU Over-Subscription Ratio

Most Citrix Virtual Apps and Desktops workloads are CPU-bound, meaning that the ultimate point of resource exhaustion is directly related to the number of physical cores available in the system. And since users are typically not active 100% of the time and we have tools such as Intel Hyper Threading available (not to mention hypervisor CPU schedulers are becoming increasingly efficient), we can often “over-commit” or over-subscribe resources such as CPU. And the ratio that one over-subscribes the CPU can also impact SSS (in a positive or negative manner if not done carefully). Citrix has found that a 2:1 CPU over-subscription ratio is often optimal in Citrix Virtual Apps SSS. For example, if a physical server is equipped with dual socket 20 core chips (that is “2 x 20”), there are 40 total physical cores available. And with Hyper Threading enabled, there would be 80 virtual or logical cores available. By applying a 2:1 ratio to the number of physical cores (40), we’d effectively recommend using 80 cores when sizing or estimating SSS. More examples are provided at the end of this article to illustrate this concept in further detail.

Microprocessor Architecture and Features

The underlying chip and memory architecture can also play an important role in SSS. And Intel has recently made significant improvements in the underlying microprocessor architecture design. On older chips, such as Broadwell and Haswell, Intel connected processors using a ring-based architecture. But as the number of cores increased, access latency increased and bandwidth per core diminished so Intel would mitigate this by splitting the chip into two halves and adding a second ring to reduce distances. And this invisible split was something that needed to be factored into Citrix Virtual Apps and Desktops SSS to provide optimal results. This has been referred to in the past as “NUMA” or Non-Uniform Memory Access. And the leading guidance was to ensure that you are sizing Citrix Virtual Apps VMs as large as possible but not crossing NUMA nodes, sub-NUMA clusters or rings at the same time. If you sized your Citrix Virtual Apps VMs too large and they effectively spanned NUMA nodes or rings, it can lead to NUMA “thrashing” by accessing non-local resources and this would yield reduced SSS. Fast-forward to today and Intel has moved from a ring-based architecture to a mesh-based architecture. And this new mesh architecture introduced in Skylake does not have the same limitations as before where we have to split chips, divide cores or add rings. And this changes the way that we size Citrix Virtual Apps servers in particular. It is important to understand the specific chip that is being used in the hardware you purchase and how the underlying microprocessor architecture is designed and constructed

The Magic Multipliers

If you’d like to quickly gauge or estimate Citrix Virtual Apps and Desktops SSS, this guidance is effective. It’s as easy as this – take the number of physical cores in a server, multiply it by 5 or 10, and the result will be your SSS. If you’re looking for the number of Citrix Virtual Desktops VMs you can support, then you would use the “magic multiplier” of 5. If you’re trying to understand how many Citrix Virtual Apps users can run on a single piece of hardware, you’d use 10. Let’s look at a few real-world examples to see how this is applied in practice.

Example 1: Citrix Virtual Desktops

Let’s assume you’re running Windows 10 with standard Office applications and a few custom applications. You’ve identified that a 2 vCPU / 4 GB RAM VM specification will work best given the workload/image. You’re considering buying Cisco blades with 36 physical cores (2x18) and 768 GB of RAM. And you would like to understand what kind of density you can expect. Let’s use the Rule of 5 for Citrix Virtual Desktops:

5 x 36 = 180 VMs per host

Note: Citrix specialized VDI and RDSH-based workloads are CPU bound 99.9% of the time. CPU has become the scalability bottleneck as opposed to memory, disk storage, or network storage. These multipliers omit other areas aside from CPU because CPU has become the main factor. Although hyper-threading, clock speeds, and virtual cores are all important, nothing is more important than the number of physical cores on a server. When using the rule of 5 and 10, it is best to ignore all the other numbers at first to do the initial sizing to avoid confusion.

Example 2: Citrix Virtual Apps (older hardware)

Let’s assume you’re running an application such as SAP on Windows Server via Citrix Virtual Apps. You’re repurposing some older HP blades with 24 physical cores (2x12) and 256 GB of RAM. You’ve researched on Intel’s website that the underlying chip employs a ring buffer architecture and each socket is effectively split into 2 NUMA nodes with 6 cores each. Therefore, a 6 vCPU / 24 GB RAM VM specification seems optimal to maximize linear scalability and minimize NUMA thrashing. Using a 2:1 CPU over-subscription ratio, you use all 48 logical cores and deploy 8 XenApp servers on each host (48 / 6 = 8). Using the Rule of 10 for Citrix Virtual Apps:

10 x 24 = 240 users per host

Example 3: Citrix Virtual Apps (newer hardware)

Let’s assume you’re running a popular healthcare application on Windows Server 2022 via Citrix Virtual Apps. You’re considering purchasing Dell blades with 32 physical cores (2x16) and 256 GB of RAM. You’ve researched on Intel’s website that the underlying chip employs a mesh architecture and there is a business directive to decrease your VM footprint as much as possible. You decide on a 16 vCPU / 48 GB RAM VM specification. To use a 2:1 CPU over-subscription ratio, you use all 64 logical cores and deploy 4 XenApp servers on each host (64 / 16 = 4). Using the Rule of 10 for Citrix Virtual Apps:

10 x 32 = 320 users per host

As mentioned previously, we realize there are many more variables or parameters that influence scalability versus just the number of physical cores in a server. There can be certain situations where the Citrix Virtual Apps and Desktops workload is not CPU-bound so extra care is required when sizing. In addition, other factors we haven’t discussed such as CPU clock speed and logon storms also matter and further complicate sizing exercises. But we have found through years of field experience and hundreds of deployments that nothing matters as much as the number of physical cores. If you’d like to understand your exact SSS for your particular workload and unique environment, Citrix strongly recommends using a tool such as Login VSI to properly test and validate results.

References

Intel Xeon Processor Scalable Family Technical Overview

Antivirus Best Practices

Login VSI Load Testing

To provide the best possible advice to our customers, we decided to determine the answer to four key questions that impact Citrix architecture and design decisions:

1. What is the most efficient instance series for hosting DaaS
2. What is the most cost-effective instance type in the most efficient family
3. What impact does the Machine Creation Services I/O (MCSIO) cache have?
4. How does Windows 10 Multisession scalability compare to Windows Server OS?

A more detailed paper is available from Citrix that goes into the specifics of the testing methodology and the performance results captured during the evaluation.
This paper focuses on the high-level results and provides guidance on how to design an efficient Citrix implementation in the Microsoft Azure cloud.

To determine the performance, we used LoginVSI 4.1.32.1, which creates simulated sessions against a single Citrix server. The two workloads that we used for testing are described as follows:

• Task Worker Workload – includes segments with Microsoft Office apps, Internet Explorer, Adobe Acrobat, and PDF Writer. The Task Worker workload does not place a high demand on the environment and represents users that do not heavily use the system.
• Knowledge Worker Workload – includes segments with Microsoft Office apps, Adobe Acrobat and other specific apps including viewing of 360p movies. The Knowledge Worker workload places a higher demand on the environment, including more use of the available memory, and represents users that use the system more heavily.

The number of users successfully completing the multi-session test provides a key performance indicator under real-world conditions. This value - referred to as the VSImax session count - is used for the comparative analysis. The Login VSI workloads calculate the VSImax session count by observing the response time of a single user on the system.
VSImax is reached when the response time has diminished significantly below the expected threshold derived from the baseline value taken with only a single user on the system.

To provide conservative numbers which can be replicated consistently without specialized knowledge, all results reflect test execution using default Citrix policies and unoptimized default settings for Windows and Office products.
Both performance and density can be improved by applying Citrix optimization tools such as the Citrix WEM and Citrix Optimizer.

What is the most efficient instance series?

To find the most efficient instance series, we needed to test the different instance series without changing any other variables in the mix. The base image was Windows Server 2016 with the 1903.1 version of the Citrix VDA and a standard hard disk drive (HDD) 128 GB disk for the system C: drive. We selected the 8-core instance types for two primary reasons:

1) They represent the workhorse of Azure instance types for hosted sessions and are generally the most popular size deployed 2) They provide a good balance of CPU/RAM and minimal OS impact as opposed to a smaller 2-core system.

The following graph shows the instance family results along with the average cost per user per hour based on pay-as-you-go pricing for the Azure US-West-2 region where the tests were performed.

Analysis

Most of these instance types use the same processor, Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz. The primary difference being the amount of memory available to the virtual machine. More information about these different series can be found on Microsoft's website.

Generally speaking, the 8-core instances have fairly similar performance, especially when you consider the physical cores (D13_v2, D4_v2, L8s) compared to hyper-threaded cores (F8s_v2, D8_v3, E8_v3). However, when the instance's hourly cost is considered, the D13_v2, and F8s_v2 instances provide the more efficient use. The E_v3 and LS_v1 series are less cost-efficient because Microsoft charges a higher premium for Memory optimized and Storage optimized instances. In situations where your user's applications are extremely memory or storage intensive, these instances often provide a good return on investment.

Recommendations

If your typical user's applications are CPU-intensive and do not require significant memory to run, the most cost-efficient performance is the F series. Select the F series when you need excellent CPU response times and do not require a significant amount of memory. If your user's applications consume a fair amount of memory, use one of the D instance types depending on how much extra memory per core is required for your user's environment.

What is the most cost-effective instance type in the most efficient family?

When we completed the broad test across families, we expected a single series to be a clear leader. However the results ended up convincing us that the two best instance families for extra testing were the D series and the F series when tested with standard HDD storage. The next step was to test the specific sizes ranging between 2 and 16 vCPUs within the D_v2 and FS_v2 families. The results of these tests:

Analysis

The performance graph shows the results of those tests with the highest densities of 74 and 63 user sessions, for task worker and knowledge worker respectively, being obtained on the D14_v2 instance type (16 cores, 112 GB RAM). Since pricing varies between the instance types, a higher density does not necessarily imply a lower cost per user.

The pricing model for Azure instances varies according to the region, the instance type, and the resources provided. The graphs also includes the cost efficiency of each instance type based on VDA user densities achieved in the single-server testing. The costs reflect U.S. West 2 Pay-As-You-Go pricing for standard VM instances as of September 2019 and includes the cost of Microsoft Windows licensing.

As shown in the graph, the D13_v2 instance type shows the lowest cost per hour per user of $0.018 for a task worker, with the F16s_v2 and F8s_v2 coming in second with a cost of $0.019. As for knowledge worker, both the F16s_v2 and F8s_v2 instance types share the best hourly cost of $0.025, followed closely by the D13_v2 instance type at $0.026.

Recommendations

In the testing, the density results showed a clear benefit from the faster processors available with the FS v2-series instances when under the heavier workload of the knowledge worker. However, the FS v2 memory-to-core ratio is lower than the D v2-series ratios and we recommend using the FS v2-series instances only when the memory consumption for the workload is low. If users run applications that are memory-intensive, the D_v2-series is the best choice.

When the cost per user is similar, such as the case with F8S_v2 and F16s_v2, select the smaller instance sizes, when either of the following conditions exist:

• Need for resiliency: you want to impact fewer users during maintenance windows
• Need for efficient power management: you want to power off unused machines quickly

Select the larger instance sizes when either of these conditions exist:

• Need for reduced management: you want to manage fewer machines in the environment
• Need reduced API calls: you need less API calls to Azure infrastructure for operations

What impact does the Machine Creation Services I/O cache have?

The instance types used for testing were configured with standard storage rather than more expensive SSD storage for the system drive on the virtual machine. Because the instance types with SSD storage have smaller ephemeral disks where the page file is stored, even though the disks were faster, the scalability was lower because the instance did not have enough swapfile space available to support the need virtual memory under a higher load.

At the disk sizes that we are using, the HDD and SSD disks have similar IOPS performance (500). While the SSD disks have a more consistent performance, the additional cost is not always justified.

We then decided to consider the Machine Creation Services I/O (MCSIO) cache, as a way to achieve SSD-like performance with the larger standard disks. The tests were completed using the Citrix VDA version 1903.1 and Windows Server 2016 on a D5_v2 (16 vCPU, 56 GB of RAM) instance type. The chart shows that the increase in user density gained by enabling the MCSIO cache with the Knowledge worker load.

Analysis

When the operating system disk has no MCSIO cache enabled, the VSImax User score was 61 on a 128 GB HDD, 74 on a 64 GB SSD disk and 75 on a 128 GB SSD disk. Enabling the MCSIO cache on a standard HDD disk actually provided better performance than an SSD, with a 4 GB cache enabled on the 64 GB HDD the score increased to 76 and with a 2 GB cache the score increased slightly more to 77. The loss of the additional user between the 4 GB and 2 GB cache sizes is attributed to the additional RAM being used for the cache and not available for user workload.

While MCSIO contributes to a lower cost per user per hour, that number is not significant on its own. The real impact of MCSIO can be found out when looking at the end user experience.
The graph shows the average response time drop when using MCSIO:

Recommendations

If user experience is a driving factor when considering performance, we recommend enabling the MCSIO cache. When enabled, the recommendation is to use a standard disk with the 2GB cache since it provides the best improvement without affecting user density. However, the MCSIO cache must not be enabled on virtual machines that are memory constrained, such as the F or FS series instance types that are optimized for Compute but have low memory-to-cpu core ratios.

How does Windows 10 Multisession scalability compare to Windows Server OS?

With the release of both the Windows Server 2019 and Windows 10 Multi-session operating systems, we thought it would be best to provide some guidance about how the client operating system would impact the scalability. Both Windows Server 2019 and Windows 10 Multi-session operating systems require the newer Citrix VDA version 1906.1. Windows 10 Multisession is available with Azure Virtual Desktop (AVD) Entitlement and grants the tenant the base price of the VM (Linux pricing). That entitlement also extends the VM pricing to Windows Server 2016 and Windows Server 2019.

The graph shows the density changes when compared against the same test runs with Windows Server 2016 using the Citrix VDA version 1906.1 on the same D4_v2 (8 vCPU, 28 GB of RAM) instance. The prices are using the Linux VM pricing in line with the AVD Entitlement required:

Analysis

When compared to the Windows Server 2016 results, the Windows Server 2019 provided a slightly lower user density for both the knowledge worker and the task worker, with a 7% decrease for task workers and a 12% decrease for knowledge workers.

Comparing Windows Server 2019 to Windows 10 Multisession workload resulted in 19% less task workers and 32% fewer Knowledge workers. This performance decrease is expected because Windows 10 is a full client version and is not optimized for server-based computing like Windows Server 2016 and Windows Server 2019.

One cost advantage of using Windows 10 Multisession is that it does not require RDS CAL licenses for clients to connect to the virtual machine. This cost advantage is not included in the calculations since it is a Microsoft licensing cost in addition to the Azure cost per hour.

Recommendations

When planning to upgrade from Windows Server 2016 to Windows Server 2019, expect to increase the number of virtual machines by approximately 10%. If you are planning on using Windows 10 Multisession for hosting applications that require the Windows client for compatibility, keep in mind the density will be lower, resulting in approximately 30% extra costs over the server operating systems. The Windows 10 Multisession though does allow users to access the Windows Store, something that the server operating systems do not have available.

Conclusion

The Azure instance type that you select to deploy Citrix virtual application workloads is a critical element that determines the user density and scalability, and in turn the cost-per-user for an Azure delivery model.
As shown, the different instance types in Azure have advantages for specific workloads, such as high computational requirements or extra memory.
Usually, a D13_v2 instance with standard HDD disks and a 2GB MCSIO cache enabled provides the best user performance at the lowest cost. Consider the Windows 10 Multisession operating system when you need Windows Store, application compatibility, or a true Windows client experience.

The Citrix on Azure results presented here represent only guidelines in configuring your Azure solution. If you do not have data about your specific user workloads, the numbers we provided here serve as your design estimates. Before making final sizing and deployment decisions, we strongly suggest that you run proof-of-concept tests on different Azure instance types using your own workloads, then use that data for your final designs.

Learn more

For more information about deploying Citrix Virtual Apps workloads on Microsoft Azure Cloud Services, see the Citrix and Microsoft partner website at https://www.citrix.com/global-partners/microsoft/resources.html

Azure Active Directory (Azure AD): This service is Azure’s authentication cloud-based identity and mobile device management service that provides user authentication. Azure AD does not support device authentication, domain joins or group policy objects (GPOs). However, Azure AD can be paired with Azure Active Director Domain Services (AAD DS) to provide the minimum level of support needed for Citrix in Azure.

Azure Active Directory Domain Services (Azure AD DS): This service is a managed domain service hosted in the cloud. This service supports GPOs, Kerberos authentication, and domain joins. The difference between Azure AD DS and AD DS is that the AD domain controllers are managed by Microsoft rather than you. Azure AD DS integrates directly with Azure AD and is a great option for cloud-based Citrix deployments.

Here are the questions that you need to answer regarding Active Directory infrastructure:

Should I continue to use only my on-premises Active Directory infrastructure?

• Easy to deploy by just installing Cloud Connectors and joining them to the on-premises domain

• Citrix Cloud can be configured to use only the on-premises AD Domain

• Using or synchronizing to Azure AD is not required, leaving Azure AD to be solely used as identity management for Azure administration

• To prevent latency introduced by domain authentication, Citrix recommends placing domain controllers near the Citrix Virtual Delivery Agent (VDA) hosts and the Cloud Connectors

• Approval from information security (infosec) may be required to place domain controllers in Azure

• Not recommended for deployments that use Microsoft 365 and that have users logging into Azure AD for that service

Should I extend on-premises Active Directory using Hybrid Mode with Azure AD Connect?

• Microsoft recommends this design when you have an existing on-premises AD infrastructure and need either of these features:

  • schema extensions
  • account-based Kerberos constrained delegation

• At least one Active Directory domain controller should always be available for the Cloud Connectors and VDAs. This design prevents any authentication bottlenecks or latency during the group policy processing, domain joins, and authentication events

• Citrix recommends this model when you have Citrix workloads that are still on-premises

• Citrix recommends this model if Citrix Cloud services such as Endpoint Management will be used

• Place at least two domain controllers in Azure and use Azure AD Connect to synchronize AAD with AD DS over ExpressRoute or VPN

• Using Windows 10 under a Hybrid Use Benefit license requires computer accounts and user accounts be in the same Azure Active Directory

• Approval from information security (infosec) may be required to place domain controllers in Azure

• If using smart cards, Kerberos must be enabled on a domain controller

Should I establish a new Azure Active Directory(AAD)?

• Microsoft recommends this model when you are using a cloud-only deployment or when you do not have an existing on-premises AD infrastructure

• Citrix recommends this design when all your Citrix workloads are in Azure and using one of these services:

  • Citrix DaaS

• Plan to use Azure AD Connect to synchronize Azure AD with Azure AD DS and enable password hash synchronization

• If using smart cards, Kerberos must be enabled for Azure AD DS

• Using Windows 10 under a Hybrid Use Benefit license requires computer accounts and user accounts be in the same Azure Active Directory

• Azure AD DS does not support schema extensions, one-way trusts or account-based constrained delegation for Kerberos

• Azure AD DS does not support Domain or Enterprise Admin privileges

Should I use Azure AD as the Citrix Cloud Identity provider?

• Citrix Cloud supports both Azure AD and AD DS for authentication

• When using Azure AD as the Citrix Cloud Identity provider you maintain control of password policies and can easily disable accounts

• Using Azure AD provides multifactor authentication (MFA) to increase the security posture for Citrix Cloud

• When Azure AD is branded, Citrix Cloud has a branded sign-in page

• Azure AD extends Citrix Cloud to support federated identity options such as Okta, Ping, or ADFS

• Requires Global Admin role for consent to allow Citrix Cloud to connect to Azure AD. If access to this role is not available, consider using other identity providers such as on-premises AD or the default Citrix identity provider.

Should I enable Multifactor Authentication (MFA) on the Azure Active Directory Accounts?

• Multifactor authentication is always recommended for any resource that is accessed over the internet. Multifactor authentication decreases the available attack vectors and increases the security posture of the system.

• Azure AD makes enabling MFA simple and integrates easily with the Citrix Cloud identity provider

• If not using Azure AD for MFA, consider other identity providers such as Okta to provide this additional security

Links to Other Resources

Azure Active Directory and Citrix XenApp and XenDesktop

Azure Active Directory Hybrid Identity Design Considerations

Citrix Cloud identity and access management

Citrix TIPs: Citrix Tips on Azure Enterprise-Scale Landing Zones - Part 2

Compare self-managed Active Directory Domain Services, Azure Active Directory, and managed Azure Active Directory Domain Services

What is hybrid identity with Azure Active Directory?

XenApp and XenDesktop Services Support Azure AD Domain Services

Profile Management solutions are designed to make a user’s local profile portable so that it can be accessed from any session or device. Both Citrix User Profile Management (UPM) and Microsoft FSLogix improve on the traditional roaming profile model used in data centers. Both solutions improve the response time for users and store the user profile using Azure Files. The benefits of Citrix Profile Management are outlined later on.

Citrix User Profile Management

• Integrates with the following products:
  • Citrix Virtual Apps and Desktops (Citrix DaaS)
  • Citrix Workspace Environment Management (WEM) service
  • Azure Files
• Virtualizes user profiles so the user settings can be applied to the user desktop or application
• Streams profile data so that it is not downloaded until needed
• Offers large file handling which allows large files to be redirected individually providing a native (local) file experience
• Supports profile exclusions to reduce bloat
• Supports multiple concurrent file access for multi-session users
• Supports profile containerization
• Improves logon speed
• Implements containerization through redirection of users profiles to a virtual hard disk
• Supports profile containers and Microsoft Office containers
• Maintains user data for non-persistent environments, such as Citrix session or Azure Virtual Desktop
• Reduces logon times by mounting a VHD instead of copying user profile data across the network
• Supports profile exclusions to reduce bloat
• Provides a native (local) profile experience for users
• Integrates with OneDrive and Azure Files

Profile Design Considerations

Some applications are not designed with roaming users in mind and rely on local file caches and indexes that do not roam between sessions. Microsoft Outlook is one of the more popular applications with this behavior. Both Citrix User Profile Manager and Microsoft FSLogix can provide an improved user experience with these types of applications. Other design considerations include:

• Users accessing their data from multiple sessions simultaneously require solutions that support that level of file access.

• Keep user data as close as possible to the user’s session. When users are accessing from both on-premises and cloud-based sessions, choose the cloud when possible.

• With both Profile Management solutions, permissions for profile stores must be configured manually. Support for multi-session simultaneous access requires extra configuration.

  • Always combine Profile Management with folder redirection to reduce the amount of data copied locally
  • Always configure profile folder exclusions to reduce bloat, since they are not configured by default
  • Always enable large file handing for Citrix User Profile Management so that large files, such as PSTs or OSTs, are not copied down

• Antivirus exclusions are required for both FSLogix and Citrix User Profile Management profile solutions because they implement system-level drivers for redirection.

• When using Microsoft FSLogix (or Citrix Profile Containers), you must exclude VHD(X) containers from AV scanning when hosted internally on traditional file shares.

• Azure Files sync can replicate containers quickly and easily for staged deployments.

User Data Challenges

One of the biggest challenges with migrating to Azure includes how to manage user profiles and access to personal and department data. Users require their data to perform their jobs and they need to access it from any device they are using. This section provides guidance for the challenges associated with user data and considerations that influence the design.

The goals for user data are:

• provide access to the data securely
• provide access to it always from any location
• provide access with the lowest latency possible

Meeting these goals is a challenge with a hybrid environment where some Citrix workloads are in the cloud and some remain on-premises. The user’s data cannot be in both places at once without creating data collision opportunities. Selecting a single location introduces security, latency, or access concerns. This dilemma is true for both the user data and the shared department data.

Windows Profiles

Windows still relies on the concept of profiles to store user data. The loading of those profiles can significantly impact a user’s logon experience, especially when the user’s desktop contains a large amount of data. The logon experience is made worse when the user's session has a significant amount of latency between the profile store and the session host. Several technologies, such as Citrix Profile Manager and Microsoft FSLogix, have been developed to help remove these pain points. The following information helps you select which technology is best for your users.

When should I use the traditional file server technologies for hosting user data?

The traditional file server technologies are file sharing solutions that are used in data centers today. Often these technologies use Distributed Files System Replication (DFS-R) or Distributed File System - Namespaces (DFS-N) to make file shares highly available across multiple locations. Accessing these file shares from an on-premises location typically introduces high latency because of routing and protocol latency. The different file server technologies along with their benefits and drawbacks are provided below.

• Standalone File Servers: Windows Server configured as file servers

  • Requires management and maintenance
  • Has potential cost advantages when hosted in the cloud compared to other server-based technologies
  • Compatible with familiar backup/restore products
  • The standalone server is a single point of failure since it has downtime during updates that force the server to reboot

• Storage Replicas: Windows Server technology that enables synchronous replication of volumes between servers or clusters

  • Requires management and maintenance
  • Supports block-level replication (synchronous or asynchronous)
  • Supports the SMB 3.0 protocol which includes security enhancements such as encryption
  • Has only minimal downtime during manual failover between replicas

• Storage Spaces: Windows Server technology that allows drive pooling in a RAID-type configuration and can be clustered across multiple server nodes for high-availability

  • Requires management and maintenance
  • Supports SMB 3.1 which includes transparent failover mechanisms
  • Has a multi-node topology that can scale up/out as necessary,
  • Has a transparent failover,
  • Uses 3 times more disk space than a traditional file server
  • Not always supported by third-party backup/restore products

• Traditional file servers work best in a data center where the Citrix workloads have direct access to the file share.

• Traditional file servers support the installation of governance and security software, such as:

  • Data loss prevention (DLP)
  • Antivirus (AV)
  • Backup
  • Encryption software
  • Host-based Intrusion Prevention System (HIPS)
  • Host-based Security System (HBSS)

• Some traditional file server deployment configurations result in lower overall costs compared to the PaaS shares.

• Use traditional file servers when your organization needs complete control over the data. With a traditional file server, governance and legal ownership are easy to maintain and data classification is easier to implement.

• Traditional file server technologies are used when applications need nearby for compute or when an extensive number of read/writes are expected on the data.

• Traditional file server technologies represent less durable data storage when compared to cloud-based alternatives such as Azure Files.

• Look at the scalability path for meeting demand, scale out or scale up with current hardware.

When should the PaaS file shares be used?

These cloud-based file services were built specifically as a service instead of an application and optimized to operate over the internet.

• Azure Files: File shares as a service backed by Azure storage

  • Platform as a Service (PaaS)
  • Supports SMB 3.1/NFS 4.1 protocols
  • No server maintenance, Microsoft handles all maintenance
  • Mountable in Azure VM and Windows Server 2012 and later
  • Can be mounted from on-premises hosts
  • Supports different performance tiers: hot, cold, and high performance
  • Supports NTFS permissions and ACLs
  • Costs vary based on storage performance requirements

• Azure NetApp Files: NetApp Filers as a service backed by Azure Files

  • Platform as a Service (PaaS) using NetApp Filers
  • No server maintenance, Microsoft handles all maintenance
  • Mountable in Azure VM and Windows Server 2012 and later
  • Can be mounted from on-premises hosts
  • Includes Extreme Performance compared to Azure Files options
  • Supports NTFS permissions and ACLs
  • Increased cost compared to Azure Files

• PaaS file shares have unlimited highly durable data storage.

• PaaS file shares have limits on performance, throughput, and protocol support.

• PaaS file shares are more complex to set up with Active Directory NTFS permissions.

• PaaS file shares can be mounted from most operating systems.

• PaaS file shares integrate with other cloud-bases services such as logging and metrics.

• PaaS file shares are best when the user workloads are also in Azure. Using PaaS file shares reduces the egress data charges and saves on monthly charges.

• PaaS file shares work best for sharing data internally across departments when user workloads are in Azure.

• PaaS file shares do not work as well for sharing files externally because permissions are tied to the Azure AD users.

• When using PaaS file shares, verify that users accessing their data shares from on-premises are receiving acceptable response times.

• Select the lowest performance tier that meets the user's expectations.

• Governance and legal requirements are imposed based on the region hosting the data.

• Cloud-based file services do not support the installation of third-party software such as DLP, AVS, or encryption software.

• Backups can be easily configured using Azure Backup.

When should I use Azure NetApp Files vs Azure Files?

This decision is based on what your users consider acceptable. Generally speaking, when you have fewer than 100 users accessing the file share simultaneously, an Azure Files, Transactional performance level works best. With workloads of between 100 and 2000 users, depending on the frequency of the file updates, consider Azure Files Premium performance level. With workloads over 2000 users, consider using the Azure NetApp Files. To reduce the traffic on the file share, consider using Citrix User Profile Management with profile streaming and large file handling enabled. You can also reduce traffic on the file share by using Microsoft FSLogix containers or Citrix Profile Management Containers.

When should the File Sharing Collaboration solutions be used?

File Sharing and collaboration services are designed to make shared files accessible using the HTTPS protocol over the internet. These services allow not only individuals to store and retrieve files from the service, but also support collaboration between departments and even outside entities. They have security built in and provide a single point of access. These solutions are best for storing data securely that must be shared both internally and externally. Though they have been adapted to work as personal storage locations for users, they don’t always work well for storing user profiles.

• ShareFile: Secure file-sharing cloud-based service hosted by ShareFile.

  • File sharing repository accessible over HTTPS/CIFS
  • Light management required around user security
  • Limited maintenance of local StorageZone controllers. ShareFile maintains the rest of the infrastructure
  • Data owners can easily grant permissions to internal and external entities
  • Integrates with Active Directory
  • ShareFile-managed StorageZones are protected by durable cloud storage (in Azure)
  • Customer-managed StorageZones allows use of local customer data centers
  • ShareFile handles all the backups, antivirus, and indexing operations
  • All files stored encrypted with AES-256
  • Integrates with SharePoint and OneDrive
  • Supports mobile access to network shares
  • The ShareFile Sync client can synchronize local user data with ShareFile storage
  • Includes document management, workflow management, content collaboration, and e-signing capabilities
  • Costs vary depending on the functionality selected

• SharePoint in Microsoft 365: Cloud-based SharePoint service hosted by Microsoft

  • Limited management of user security and access
  • Microsoft maintains all servers
  • File-sharing repository accessible over HTTPS/CIFS
  • Light-weight version of SharePoint server
  • Data owners can easily grant permissions to internal and external entities
  • Integrates with Active Directory
  • Supports mobile access to network shares
  • Includes document management, workflow management, and content collaboration capabilities
  • Costs vary depending on the functionality selected
  • Integrates with OneDrive to synchronize content

• OneDrive: OneDrive provides synchronization of user data between a local Windows workstation and a back end data storage location

  • A local agent client installed and configured to synchronize user data with SharePoint or ShareFile
  • Can be configured to automatically back up the Documents, Desktop, and Pictures folders to OneDrive in Microsoft 365
  • Included with Microsoft 365 licenses

Cloud Storage Design Considerations

Cloud storage technologies have changed the landscape of personal data storage expectations. Fortunately, most users now treat the extra latency as an acceptable tradeoff for being able to access their data securely from anywhere. Other design considerations when using these collaborative file shares include:

• Cloud-based file sharing and collaboration services have unlimited data storage.

• With collaborative file sharing services, highly durable data storage is used and backups are included in the cost of the service.

• Cloud-based file sharing and collaboration services do not support the installation of third-party data protection software. You are expecting the vendor to provide the protection against data loss, viruses, and loss of confidentiality.

  • These technologies are preferred when a need exists to share the files externally, such as with other businesses or third parties.
  • Using the ShareFile Sync agent with ShareFile or the OneDrive with SharePoint provides an excellent user data backup solution for their local device files.
  • Collaborative file shares are excellent for remote users that keep many documents locally on their assigned device.
  • When collaborative file shares are used from non-persistent sessions, use the Session Linger setting so data can be synchronized before the session terminates.

• When using SharePoint

  • Keep the top-level parent portals to a minimum to improve security, usability, navigation, and adoption.
  • Avoid using deep hierarchies with unique permissions to improve performance.
  • Do not bury content or keep stale content as it impacts usability and deters users from using the site.
  • Use standard groups first (Members, Visitors, Owners) followed by AD Groups or SharePoint groups next, and direct user access last.
  • Take advantage of permission inheritance.

• OneDrive clients interoperate with GPOs and support folder redirection.

• OneDrive clients integrate with Profile Management solutions.

What if my users are accessing their data from both on-premises and in the Azure cloud?

• Collaborative file services work for sharing data internally and externally and also function as user data file repositories

• PaaS file services can be integrated directly with Windows since they can be mounted and accessed like internal file shares

• Augment PaaS file services with Profile Management solutions that virtualize the file system. This approach reduces the amount of data on the wire and reduces the monthly charges from outbound Azure Files data.

• Using PaaS file services prepares the way for complete cloud adoption while providing an improved experience for users accessing their data out of the cloud

What data methods work best together?

• Different methods are acceptable for different user groups based on their data access requirements

• To reduce costs, avoid combinations that store the same data in multiple locations, for instance do not use Citrix ShareFileSync and Citrix User Profile Manager with Azure Files

• For cloud-based Citrix workloads, combining a Profile Management solution with a cloud-based file service has proven to be a good combination

  • Citrix User Profile Management with large file handling enabled on an Azure Files share
  • Using Microsoft FSLogix with Azure Files
  • Citrix ShareFileSync backed by Citrix ShareFile with Microsoft FSLogix

• For on-premises Citrix workloads, combine a Profile Management solution with the traditional file server technologies

  • Citrix User Profile Management with large file handling enabled on a traditional file server share
  • Using Microsoft FSLogix with a traditional file server share

Links to Other Resources

Compare self-managed Active Directory Domain Services, Azure Active Directory, and managed Azure Active Directory Domain Services

Deployment Guide: Deploying Azure Files for Citrix Profile Management and Citrix User personalization layers

Design Decision: Citrix Profile Management with Azure Files

Introduction to SharePoint in Microsoft 365

Introduction to SharePoint information architecture

Provisioning ShareFile on Microsoft Azure Storage

Redirect and move Windows known folders to OneDrive

Storage Replica overview

Storage Spaces overview

Use an Azure file share with Windows

What is FSLogix?

Selecting the best VDI model starts with properly defining user groups and then aligning the requirements with the capabilities of the VDI models. Although there are multiple approaches towards defining user groups, it is often easiest to align user groups with departments. Typically most users within the same department or organizational unit consumes the same set of applications.

User Segmentation

Depending on the size of the department, there might be a subset of users with unique requirements. Each defined user group is evaluated against the following criteria to determine if the departmental user group needs to be further divided into more specialized user groups.

• Primary data center – Each user has a primary data center or cloud resource location assigned that hosts their virtual desktop, data, and application servers. Identify the data center that the user is assigned to rather than the data center they are currently using. Users are grouped based on their primary data center so that a unique design can be created for each one.
• Personalization – Personalization requirements are used to help determine the appropriate VDI model for each user group. For example, if a user group requires complete personalization, a personal desktop is recommended as the optimal solution. There are three classifications available:

• Security – Security requirements are used to help determine the appropriate desktop and policy (or policies) for each user group. For example, if a user group requires high security, a hosted pooled desktop or a local VM desktop is recommended as the optimal solution. There are three classifications available:

• Mobility – Mobility requirements are used to help determine the appropriate desktop model for each user group. For example, if a user group faces intermittent network connectivity, then any VDI model requiring an active network connection is not applicable. There are three classifications available:

• Desktop Loss Criticality – Desktop loss criticality is used to determine the level of high availability, load balancing, and fault tolerance measures required. For example, if there is a high risk to the business if the user’s resource is not available, the user is not allocated a local desktop. If that local desktop fails, the user is not able to access their resources. There are three classifications available:

• Workload – Types and number of applications accessed by the user impacts overall density and the appropriate VDI model. Users requiring high-quality graphics must utilize a local desktop implementation or a professional graphics desktop. There are three classifications available:

Note:

Performance thresholds are not identified based on processor, memory or disk utilization because these characteristics will change dramatically following the application rationalization and desktop optimization process. It is likely that the user’s management tools and operating system will change during the migration process. Instead, workload is gauged based on the number and type of applications the user runs.

Assign VDI Models

As with physical desktops, it is not possible to meet every user requirement with a single type of VDI. Different types of users need different types of resources. Some users may require simplicity and standardization, while others may require high levels of performance and personalization. Implementing a single VDI model across the entire organization inevitably leads to user frustration and reduced productivity.

Citrix offers a complete set of VDI technologies that have been combined into a single integrated solution. Because each model has different strengths, it is important that the right model is chosen for each user group within the organization. The following list provides a brief explanation of each VDI model.

• Hosted Apps – The hosted apps model delivers only the application interface to the user. This approach provides a seamless way for organizations to deliver a centrally managed and hosted application into the user’s local PC. The Hosted Apps model is often utilized when organizations must simplify management of a few line-of-business applications. Hosted apps include a few variants:
  • Windows Apps – The Windows apps model utilizes a server-based Windows operating system, resulting in a many users accessing a single VM model.
  • VM hosted apps – The VM hosted apps model utilizes a desktop-based Windows operating system, resulting in a single user accessing a single VM model. This model is often used to overcome application compatibility challenges with a multi-user operating system, like Windows 2019 and Windows 2022.
  • Linux Apps – The Linux apps model utilizes a server-based Linux operating system, resulting in a many users accessing a single VM model.
  • Browser Apps – The browser apps model utilizes a server-based Windows operating system to deliver an app as a tab within the user’s local, preferred browser. This approach provides a seamless way for organizations to overcome browser compatibility challenges when users want to use their preferred browser (Internet Explorer, Microsoft Edge, Google Chrome, Mozilla Firefox and so on) but the web application is only compatible with a specific browser.
• Shared Desktop – With the shared desktop model, multiple user desktops are hosted from a single, server-based operating system Windows Server 2016, 2019, 2022, Windows 10 and 11 Multi-session, Red Hat, SUSE, CentOS, and Ubuntu). The shared desktop model provides a low-cost, high-density solution; however, applications must be compatible with a multi-user server based operating system. In addition, because multiple users share a single operating system instance, users are restricted from performing actions that negatively impact other users, for example installing applications, changing system settings, and restarting the operating system.
• Pooled Desktop – The pooled desktop model provides each user with a random, temporary desktop operating system (Windows 10, Windows 11). Because each user receives their own instance of an operating system, overall hypervisor density is lower when compared to the shared desktop model. However, pooled desktops remove the requirement that applications must be multi-user aware and support server based operating systems.
• Personal Desktop – The personal desktop model provides each user with a statically assigned, customizable, persistent desktop operating system (Windows 11, Windows 10, Red Hat, SUSE, CentOS, and Ubuntu). Because each user receives their own instance of an operating system, overall hypervisor density is lower when compared to the shared desktop model. However, personal desktops remove the requirement that applications must be multi-user aware and support server based operating systems.
• Pro Graphics Desktop – The pro graphics desktop model provides each user with a hardware-based graphics processing unit (GPU) allowing for higher-definition graphical content.
• Local Streamed Desktop – The local streamed desktop model provides each user with a centrally managed desktop, running on local PC hardware
• Local VM Desktop – The local VM desktop model provides each user with a centrally managed desktop, running on local PC hardware capable of functioning with no network connectivity.
• Remote PC Access – The Remote PC Access desktop model provides a user with secure remote access to their statically assigned, traditional PC. The Remote PC Access desktop model is often the fastest and easiest VDI model to deploy as it utilizes already deployed desktop PCs.

Each user group is compared against the following table to determine which VDI model best matches the overall user group requirements. In many environments, a single user might utilize a desktop VDI model and an app VDI model simultaneously.

Don’t forget to follow these top recommendations from Citrix based on years of experience:

1. Start with Windows apps, shared desktops, and pooled desktops – As you can see in the VDI capability table, the Windows apps, hosted shared desktop, and pooled desktop models can be used in most situations. By reducing the number of VDI models required, you help reduce deployment time and simplify management.
2. Perfect match – It might not be possible to select a VDI model that is a perfect match for the user group. For example, you can’t provide users with a desktop that is highly secure and offers complete personalization at the same time. In these situations, select the VDI model which is the closest match to the organization’s highest priorities for the user group.
3. Desktop loss criticality – There are only three VDI models that meet the needs of a high desktop loss criticality user group (backup desktops available) – none of which allow for complete personalization. If a high-desktop loss criticality user group also requires the ability to personalize their desktop they are provided with a pool of backup desktops (hosted shared, pooled) in addition to their primary desktop. Although these desktops would not include customizations made to their primary desktop, they would allow users to access core applications such as mail, Internet, and Microsoft Office.
4. Consider Operations & Maintenance – The ongoing support of each VDI model should be factored in when deciding on a VDI model. For example, pooled desktops can be rebooted to a known good state which often leads to reduced maintenance versus a personal desktop where each desktop is unique.

Quote

Note:

This guide is not intended for Infrastructure as a Service (IaaS) deployments of Citrix in the Azure cloud. The guide focuses solely on deploying using the Citrix Cloud.

System Level Design Considerations

The System level refers to the infrastructure that is core to the Virtual Desktop Infrastructure technology. This is the base layer of the solution and needs to be crafted carefully. Put the required time in planning this layer before rushing into deploying the workload! In this section you will find a list of items to help you focus on the appropriate design decisions related to the Azure and Citrix Cloud control planes.

Azure Specific Considerations

Azure accounts are used for consolidated billing, but cannot contain Azure resources directly. Azure accounts contain one or more subscriptions. Subscriptions serve as security boundaries and they contain the actual Azure resources, such as virtual machines.

A subscription is an agreement with Microsoft to use one or more Microsoft cloud platforms or services. Charges accrue based on either a per-user license fee or on cloud-based resource consumption. Subscriptions can be used to further subdivide the costs or administrative access as required.

Management groups are used within Azure to efficiently manage access, policies, governance, and compliance across subscriptions. They are invaluable for operating multi-subscription tenants in Azure at scale. Each subscription automatically inherits the conditions, policies, and access of its parent management group.

Here are the questions that you need to answer about Azure infrastructure

How many Azure tenants do I need?

• Use a single Azure tenant for the Citrix resources and the users and devices that access those resources

• Use multiple tenants where multiple Azure Active Directories are required. Development/Test having a separate authentication directory or an enterprise that has multiple on-premises AD directory services are two examples.

• The Azure account owner must be associated to the same tenant where the subscriptions for the account are provisioned

• Azure account owners are automatically subscription owners for all the subscriptions in the account

What Microsoft license models should I use?

• Apply the Hybrid Use Benefit (HUB) of your current EA license if it includes Windows Server Software Assurance. HUB significantly reduces compute costs in the cloud. This licensing model can save you up to 40% of the hourly cost because you can use the base VM pricing for Windows Server or SQL Server instances in Azure.

• If using the Microsoft Office suite, use per user licenses that include the Windows 10 Virtual Desktop licenses such as the E3/E5 subscriptions

  • Microsoft 365 E3/E5: Includes Azure Virtual Desktop licenses and Microsoft Office licenses
  • Microsoft 365 Business Premium: Includes Azure Virtual Desktop licenses and Microsoft Office licenses
  • Windows 10 Enterprise E3/E5: Includes Azure Virtual Desktop licenses

How many Azure subscriptions will I need?

• All subscriptions within the same management group must trust the same Azure Active Directory tenant

• A subscription can be associated with only a single account at a time and must have an associated account owner

• Subscriptions cannot share networks, but they can communicate through VNET peering and Azure ExpressRoute

• Subscriptions are boundaries for Azure policies, management, governance and administrative, so plan subscriptions for business units that have separate administrative or billing requirements

• Multiple subscriptions reduce the blast radius and exposure in case credentials are compromised

• Plan to isolate development and test subscriptions from production subscriptions to provide extra performance, security, governance, and compliance

• Some environments such as production and user acceptance testing or preproduction can be shared in a single subscription

• Dedicating subscriptions to Citrix workloads simplify administration and policy management

• Citrix recommends limiting a subscription to 2,500 Virtual Delivery Agents (VDAs)

• Use subscriptions as scale units and scale them out as needed to support the required resources

• Microsoft sets limits on resources within a subscription and those limits must be considered when determining how many subscriptions are necessary to support the Citrix workloads

How many management groups will I need?

• Subscriptions can belong to only one management group at a time

• Management groups are associated with a single parent

• Management groups can be up to 6-levels deep and Microsoft recommends keeping the management group hierarchy as flat as possible

• Management groups are used for policies, not for billing or line-of-business groups. Create management groups based on policy requirements such as instance types, firewall rules, logging, storage, encryption, RBAC model, and so forth

• Limit the number of Azure policy assignments at the management group root, instead of placing them on the individual management groups

• Citrix recommends creating a management group for Citrix workload subscriptions

• Management groups are used for aggregating Azure Policies, so group subscriptions with similar policy requirements together under the same management group

• Use resource tags that can be referenced by Azure policy

For Citrix Cloud to connect and deploy machine catalogs in the Azure cloud, a service principal account is required. That account needs the correct permissions to create, delete, and maintain Citrix resources in each subscription. The service principal account is created through an application registration within the Azure AD tenant. The creation of the service principal account can be created automatically by Citrix or manually by an Azure AD global administrator.

The creation of the service principal object can be accomplished automatically by Citrix if the user running the Citrix Host Connection Wizard has contributor permissions on the subscription. During the host connection setup, the Wizard requests all the required permissions, including contributor permissions on the subscription, and keeps that acceptance for future connections.

Security-sensitive environments do not allow service principals to have contributor permissions at a subscription level. Citrix provides an alternative solution referred to as a Narrow Scope service principal. An Azure AD global administrator needs to manually create an application registration. Then a subscription administrator manually grants the service principal account appropriate permissions. Narrow-scoped service principals do not have contributor permissions on the entire subscription. Their permissions are scoped to just the resource groups, networks, and images that are required to create and manage the Machine Catalogs.

Here are the questions you need to answer regarding the service principal account:

Should I use a subscription-scope service principal account?

• Requires Azure AD global administrator permissions

• Contributor role for the entire subscription is created automatically and Azure will prompt for permissions approval at initial connection

• Use when information security allows a service principal account to be granted contributor permissions on the entire subscription and Citrix administrators have contributor access to the subscription

• Accounts used for authentication during the host connection creation must be at least co-administrators on the subscription and a member of the Azure Active Directory

• Recommended when subscriptions are dedicated to Citrix resources or the environment will contain many resource groups

• Use when a simple management experience is wanted

• Use when Citrix Studio is used to manage the environment more than PowerShell

• Preferred during proof-of-concept deployments

Should I use a narrow-scope service principal?

• The narrow-scope service principal is created manually by an Azure AD global administrator

• Before running the machine catalog Add Machines wizard, the target resource group must be precreated and granted these permissions:

  • Pre-Created Resource Group: Virtual machine contributor, Storage account contributor, and Disk snapshot contributor
  • Virtual Network: Virtual machine contributor
  • Storage Account: Virtual machine contributor

• Recommended when the number of resource groups is manageable either through the Azure console or through automation

• Recommended for higher-security environments where permissions are tightly controlled and fine-grained access control is prevalent

• Recommended when subscriptions cannot be dedicated to Citrix resources and are hosting other services

• Recommended when Azure administrators have different subscription permissions depending on their role

• For larger environments, consider using build scripts or ARM templates to pre-create resource groups and grant the required permissions

Should I use custom roles for the service principal?

• Citrix recommends the use of custom roles for setting permissions for the service principal when more than one subscription will be used

• Microsoft recommends setting the role permissions at the management group level through Azure policy

Citrix Cloud Considerations

Citrix Cloud Considerations

Citrix Cloud, much like Azure, supports multiple tenants and provides one or more Citrix Cloud services for use by the tenant. Each tenant is identified by one or more organizational identifiers (OrgIDs). Companies set up OrgIDs based on how they would like to manage their Citrix assets. OrgIDs are assigned to one of three Citrix control plane regions: United States, European Union, or South Asia Pacific.

Here are the questions that you need to answer about Citrix Cloud

How many Citrix OrgIDs do I need?

• Citrix customers can have one or more OrgIDs based on their organizational structure

• When multiple OrgIDs are in use and are configured in separate regions, use of multiple Citrix tenants can improve user experience.

• Use Citrix OrgIDs to isolate resource usage and for billing simplification

• A single Citrix Cloud OrgID has a limit of 100,000 concurrent users or 3,000 sessions per minute for Citrix DaaS.

• A Citrix Cloud OrgID only supports a single Azure AD tenant for authentication, though multiple Active Directory domains are supported

• Citrix Cloud customers must select a host region for each OrgID

• Citrix recommends using isolated OrgIDs for development and test to keep them separated from production environments

Which Citrix region should I select?

• The Citrix region selected identifies the location of the Citrix control plane. The location of the Citrix VDAs is independent and maps to Azure regions.

• The Citrix Cloud region for an OrgID should be as close as possible to the Azure regions that uses that control plane

• The Citrix Cloud region cannot be changed after it is selected

• Citrix Cloud regions are fully redundant within the geographical region where they are hosted

What Citrix licensing model should I choose?

• For cloud-deployment only customers, you need the DaaS Premium Plus per user license to access the Citrix DaaS.

• For customers with on-premises licenses, the on-premises licenses for Virtual Apps and Desktops, Virtual Apps and Desktops Standard for Azure, and Endpoint Management include cloud services support

• On-premises licenses can be added to your Citrix Cloud OrgID by registering the on-premises Citrix License Server with Citrix Cloud. You can also add the license using the 8-digit short code

What do I need to do to configure Citrix Workspaces from Citrix Cloud?

• Citrix Workspaces support multiple authentication providers including, Active Directory, Azure Active Directory, Citrix Federated Authentication Services, and Okta

• Citrix Workspaces does not support the legacy Program Neighborhood clients (PNAgent). If your environment includes these legacy clients, plan to use on-premises versions of StoreFront servers and Citrix Gateway or upgrade the clients to the Workspace application

• Disabling Workspace Integration prevents users from accessing the service to launch resources, but it does not disable the URL itself

• To support auto-launching of a desktop or application, verify that the Workspace URL is in the local zone or the trusted sizes zone

What is the best way to migrate my on-premises WEM infrastructure to the Azure cloud?

• The WEM Service must be provisioned in Citrix Cloud before the WEM database can be migrated

• Download the WEM migration tool from Citrix Downloads

• Before starting the migration process, configure the database maintenance on the Database Maintenance tab to reduce the database size and shorten the migration process

• The WEM migration toolkit requires .NET Framework version 4.7 or later. If not running 4.7, upgrade first to 4.7 then migrate the database to the WEM service using the toolkit

• A successful migration means that all WEM data in your current database will be lost. Make sure you back up your existing database before starting the migration

• After upgrading the database, on the VDA hosts switch to the service agent mode and point it to the Cloud Connectors then restart the VDA host.

Citrix Infrastructure Considerations

Moving Citrix workloads to the Azure cloud include more than just moving the Citrix server images. Citrix users rely on print services, while applications rely on supporting infrastructure such as databases or web servers. To provide the best experience for users, these supporting infrastructure services and workloads need to be accessible during the transition.

Here are the questions that you need to answer about the supporting infrastructure:

What information do I need to have about my infrastructure servers?

• Basic information: Host name, physical or virtual, appliance or server, the hypervisor and hardware version, server role and purpose

• VM Specs: Number of cores/vCPUs, core speed, Memory

• OS Information: OS Name and Version, End of Support Date, End of Life Date

• Networking: Number of required vNICs, IP addresses, NIC Emulation mode, VLANs Assigned Name / Number, VLANs communications flows, Inbound ports and protocols, Outbound ports and protocols, load-balancer required

• Storage: Root Device Volume (GB), other attached storage and size (GB), virtual disk format, encryption, number of volumes, disk configuration

• Backup: Backup Frequency and Type, Backup Available, Known good backup, Backup Date/Time

• Software: Required Software List, Software Versions, Vendors, File systems accessed, software verified to work in Azure Cloud

• Metrics: Have available metrics over the last 30 days for Max CPU, Steady-state CPU, Max IOPS, Steady-state IOPS, Network Bandwidth (ingress/egress), Max Memory, and Stead-state Memory

• SSL Certificate Requirements

• Migration: System Downtime window, Preferred Application (Infrastructure) Migration Strategy, Recovery Time Objective, Recovery Point Objective, Acceptable downtime during migration, Application Migration Priority, Data Criticality, Assigned Disaster Recovery Tier

• Dependencies: Uses any type of Hardware-based key (like a USB Key), other workloads dependent on this server, does this workload depend on another server or service

How do I verify the applications running on them can be hosted in Azure and determine if the server should be migrated, rebuilt, or sunset?

• Reach out to the application vendor and verify they can be virtualized in Azure. If the vendor does not know, then install and test the application in a development or test subscription

• Determine if applications are necessary to have available in Azure. If not, plan to keep them on-premises until they can be sunset

• Reach out to the application vendors to determine if all the applications in a Citrix workload are supported in Azure. If not, the applications that are not supported may need to be placed in a different Citrix workload that will be kept on premise and eventually sunset

• If the applications are supported in Azure, request information on migrating the application from the vendor

• Use the Azure Migrate: Server Migration Tool to determine if a server can be live-migrated

• If the server is virtual and is supported in Azure, but unable to be live-migrated, consider exporting the OS boot disk and importing it into Azure and creating an image in the gallery

• If the application server is supported but cannot be migrated or the OS disk copied, then plan to rebuild the server in Azure

How will printers and print servers get migrated to Azure?

• Print servers can be migrated to Azure like other infrastructure servers. Any printers that are connected to that print server need to be accessible via a routable protocol like TCP

• Print servers in the cloud will need to be managed just as they were locally

• Consider using Citrix Universal Printing

• Consider looking at centralized SaaS cloud printing services

• Test and verify any printing solutions

What are the best practices for migrating databases to Azure?

• Always keep the database as close as possible to the applications that use the database to avoid unnecessary latency or Azure egress charges

• Azure RDS supports SQL Server, PostgreSQL, and MySQL database engines. Other database engines either need to have their schema, data, and code converted to one of the RDS supported formats or be migrated to IaaS servers running the source database engine

• Always migrate to RDS versions where possible

• Consider using the Database Migration Assistant to assist with any database migrations

What about physical or virtual appliances?

• Identify the best process to patch and secure the appliances in Azure

• Physical appliances cannot be migrated to the cloud, so check with the vendor to verify that virtual versions of the appliances are available that will work in Azure

• For virtual appliances, check with the vendor to verify Azure compatibility, since in some situations you may need to change versions

When is Azure Files a good solution for application data?

• Azure Files supports Windows, macOS, and Linux clients through direct mount of SMB and NFS file shares

• Azure Files supports Active Directory Authentication

• Azure File Sync can be used to replicate SMB file shares between Azure Files and Window on-premises file servers

• Applications can share data through the File REST API interface from any location

• Use Azure Files as persistent storage volumes for stateful containers, such as user layers or profiles

Migrating supporting infrastructure

Use of Azure Migrate: Server Assessment or Movere (Microsoft SaaS offering) is recommended for planning the migration

-  Azure Migrate offers an Azure VMware Solution tool in Preview to assist you in migrating if you are currently using VMware for virtualization in your on-premises data center and plan to use VMware in Azure as well

-  Movere is available through the Microsoft Solution Assessment and Microsoft Cloud Economics Program

Use of Azure VMware Solutions provides the following advantages if you are currently using VMware virtualization in your on-premises data center:

-  Administrators are familiar with the VMware interface and are comfortable with the administration tools and interfaces

-  Migration of VMs can be accomplished through a Microsoft agentless approach that supports replicating up to 500 VMs simultaneously (similar to using vMotion)

Ancillary Infrastructure Considerations

Moving Citrix workloads to the Azure cloud include more than just moving the Citrix server images. Citrix users rely on print services, while applications rely on supporting infrastructure such as databases or web servers. To provide the best experience for users, these supporting infrastructure services and workloads need to be accessible during the transition.

Here are the questions that you need to answer about the supporting infrastructure:

What information do I need to have about my infrastructure servers?

• Basic information: Host name, physical or virtual, appliance or server, the hypervisor and hardware version, server role and purpose

• VM Specs: Number of cores/vCPUs, core speed, Memory

• OS Information: OS Name and Version, End of Support Date, End of Life Date

• Networking: Number of required vNICs, IP addresses, NIC Emulation mode, VLANs Assigned Name / Number, VLANs communications flows, Inbound ports and protocols, Outbound ports and protocols, load-balancer required

• Storage: Root Device Volume (GB), other attached storage and size (GB), virtual disk format, encryption, number of volumes, disk configuration

• Backup: Backup Frequency and Type, Backup Available, Known good backup, Backup Date/Time

• Software: Required Software List, Software Versions, Vendors, File systems accessed, software verified to work in Azure Cloud

• Metrics: Have available metrics over the last 30 days for Max CPU, Steady-state CPU, Max IOPS, Steady-state IOPS, Network Bandwidth (ingress/egress), Max Memory, and Stead-state Memory

• SSL Certificate Requirements

• Migration: System Downtime window, Preferred Application (Infrastructure) Migration Strategy, Recovery Time Objective, Recovery Point Objective, Acceptable downtime during migration, Application Migration Priority, Data Criticality, Assigned Disaster Recovery Tier

• Dependencies: Uses any type of Hardware-based key (like a USB Key), other workloads dependent on this server, does this workload depend on another server or service

How do I verify the applications running on them can be hosted in Azure and determine if the server should be migrated, rebuilt, or sunset?

• Reach out to the application vendor and verify they can be virtualized in Azure. If the vendor does not know, then install and test the application in a development or test subscription

• Determine if applications are necessary to have available in Azure. If not, plan to keep them on-premises until they can be sunset

• Reach out to the application vendors to determine if all the applications in a Citrix workload are supported in Azure. If not, the applications that are not supported may need to be placed in a different Citrix workload that will be kept on premise and eventually sunset

• If the applications are supported in Azure, request information on migrating the application from the vendor

• Use the Azure Migrate: Server Migration Tool to determine if a server can be live-migrated

• If the server is virtual and is supported in Azure, but unable to be live-migrated, consider exporting the OS boot disk and importing it into Azure and creating an image in the gallery

• If the application server is supported but cannot be migrated or the OS disk copied, then plan to rebuild the server in Azure

How will printers and print servers get migrated to Azure?

• Print servers can be migrated to Azure like other infrastructure servers. Any printers that are connected to that print server need to be accessible via a routable protocol like TCP

• Print servers in the cloud will need to be managed just as they were locally

• Consider using Citrix Universal Printing

• Consider looking at centralized SaaS cloud printing services

• Test and verify any printing solutions

What are the best practices for migrating databases to Azure?

• Always keep the database as close as possible to the applications that use the database to avoid unnecessary latency or Azure egress charges

• Azure RDS supports SQL Server, PostgreSQL, and MySQL database engines. Other database engines either need to have their schema, data, and code converted to one of the RDS supported formats or be migrated to IaaS servers running the source database engine

• Always migrate to RDS versions where possible

• Consider using the Database Migration Assistant to assist with any database migrations

What about physical or virtual appliances?

• Identify the best process to patch and secure the appliances in Azure

• Physical appliances cannot be migrated to the cloud, so check with the vendor to verify that virtual versions of the appliances are available that will work in Azure

• For virtual appliances, check with the vendor to verify Azure compatibility, since in some situations you may need to change versions

When is Azure Files a good solution for application data?

• Azure Files supports Windows, macOS, and Linux clients through direct mount of SMB and NFS file shares

• Azure Files supports Active Directory Authentication

• Azure File Sync can be used to replicate SMB file shares between Azure Files and Window on-premises file servers

• Applications can share data through the File REST API interface from any location

• Use Azure Files as persistent storage volumes for stateful containers, such as user layers or profiles

Migrating supporting infrastructure

Use of Azure Migrate: Server Assessment or Movere (Microsoft SaaS offering) is recommended for planning the migration

-  Azure Migrate offers an Azure VMware Solution tool in Preview to assist you in migrating if you are currently using VMware for virtualization in your on-premises data center and plan to use VMware in Azure as well

-  Movere is available through the Microsoft Solution Assessment and Microsoft Cloud Economics Program

Use of Azure VMware Solutions provides the following advantages if you are currently using VMware virtualization in your on-premises data center:

-  Administrators are familiar with the VMware interface and are comfortable with the administration tools and interfaces

-  Migration of VMs can be accomplished through a Microsoft agentless approach that supports replicating up to 500 VMs simultaneously (similar to using vMotion)

Business Continuity/Disaster Recovery Considerations

Every business needs to be operational to generate revenue. The longer a system is down or not functioning, the more revenue that is lost for the business. At some point in the future, if the system is down long enough, the business becomes a going concern and eventually end. For some businesses, the outage can be several months, while other business cannot survive after several days. The key to a good plan is identifying what systems are critical for the business and setting the appropriate mitigation in place.

Here are the questions that you need to answer about Business Continuity and Disaster Recovery:

What Citrix components use Availability Sets or Availability Zones?

• Not all regions support Availability Zones, if they are available in your region, prefer Availability Zones over Availability Sets

• Citrix recommends using the Availability Set or Availability Zones for the following Citrix components:

  • Cloud connector
  • Citrix Gateway
  • StoreFront
  • VDA Hosts

What information should I back up?

• Backup anything that is important. Set the backup vault frequency to match the RTO and RPO requirements for the server and set the retention based on corporate data retention policies

• Backups are not automatically copied to the paired region. This task must be done manually or scripted

• Persistent desktops nmust be backed up

• Golden image machines should be backed up

What regions should I place my resources in?

• Azure regions should be selected primarily based on data sovereignty, governance, and compliance

• Place Citrix resources in Azure regions that are close to your users

• Azure regions are paired for disaster recovery/business continuity. Take note of the regional pairs and select the pairs that work best for your users and business continuity plan

• Depending on where your users and data centers are located, one regional pair may be a better choice than another regional pair

• When planning for disaster recovery, use the paired region for backups and failover configurations to increase the probability of a successful recovery in the unlikely event of a regional failure

What information should I store in the paired region?

• Identify key servers (recovery tier 0) that absolutely must be up 100% of the time, such as a domain controller, since they are good candidates for replication

• Any servers that are using Azure Site Recovery (ASR) should be pointing to the paired sister region for

• Identify key images/backups for servers that have to be brought online quickly and do not have an automated build available

• Schedule PowerShell jobs to replicate the key images, backups, and snapshots across to the other region

• Regularly export the ARM templates for your configurations and store them in the paired region. This task can be scripted.

What is appropriate to use Azure Site Recovery Replication?

• Any core infrastructure that is considered Recovery Tier 0

• Keep at least one domain controller at the recovery site so the domain passwords are current. That allows the users doing the restore to authenticate to the ASR environment.

• Keep at least one Citrix Gateways for remote access into the system.

Workload Level Design Considerations

The most dynamic part of a Citrix virtualization system is the VDA. Remember that VDAs are where the actual work is happening. The apps and desktops you provide users on a Citrix virtualization system run from VM instances on Azure. You want to make sure you get this layer right! Do your homework up front. Set the expectation with users that the system will change over time and build simple and effective processes to handle change. With the power and flexibility of Citrix virtualization tech, managing change doesn’t have to be a major burden.

In this section, we’ve attempted to logically break the topic up such that we can dive deep without losing context. We do our best to provide the details you need in each section and call out leading practices and recommendations along the way.

Migration Considerations

Citrix servers can be hosted on various platforms, including Hyper-V, VMware vSphere, and physical servers. The most time-consuming part of moving Citrix to the cloud is the planning process. The planning process requires both discovery and analysis to determine the best path for the migration. The end result of the analysis is a document that provides a migration plan. Here are the questions that you need to answer about Citrix Workload Migrations:

How do I migrate my Citrix VDA hosts to Azure?

• Azure Migrate integrates with Azure services and supports other third-party tools. Azure Migrate helps you complete the migration process from an on-premises data center to Azure.

Are there any utilities available to help migrate applications and application data to Azure?

• Azure Migrate includes tools to help discover and assess the current environment and plan the migration of applications, servers, and data to Azure.

How do I use Azure for capacity planning?

• Use the Azure Migrate Server Assessment for planning the migration when possible.

• Movere (Microsoft SaaS offering) is another possible Microsoft tool to assist with migration assessment. Movere is only available through the Microsoft Solution Assessment and Microsoft Cloud Economics Program.

How do I move my application data to Azure?

• Azure Migrate offers an Azure VMware Solution tool in Preview to help with migrating all of your servers, including the Citrix workloads.

• Use Azure VMware Solution (AVS) integration to provision your Citrix VDA workload just like other vSphere workloads in your on-premises data center

Azure VMware Solutions

Using Azure VMware Solution (AVS) provides the following advantages if you are currently use VMware virtualization in your on-premises data center:

• Cloud Migration Support: Easily migrate desktops and applications between VMware deployments, including replicating up to 500 VMs simultaneously.

• Familiar Administration: Administrators are familiar with the VMware interface and are comfortable with the administration tools and interfaces.

• Data center extension: Use the data center extension to provide burstable capacity and support remote locations. The data center extension helps during periods of high demand, disaster recovery or business continuity by taking advantage of the rapid elasticity provided within AVS.

Scalability Considerations

Which Azure instance series are best for hosting my Citrix Virtual Apps and Desktops workload?

• Select the D/DS series instances when your applications consume a fair amount of memory. The D/DS series have a higher memory-to-CPU ratio.

• Select the F/FS series instances when you need excellent CPU response times and do not require a significant amount of memory. The F/FS series have faster processors but lower memory-to-CPU ratios than the D/DS series instance family.

• The most cost-effective instance type is the F/FS series with 8 or 16 vCPUs followed by the D/DS series with 4 vCPUs.

• Choose instance types with fewer vCPUs when you want to:

  • affect fewer users during maintenance windows or unexpected server performance issues
  • scale down quicker to take advantage of cost savings with Autoscale

• Choose instance types with more vCPUs when you want to:

  • Reduce the API calls to Azure infrastructure for operations
  • Manage fewer machines

Performance Coniderations

Should I enable Machine Creation Services I/O (MCSIO) cache?

• If Service Level Agreements (SLAs) are not required in your environment, use the less expensive standard disk for the MCSIO cache

• Consider enabling MCSIO cache when user experience is a high priority. The fast response times of the memory cache make a noticeable difference for the users.

• Using a memory cache size of 2 GiB provides the best improvement without negatively impacting user density. Always account for the extra memory when choosing an instance size for the workload

• Do not enable MCSIO cache on memory-constrained machines, such as the F/FS series where the memory to CPU ratios are low

Cost Optimization Considerations

How can I use Citrix Autoscale for both on-premises and cloud workloads?

• Use cloud workloads for burst capacity or for business continuity by tagging cloud workloads in the delivery group. Set Autoscale to power manage these tagged workloads. Use the selective power management feature to set the zone preference and failover to prefer on-premises workloads.

• Use the dynamic provisioning feature of Autoscale with low and high watermark machine counts. This approach reduces costs and still support demand under high loads. This feature is especially useful for instance types with high monthly charges.

• If usage and capacity patterns can be predicted on a consistent basis, use the schedule-based scaling feature to manage available capacity. Configure the settings to start within a 30-minute window. When paired with the capacity buffer, the scheduler prevents users from waiting for capacity to come online.

• When enabling Autoscale, prefer smaller instances over larger instances. Smaller instances drain faster and go offline sooner.

Should I use Windows 10 Multisession or Windows Server OS?

• Windows 10 Multisession has about 10% lower densities than the Windows Server operating systems

• Windows 10 Multisession allows users access to the Windows Store, which is not available on the server operating systems

• Azure Virtual Desktop (AVD) entitlement grants you the base VM price (Linux Pricing), saving a considerable amount over the normal Windows VM price.

Application Considerations

Microsoft Office and Microsoft 365 are among the most popular workloads delivered by Citrix today. Both Microsoft and Citrix have worked together to develop the best user experience when running Microsoft 365 from a Citrix session in Azure. Their collaboration created applications, processes, and guidance to help you deliver the best of breed solution. You likely have other applications hosted on the Citrix servers that must be analyzed and migrated to Azure. These applications have application data that must be accessible regardless of where the application resides. Here are the questions that you need to answer about Applications and Application Data:

How do I integrate the VDA-hosted applications with Microsoft 365?

• Use Office 365 ProPlus when installing office

• Microsoft 365 requires a plan that supports shared computer activation, which is required for any multi-user session hosts

• After installing Office on a golden image, do not open any Office applications. If you open one Office application, you must reset the image to remove the temporary key which prevents user-level activation. To reset the image, uninstall Office, reboot and then reinstall Office.

• For earlier versions of Office that use KMS licensing, such as Office 2010 and 2013, you must verify your KMS server is reachable from the Azure cloud. You can make your KMS server accesslble to your Citrix workloads by one of these methods:

  • Migrate your KMS server to the Azure cloud
  • Connect your on-premises data center to Azure using either an ExpressRoute or a Site-2-Site (S2S) Virtual Private Network (VPN)

• When using FSLogix and Office 365 containers, follow these steps to integrate it with Windows Search, rebooting between each step:

  • Configure Automatic Startup (not delayed) for the Windows Search service. This configuration should be completed before installing Office so Office sets the required hooks.
  • Install Microsoft Office.
  • Install the FSLogix agent.
  • If you do not need Windows Search, you can disable the service. Before disabling the service, go ahead and complete the install steps to save on compute resources, then disable the service. With this approach, if later it is needed, you can enable the service easily.

Should I use FSLogix with Citrix workloads?

• Use Microsoft GPOs to manage all Microsoft 365 Office settings

• Microsoft FSLogix is the recommended approach to handle Microsoft365 integration. It handles the Outlook Search, Outlook PSTs and Office activation seamlessly.

• Microsoft recommends using SSO (ADFS) with Microsoft365 Apps 1704 and above:

  • When ADFS is available, enable the "Automatically activate Office with federated organization credentials” GPO and configure the automatic logon in GPO Security Logon
  • if ADFS is not available, use FSLogix or Citrix Profile Manager to synchronize the following registry key %localappdata%\Microsoft\Office\16.0\Licensing to roam the Microsoft 365 token with the user

How should I configure Outlook? (Cached mode or online mode)?

• Use Cached Exchange Mode when the following conditions are true:

  • A profile management solution such as FSLogix or Citrix Profile Manager is available to manage the OST file and the search index
  • Users require a more responsive email system
  • Connections between the Outlook client and the mail server have high latency or are frequently disrupted

• Use Online Mode when the following conditions are true:

  • Low latency network connection is available

• Use Active Directory Group Policy to configure Outlook Exchange mode, recommended settings include the following:

  • File > Cached Exchange Mode
  • Sync Settings
  • Disable Fast Access
  • Use Cached Exchange mode
  • Cache file

What settings should I use for Microsoft 365 when using Citrix Profile Management?

• When using Citrix Profile Management use these items to provide a robust user experience and support the OST/PST storage locations and the Search Index locations

  • Use the latest version of Citrix Profile Manager. The latest version has features such as Native Outlook Search and Large File Handling which provide optimizations for Outlook.
  • Enable Large File Handling to allow storing OST/PST files on Azure Files.
  • Include these folders and registry in the Citrix Profile Management configuration:
    • %localappdata%\Microsoft\Office\16.0\Licensing
    • %localappdata%\Microsoft\Credential
    • AppData\Local\Microsoft\Credentials
    • AppData\Local\Microsoft\Windows\WebCache
    • AppData\LocalLow\Microsoft\CryptnetUrlCache
    • AppData\Local\Microsoft\Outlook
    • AppData\Local\Microsoft\Vault
    • AppData\Local\Microsoft\Office
    • AppData\Local\Microsoft\Office\*.qat
    • AppData\Local\Microsoft\Office\*.officeUI
    • AppData\Local\Microsoft\Windows\UsrClass.*
    • HKCU\Software\Microsoft\Office\16.0\Common\Identity\DisableADALatopWAMOverride

Where should I store my application data?

• Use Azure Migrate or Movere to assess and plan the application migration

• Check with the application vendors to determine if the software is supported in Azure. If planning to use PVS for streaming, also verify that the vendor supports Gen 2 VMs.

Where should my data be located relative to my application?

• Data leaving Azure incurs an egress charge. Try to keep the applications and their data as close as possible to one another. Although ideal, this configuration is not always possible. When you cannot keep your data close to the application, focus on minimizing the latency between them.

What costs should I consider when determining my data location?

• When working in a hybrid cloud environment that prevents both the application and its data from moving to Azure together, move the application first then move the application data. With this approach, the data egress charges are reduced.

How do I integrate Citrix Workspace with Microsoft 365 and Microsoft Teams?

• For multi-session hosts, install Microsoft Teams after the VDA is installed on your golden image and install it under c:\program files using the ALLUSER=1 flag

• Updates to the Microsoft Teams agent require an uninstall of the previous version before installing the new version

• Set the Citrix Microsoft Teams redirection policy to allowed

• Microsoft Teams relies on Azure Transport Relays the following ports and IP address ranges must be accessible

  • UDP 3478-3481
  • TCP 443
  • 137.106.64.0/18
  • 52.112.0.0/14
  • 52.120.0.0/14

• Use Citrix Director’s Activity Manager to monitor Microsoft Teams applications such as WebSocketAgent.exe, WebSocketService.exe, and CtxSvcHost.exe.

Image Management Considerations

The primary image management solution used in Azure is Machine Creation Services (MCS) and until recently was the most common option for Citrix image management. Citrix has been focused on improving the image management within Citrix Cloud. These improvements help ease our customer's migration to the Azure cloud and spin up any workload in a matter of minutes.

One of the new Citrix services includes the Image Portability Service (IPS). This service provides a way to port images between your on-premises data center and your Azure environment. The service uses a temporary Virtual Machine (VM) to host the Compositing Engine (CE). The CE has two modes: prepare or export. The prepare mode converts virtual disk formats used for Citrix Workloads and updates the portable properties. The export mode copies the prepared disk to the target cloud. A connector appliance controls the individual jobs, spawns the CE VM, and secures communications between Azure and your on-premises environment.

Azure supports a version of Citrix Provisioning Services (PVS) which allows you to stream a virtual disk to multiple VMs simultaneously. With PVS, you can create 2500 identical VMs within a single subscription. Most of the administration and function is the same as for an on-premises PVS environment, except for a few changes. Azure does not allow the use of the Pre-boot Execution Environment (PXE), so changes were required for the streaming and boot process. This version of PVS includes a UEFI-based Boot Disk Manager (BDM) to create UEFI boot disks. These boot disk require Azure Gen 2 VMs and cannot support 32-bit operating systems. The Citrix broker is now responsible for all power management of the PVS VMs, though the PVS console can still power off the targets. The Citrix Virtual Apps and Desktops Setup Wizard handles all the provisioning steps.

How can I use the image portability service to move my Citrix workloads?

• Deploy golden images on-premises, using either Machine Creation Services or PVS.

• The entire process for using IPS is completed through PowerShell commands from a remote workstation.

• The latest version of PowerShellGet should be installed on the remote workstation before beginning the process.

• Citrix connector appliances must be installed at each resource location where IPS is used.

• At the on-premises location, a Windows SMB File share is required for temporary data storage during export jobs. The file share should have enough free space to hold two copies of the disk to be exported.

• Automated publishing is only available with PVS on Azure deployments. Manual publishing to Azure is available for both PVS images and MCS images.

• The following machine catalog configurations have been tested with the IPS

  • Windows Server 2016, Windows Server 2019, or Windows 10 2004 or later
  • Source images provisioned by Citrix Provisioning 1912 or later
  • Citrix Virtual Apps and Desktops VDA 1912 or later

What are the limitations and requirements for deploying PVS in Azure?

• The Azure subscription must have the ReserveMacOnCreateNic feature enabled.

• At most 2500 VMs can be streamed in a single subscription.

• All provisioned VMs must be created in the same region as the hosting unit. VMs are automatically spread across all availability zones in the target region.

• Cross-region provisioning is not supported.

• Requires UEFI boot of Generation 2 Azure VMs. Generation 1 (BIOS-based) VMs are not supported. Boot images that are created using the PXE or ISO format are not supported.

• Supports only 64-bit versions of Windows 10 and Windows Server 2019 for streaming.

• Use the Citrix Image Portability Service to import an existing image.

• Active Directory support is required for machine naming using one of these methods:

  • Azure Active Directory Domain Services (AADDS) added to your Azure Active Directory (AAD) tenant
  • ExpressRoute connection to your on-premises Active Directory environment
  • Active Directory domain controllers installed in Azure and synchronized with your on-premises AD environment and the AAD tenant

• When using Azure Files Services for vDisk storage, premium storage or Azure NetApp Services is required and must be in the same region as the PVS server.

• SQL Server or SQL Server Express VM is required. Currently, using any authentication method except Windows Integrated Authentication or using an Azure SQL database is not supported.

• The Golden VM must have same disk and vGPU configurations.

• Set up a virtual network for streaming to targets and peer that network with the network used for the VM communication to Active Directory. Use the AD Domain Controller IP addresses for DNS servers.

• The PVS Server VM must have at least one vNIC on each virtual network where targets reside. The PVS server should also have at least 2 vCPUs and 8 GiB of memory.

What are the best practices for Image Management in Azure?

• Always make a copy or snapshot of your golden image and use the copy or snapshot for machine catalog images. This practice allows for easy image updates and provides protection against image corruption.

• Point existing image-based machines (PVS and MCS) to the cloud connectors by modifying the ListOfDDCs registry key on the golden images. The registry key can be found at HKLM\Software\Citrix\VirtualDeliveryAgent. After modifying the registry key, take a snapshot of the image. Update the machine catalog with the new snapshot when ready for the images to register with Citrix Cloud.

• Use the Citrix Group Policy (Computer Configuration > Citrix Policies > Controllers) to point the other Citrix workload servers to the FQDN of the cloud connector and set the Enable auto update of Controllers to “allowed”.

• Use Managed disks for the golden images, unless you are using App Layering.

• Be sure to keep copies of golden images is different regions.

• Automate the build of the Golden Image

  • PowerShell SDKs can be used for full automation of the Golden Image build

    • PowerShell v5.x
    • Azure PowerShell Module
    • Citrix Cloud Remote PowerShell SDK

  • Use Azure PowerShell to do the following:

    • Create a new Azure VM
    • Configure the Windows Firewall
    • Remove the Azure Public IP address (if present)
    • Automate domain join
    • Automate software installs such as the VDA
    • Seal the image
      • Reset Log files
      • Reset Office Licensing
      • Clear GPO cache
      • Remove user profiles used for build process
    • Copy golden image to business continuity region

  • Install any other software manually that cannot be automated.

  • Use the Citrix Cloud Remote PowerShell SDK to do the following:

    • Update the Machine Catalog

  • Do not store passwords unencrypted in any scripts or change any stored passwords (such as the local admin password) immediately after building.

Should I do anything different from on-premises when managing MCS images in Azure?

• Use the Azure shared image gallery for storing MCS images to reduce the time required to create and hydrate the OS disks and improves application performance.

How do I manage images across geographic regions?

• Use Azure shared image gallery to provide multiple replicas in different regions.

How do I enable MCSIO in Azure?

• MCSIO can be enabled on MCS catalogs in Azure by using PowerShell to create a new Provisioning Scheme and Catalog
  • Use VDA version 1912 or later for best results since not all earlier versions are supported.
  • Do not forget to pre-create a Resource group in Azure for the MCS catalog.
  • Install Citrix Remote PowerShell SDK to access Citrix Cloud configurations.
  • In Azure, the write-back cache is stored on non-persistent media.

Image Layering Considerations

Use App Layering inside Azure for the same use cases that you would use it in your on-premises data center:

• Manage a significant number of Machine Creation Service (MCS) images
• Provide persistent desktops for users using non-persistent VDA hosts with MCS
• Limit the rebooting of the Virtual Delivery Agent (VDA) hosts

NOTE: App Layering requires Gen 1 VMs and Provisioning Services (PVS) requires Gen 2 VMs so the two services are currently incompatible.

App Layering works almost the same way in Azure as on-premises. Here are some of the questions that you may have about App Layering.

What are the differences between using Citrix App Layering on-premises and using it in the Azure cloud?

Each application along with its related software should be installed in its own layer. These guidelines help you plan the layers.

Base Operating System (OS) Layer

Start with a new OS image and use only a single OS layer and choose Resource Manager from the listed deployment models

• Do not select the “Use managed disks” option. In Azure, Layering requires a storage account.

• Verify that the OS is set to use DHCP for IP addressing.

• If using an Azure VM that has the Page file on the D: drive, move it back to the C: drive before capturing the OS layer. With this change, the image will still deploy correctly in production. The requirement is just temporary during the single disk OS image capture process.

• Do not use or include an UNATTEND.TXT file, since the Layering process removes it automatically.

• Use ngen.exe to pre-compile .NET executables.

• Set Built-in Administrator to “Password Never Expires”.

• For server OS builds, set the PowerShell Execution policy to unrestricted and enable PSRemoting.

• Install the App Layering Services on the OS Layer.

• Install App Layer OS Machine Tools and follow the instructions for KMS scripts if using KMS Licensing. The Citrix App Layering OS Machine Tools include special scripts to automatically handle the complexities of Microsoft Licensing and prevent any misconfigurations.

Platform Layer

The platform layer consists primarily of software not included in the base OS layer that connects to other infrastructure.

• Join the Active Directory domain and verify that the user name is in the format DOMAIN\Username. Ignore the default request for just the user name.

• Install the provisioning software, Citrix VDA, and Citrix Workspace Environment Manager into this layer.

Publishing the Image

Create an image template and use that to publish an image for MCS.

• The new image appears as a VHD in the Storage account’s container citrix-al-images.

• If your App Layering version is earlier than 4.15, attach the image to an Azure VM and boot the VM to let Sysprep complete its tasks.

• If your App Layering version is 4.15 or later, use the Azure Connector for MCS, since it does not Sysprep the image.

• Choose the disk file in the storage account as the Golden image when creating or updating the machine catalogs.

What permissions are required for using App Layering in Azure?

• Use Accelerated networking for your Enterprise Layer Manager (ELM) virtual appliance to improve performance.

• The ELM appliance uses the Azure Service Principal to access Azure resources. Both the service principal and the user installing ELM must have at least contributor permissions on the resource groups used by App Layering.

• Use Azure premium storage for packaging machines and image layers to reduce packaging time.

How do I support Microsoft 365 and KMS licensing with App Layering in the Azure cloud?

• Place Microsoft Office in its own layer with all the Office Add-ons
  • Install Office into the default location.
  • Do not open any Office applications during the installation and packaging process.
  • In the Optimize script, be sure to enable “Activate MS Office via KMS” and select ONLY the versions of your installed Office products. The script will only run successfully if Microsoft Office is installed in the default location.
  • Run the Office2013Windows81_PREP.cmd for all versions of Microsoft Office starting with Office 2013, this includes Microsoft 365.
• Use larger layer sizes if users can store large files in the application layer. Increasing layer sizes later to support large PST and OST files is difficult.

Where are the Elastic Layers stored in the cloud?

• Elastic layers are mounted dynamically when a user logs on and provide access for applications. Elastic layers are normally stored on network file shares.

• Elastic layers must be available 100% of the time. If the elastic layer is unavailable, even for a short time, all connection layers fail and the VDA host must be rebooted to fix the issue. Options for storing elastic layer files on always available storage include:

  • Scale Out File Server for Application Data
  • Azure Files (premium storage recommended)

User Level Design Considerations

In this section you will find a list of items to help you focus on the appropriate design decisions specific to users and their data.

Authentication Considerations

Active Directory Domain Services (AD DS): This service is the traditional on-premises Active Directory infrastructure that supports GPOs, Kerberos authentication, and domain joins. A new AD DS can be created and hosted on virtual machines in the cloud. Alternatively, an existing AD DS infrastructure can become a hybrid model with some controllers in Azure and some on-premises. In both deployment scenarios, the AD DS domain can be synchronized to Azure Active Directory (AAD) using Azure AD Connect.

Azure Active Directory (Azure AD): This service is Azure’s authentication cloud-based identity and mobile device management service that provides user authentication. Azure AD does not support device authentication, domain joins or group policy objects (GPOs). However, Azure AD can be paired with Azure Active Director Domain Services (AAD DS) to provide the minimum level of support needed for Citrix in Azure.

Azure Active Directory Domain Services (Azure AD DS): This service is a managed domain service hosted in the cloud. This service supports GPOs, Kerberos authentication, and domain joins. The difference between Azure AD DS and AD DS is that the AD domain controllers are managed by Microsoft rather than you. Azure AD DS integrates directly with Azure AD and is a great option for cloud-based Citrix deployments.

Here are the questions you need to answer regarding Active Directory infrastructure:

Should I continue to use only my on-premises Active Directory infrastructure?

• Easy to deploy by just installing Cloud Connectors and joining them to the on-premises domain

• Citrix Cloud can be configured to use only the on-premises AD Domain

• Using or synchronizing to Azure AD is not required, leaving Azure AD to be solely used as identity management for Azure administration

• To prevent latency introduced by domain authentication, Citrix recommends placing domain controllers near the Citrix Virtual Delivery Agent (VDA) hosts and the Cloud Connectors

• Approval from information security (infosec) may be required to place domain controllers in Azure

• Not recommended for deployments that use Microsoft 365 and that have users logging into Azure AD for that service

Should I extend on-premises Active Directory using Hybrid Mode with Azure AD Connect?

• Microsoft recommends this design when you have an existing on-premises AD infrastructure and need either of these features:

  • schema extensions
  • account-based Kerberos constrained delegation

• At least one Active Directory domain controller should always be available for the Cloud Connectors and VDAs. This design prevents any authentication bottlenecks or latency during the group policy processing, domain joins, and authentication events

• Citrix recommends this model when you have Citrix workloads that are still on-premises

• Citrix recommends this model if Citrix Cloud services such as Endpoint Management will be used

• Place at least two domain controllers in Azure and use Azure AD Connect to synchronize AAD with AD DS over ExpressRoute or VPN

• Using Windows 10 under a Hybrid Use Benefit license requires computer accounts and user accounts be in the same Azure Active Directory

• Approval from information security (infosec) may be required to place domain controllers in Azure

• If using smart cards, Kerberos must be enabled on a domain controller

Should I establish a new Azure Active Directory(AAD)?

• Microsoft recommends this model when you are using a cloud-only deployment or when you do not have an existing on-premises AD infrastructure

• Citrix recommends this design when all your Citrix workloads are in Azure and using one of these services:

  • Citrix DaaS

• Plan to use Azure AD Connect to synchronize Azure AD with Azure AD DS and enable password hash synchronization

• If using smart cards, Kerberos must be enabled for Azure AD DS

• Using Windows 10 under a Hybrid Use Benefit license requires computer accounts and user accounts be in the same Azure Active Directory

• Azure AD DS does not support schema extensions, one-way trusts or account-based constrained delegation for Kerberos

• Azure AD DS does not support Domain or Enterprise Admin privileges

Should I use Azure AD as the Citrix Cloud Identity provider?

• Citrix Cloud supports both Azure AD and AD DS for authentication

• When using Azure AD as the Citrix Cloud Identity provider you maintain control of password policies and can easily disable accounts

• Using Azure AD provides multifactor authentication (MFA) to increase the security posture for Citrix Cloud

• When Azure AD is branded, Citrix Cloud has a branded sign-in page

• Azure AD extends Citrix Cloud to support federated identity options such as Okta, Ping or ADFS

• Requires Global Admin role for consent to allow Citrix Cloud to connect to Azure AD. If access to this role is not available, consider using other identity providers such as on-premises AD or the default Citrix identity provider.

Should I enable Multifactor Authentication (MFA) on the Azure Active Directory Accounts?

• Multifactor authentication is always recommended for any resource that is accessed over the internet. Multifactor authentication decreases the available attack vectors and increases the security posture of the system.

• Azure AD makes enabling MFA simple and integrates easily with the Citrix Cloud identity provider

• If not using Azure AD for MFA, consider other identity providers such as Okta to provide this additional security.

Migration and Management Considerations

Most end-users are connecting from outside of the Azure cloud when accessing cloud resources using their own devices or devices from your enterprise. The user and device characteristics influence greatly the design and architecture of the cloud environment and the recommended migration paths. How you manage your environment today imposes certain requirements if that management system is moving into Azure.

Users are managed primarily through the directory services. If you are using a cloud-only deployment of users, you would start by creating an Azure AD tenant. Then you link that tenant to Azure AD and create the users and groups directly within Azure AD. If you have an existing deployment, you can use Azure AD Connect to synchronize your AD users and groups to Azure AD automatically.

Usually, installing and configuring Azure AD Connect to synchronize with Azure AD is a bit of a time-consuming process. The time spent setting up Azure AD Connect is worth it because users are able to access resources easier. Implementing Azure AD is recommended as the best long-term cloud strategy for authentication.

Here are the questions you need to answer regarding User Management:

Which identity provider do I need for Citrix Cloud?

• Citrix cloud supports the folloing identity providers natively:

  • On-premises Active Directory
  • Azure Active Directory
  • Citrix identity provider
  • Over 20 third-party federated providers such as Okta or Ping

• Select the identity provider that makes the most sense for your Citrix Cloud deployment. Do not forget to consider all existing applications and their requirements to integrate with cloud services

• Determine if using a federation established with an on-premises deployment is a requirement for user identities. Examples of potential federations include Kerberos-based SSO, SAML, or MFA with smart cards or hardware tokens like RSA SecurID.

How do I move my existing on-premises AD users and groups to Azure AD?

• Review available Microsoft documentation to determine what design works best for your business requirements

• Verify that the licensing model for your Azure AD supports the features and number of users for your environment.

• Microsoft recommends installing a domain controller in Azure to synchronize with your on-premises domain controllers over Azure ExpressRoute or VPN. Having a domain controller in Azure improves the Azure AD Connect synchronization performance.

• Install and configure Azure AD Connect and allow it to synchronize your users and group memberships over to Azure AD

• A single Azure AD Connect server is limited to a single forest for synchronization. Using Azure AD Connect is more complicated when multiple AD domains within the same forest are involved in the synchronization process.

• Depending on the hybrid identity required, different options may be enabled:

  • Password hash synchronization (PHS)
  • Pass-through authentication (PTA)
  • Multifactor Authentication (cloud-based only)
  • Single sign-on with Federated Services (smart cards, password expiry notifications, on-premises MFA)

• If not all the users must be synchronized to Azure AD, Azure AD Connect supports filtering at the domain, organizational unit, attribute, or group level.

• Filter the AD scope to only include objects that need to be in Azure AD

• Large directories take a considerable time to import. Currently, Azure AD throttles write operations to 84,000 per hour so you need to allow adequate time for a full sync to occur.

• Monitor and configure alerts using the Azure AD Connect Health portal in Azure.

Profiles and User Data Considerations

Profile management solutions are designed to make a user’s local profile portable so that it can be accessed from any session or device. Both Citrix User Profile Management (UPM) and Microsoft FSLogix improve on the traditional roaming profile model used in data centers. Both solutions improve the response time for users and store the user profile using Azure Files. The benefits of Citrix Profile Management are outlined below.

Citrix User Profile Management

• Integrates with the following products:
  • Citrix Virtual Apps and Desktops (Citrix DaaS)
  • Citrix Workspace Environment Management (WEM) service
  • Azure Files
• Virtualizes user profiles so the user settings can be applied to the user desktop or application
• Streams profile data so that it is not downloaded until needed
• Offers large file handling which allows large files to be redirected individually providing a native (local) file experience
• Supports profile exclusions to reduce bloat
• Supports multiple concurrent file accesses for multi-session users
• Supports profile containerization
• Improves logon speed
• Implements containerization through redirection of users profiles to a virtual hard disk
• Supports profile containers and Microsoft Office containers
• Maintains user data for non-persistent environments, such as Citrix session or Azure Virtual Desktop
• Reduces logon times by mounting a VHD instead of copying user profile data across the network
• Supports profile exclusions to reduce bloat
• Provides a native (local) profile experience for users
• Integrates with OneDrive and Azure Files

Profile Design Considerations

Some applications are not designed with roaming users in mind and rely on local file caches and indexes that do not roam between sessions. Microsoft Outlook is one of the more popular applications with this behavior. Both Citrix User Profile Manager and Microsoft FSLogix can provide an improved user experience with these types of applications. Other design considerations include:

• Users accessing their data from multiple sessions simultaneously require solutions that support that level of file access.

• Keep user data as close as possible to the user’s session. When users are accessing from both on-premises and cloud-based sessions, choose the cloud when possible.

• With both profile management solutions, permissions for profile stores must be configured manually. Support for multi-session simultaneous access requires extra configuration.

  • Always combine profile management with folder redirection to reduce the amount of data copied locally
  • Always configure profile folder exclusions to reduce bloat, since they are not configured by default
  • Always enable large file handing for Citrix User Profile Management so that large files, such as PSTs or OSTs, are not copied down

• Antivirus exclusions are required for both FSLogix and Citrix User Profile Management profile solutions because they implement system-level drivers for redirection.

• When using Microsoft FSLogix (or Citrix Profile Containers), you must exclude VHD(X) containers from AV scanning when hosted internally on traditional file shares.

• Azure Files sync can replicate containers quickly and easily for staged deployments.

User Data Challenges

One of the biggest challenges with migrating to Azure includes how to manage user profiles and access to personal and department data. Users require their data to perform their job and they need to access it from any device they are using. This section provides guidance for the challenges associated with user data and considerations that influence the design.

The goals for user data are:

• provide access to the data securely
• provide access to it always from any location
• provide access with the lowest latency possible

Meeting these goals is a challenge with a hybrid environment where some Citrix workloads are in the cloud and some remain on-premises. The user’s data cannot be in both places at once without creating data collision opportunities. Selecting a single location introduces security, latency, or access concerns. This dilemma is true for both the user data and the shared department data.

Windows Profiles

Windows still relies on the concept of profiles to store user data. The loading of those profiles can significantly impact a user’s logon experience, especially when the user’s desktop contains a large amount of data. The logon experience is made worse when the user's session has a significant amount of latency between the profile store and the session host. Several technologies, such as Citrix Profile Manager and Microsoft FSLogix, have been developed to help remove these pain points. The information below helps you select which technology is best for your users.

When should I use the traditional file server technologies for hosting user data?

The traditional file server technologies are file sharing solutions that are used in data centers today. Often these technologies use Distributed Files System Replication (DFS-R) or Distributed File System - Namespaces (DFS-N) to make file shares highly available across multiple locations. Accessing these file shares from an on-premises location typically introduces high latency because of routing and protocol latency. The different file server technologies along with their benefits and drawbacks are provided below.

• Standalone File Servers: Windows Server configured as file servers

  • Requires management and maintenance
  • Has potential cost advantages when hosted in the cloud compared to other server-based technologies
  • Compatible with familiar backup/restore products
  • The standalone server is a single point of failure since it has downtime during updates that force the server to reboot

• Storage Replicas: Windows Server technology that enables synchronous replication of volumes between servers or clusters

  • Requires management and maintenance
  • Supports block-level replication (synchronous or asynchronous)
  • Supports the SMB 3.0 protocol which includes security enhancements such as encryption
  • Has only minimal downtime during manual failover between replicas

• Storage Spaces: Windows Server technology that allows drive pooling in a RAID-type configuration and can be clustered across multiple server nodes for high-availability

  • Requires management and maintenance
  • Supports SMB 3.1 which includes transparent failover mechanisms
  • Has a multi-node topology that can scale up/out as necessary,
  • Has a transparent failover,
  • Uses 3 times more disk space than a traditional file server
  • Not always supported by third-party backup/restore products

• Traditional file servers work best in a data center where the Citrix workloads have direct access to the file share.

• Traditional file servers support the installation of governance and security software, such as:

  • Data loss prevention (DLP)
  • Antivirus (AV)
  • Backup
  • Encryption software
  • Host-based Intrusion Prevention System (HIPS)
  • Host-based Security System (HBSS)

• Some traditional file server deployment configurations result in lower overall costs compared to the PaaS shares.

• Use traditional file servers when your organization needs complete control over the data. With a traditional file server, governance and legal ownership is easy to maintain and data classification is easier to implement.

• Traditional file server technologies are used when applications need nearby for compute or when an extensive amount of read/writes are expected on the data.

• Traditional file server technologies represent less durable data storage when compared to cloud-based alternatives such as Azure Files.

• Look at the scalability path for meeting demand, scale out or scale up with current hardware.

When should the PaaS file shares be used?

These cloud-based file services were built specifically as a service instead of an application and optimized to operate over the internet.

• Azure Files: File shares as a service backed by Azure storage

  • Platform as a Service (PaaS)
  • Supports SMB 3.1/NFS 4.1 protocols
  • No server maintenance, Microsoft handles all maintenance
  • Mountable in Azure VM and Windows Server 2012 and later
  • Can be mounted from on-premises hosts
  • Supports different performance tiers: hot, cold, and high performance
  • Supports NTFS permissions and ACLs
  • Costs vary based on storage performance requirements

• Azure NetApp Files: NetApp Filers as a service backed by Azure Files

  • Platform as a Service (PaaS) using NetApp Filers
  • No server maintenance, Microsoft handles all maintenance
  • Mountable in Azure VM and Windows Server 2012 and later
  • Can be mounted from on-premises hosts
  • Includes Extreme Performance compared to Azure Files options
  • Supports NTFS permissions and ACLs
  • Increased cost compared to Azure Files

• PaaS file shares have unlimited highly-durable data storage.

• PaaS file shares have limits on performance, throughput, and protocol support.

• PaaS file shares are more complex to setup with Active Directory NTFS permissions.

• PaaS file shares can be mounted from most operating systems.

• PaaS file shares integrate with other cloud-bases services such as logging and metrics.

• PaaS file shares are best when the user workloads are also in Azure. Using PaaS file shares reduces the egress data charges and saves on monthly charges.

• PaaS file shares work best for sharing data internally across departments when user workloads are in Azure.

• PaaS file shares do not work as well for sharing files externally because permissions are tied to the Azure AD users.

• When using PaaS file shares, verify that users accessing their data shares from on-premises are receiving acceptable response times.

• Select the lowest performance tier that meets the user's expectations.

• Governance and legal requirements are imposed based on the region hosting the data.

• Cloud-based file services do not support the installation of third-party software such as DLP, AVS or encryption software.

• Backups can be easily configured using Azure Backup.

When should I use Azure NetApp Files vs Azure Files?

This decision is based on what your users consider acceptable. Generally speaking, when you have less than 100 users accessing the file share simultaneously, an Azure Files, Transactional performance level works best. With workloads of between 100 and 2000 users, depending on the frequency of the file updates, consider Azure Files Premium performance level. With workloads over 2000 users, consider using the Azure NetApp Files. To reduce the traffic on the file share, consider using Citrix User Profile Management with profile streaming and large file handling enabled. You can also reduce traffic on the file share by using Microsoft FSLogix containers or Citrix Profile Management Containers.

When should the File Sharing Collaboration solutions be used?

File Sharing and collaboration services are designed to make share files accessible using the HTTPS protocol over the internet. These services allow not only individuals to store and retrieve files from the service, but also support collaboration between departments and even outside entities. They have security built in and provide a single point of access. These solutions are best for storing data securely that must be shared both internally and externally. Though they have been adapted to work as personal storage locations for users, they don’t always work well for storing user profiles.

• ShareFile: Secure file-sharing cloud-based service hosted by ShareFile.

  • File sharing repository accessible over HTTPS/CIFS
  • Light management required around user security
  • Limited maintenance of local StorageZone controllers. ShareFile maintains the rest of the infrastructure
  • Data owners can easily grant permissions to internal and external entities
  • Integrates with Active Directory
  • ShareFile-managed StorageZones are protected by durable cloud storage (in Azure)
  • Customer-managed StorageZones allows use of local customer data centers
  • ShareFile handles all the backups, antivirus, and indexing operations
  • All files stored encrypted with AES-256
  • Integrates with SharePoint and OneDrive
  • Supports mobile access to network shares
  • ShareFile Sync client can synchronize local user data with ShareFile storage
  • Includes document management, workflow management, content collaboration, and e-signing capabilities
  • Costs vary depending on functionality selected

• SharePoint in Microsoft 365: Cloud-based SharePoint service hosted by Microsoft

  • Limited management of user security and access
  • Microsoft maintains all servers
  • File-sharing repository accessible over HTTPS/CIFS
  • Light-weight version of SharePoint server
  • Data owners can easily grant permissions to internal and external entities
  • Integrates with Active Directory
  • Supports mobile access to network shares
  • Includes document management, workflow management, and content collaboration capabilities
  • Costs vary depending on functionality selected
  • Integrates with OneDrive to synchronize content

• OneDrive: OneDrive provides synchronization of user data between a local Windows workstation and a back end data storage location

  • Local agent client installed and configured to synchronize user data with SharePoint or ShareFile
  • Can be configured to automatically backup the Documents, Desktop, and Pictures folders to OneDrive in Microsoft 365
  • Included with Microsoft 365 licenses

Cloud Storage Design Considerations

Cloud storage technologies have changed the landscape of personal data storage expectations. Fortunately, most users now treat the extra latency as an acceptable tradeoff for being able to access their data securely from anywhere. Other design considerations when using these collaborative file shares include:

• Cloud-based file sharing and collaboration services have unlimited data storage.

• With collaborative file sharing services, highly durable data storage is used and backups are included in the cost of the service.

• Cloud-based file sharing and collaboration services do not support the installation of third-party data protection software. You are expecting the vendor to provide the protection against data loss, viruses, and loss of confidentiality.

  • These technologies are preferred when a need exists to share the files externally, such as with other businesses or third parties.
  • Using the ShareFile Sync agent with ShareFile or the OneDrive with SharePoint provides an excellent user data backup solution for their local device files.
  • Collaborative file shares are excellent for remote users that keep a large number of documents locally on their assigned device.
  • When collaborative file shares are used from non-persistent sessions, use the Session Linger setting so data can be synchronized before the session terminates.

• When using SharePoint

  • Keep the top-level parent portals to minimum to improve security, usability, navigation and adoption.
  • Avoid using deep hierarchies with unique permissions to improve performance.
  • Do not bury content or keep stale content as it impacts usability and deters users from using the site.
  • Use standard groups first (Members, Visitors, Owners) followed by AD Groups or SharePoint groups next, and direct user access last.
  • Take advantage of permission inheritance.

• OneDrive clients interoperate with GPOs and support folder redirection.

• OneDrive clients integrate with profile management solutions.

What if my users are accessing their data from both on-premises and in the Azure cloud?

• Collaborative file services work for sharing data internally and externally and also function as user data file repositories

• PaaS file services can be integrated directly with Windows since they can be mounted and accessed like internal file shares

• Augment PaaS file services with profile management solutions that virtualize the file system. This approach reduces the amount of data on the wire and reduces the monthly charges from outbound Azure Files data.

• Using PaaS file services prepares the way for complete cloud adoption while providing an improved experience for users accessing their data out of the cloud

What data methods work best together?

• Different methods are acceptable for different user groups based on their data access requirements

• To reduce costs, avoid combinations that store the same data in multiple locations, for instance do not use Citrix ShareFileSync and Citrix User Profile Manager with Azure Files

• For cloud-based Citrix workloads, combining a profile management solution with cloud-based file service has proven to be a good combination

  • Citrix User Profile Management with large file handing enabled on an Azure Files share
  • Using Microsoft FSLogix with Azure Files
  • Citrix ShareFileSync backed by Citrix ShareFile with Microsoft FSLogix

• For on-premises Citrix workloads, combine a profile management solution with the traditional file server technologies

  • Citrix User Profile Management with large file handing enabled on a traditional file server share
  • Using Microsoft FSLogix with a traditional file server share.

Device Management Considerations

The main challenge with device management is enforcing policies at the device level. Device policies can be enforced through GPOs or through endpoint management software.

For instance, GPOs applied through domain memberships are used to administer the devices by setting policies such as screen saver timeouts to improve security. Azure AD does not support device level management directly. However, GPOs for device management are available when Azure AD is used with an on-premises Active Directory or Azure AD DS.

Both Citrix and Microsoft provide solutions for managing mobile devices that can apply policies to iOS, Android and Windows 10 devices. Citrix provides the Endpoint Management service in Citrix Cloud while Microsoft offers Endpoint Manager, which includes Intune. Windows 10 includes modern features for managing devices and removes the legacy dependencies on Active Directory GPOs. You can choose your method of policy enforcement depending on how extensively device management is used within your organization. Here are the questions that you need to answer about Device Management:

Do I still need GPOs for my devices?

• If all of your user devices are running Windows 10 or later, GPOs may be replaced by Microsoft Intune policies

• If legacy applications require settings that cannot be deployed via Intune, then a traditional Active Directory DS is required

• Carefully review all existing GPOs in place, you may not need them any more

• Citrix VDA hosts still use GPOs

What are the requirements for Citrix Endpoint Management?

• Citrix Endpoint Management requires a Citrix Cloud Connector for directory synchronization

• Citrix Endpoint Management should be set to use the Citrix Identity provider through Secure Hub so Endpoint Management can authenticate directly to Azure AD

• Citrix Endpoint Management integrates with Azure AD as long as the users are not using local accounts

• Citrix Endpoint Management integrates with Microsoft Endpoint Manager so you can wrap your own line of business (LOB) applications with Intune and provide a micro-VPN

• Citrix Endpoint Management requires that enrollment invitations use LDAP authentication instead of Azure AD

• Citrix Endpoint Management requires user names, email addresses, and groups match between Active Directory DS and Azure AD

• Citrix Endpoint Management requires a Citrix Gateway (v12.1 or later) installed at your resource location for micro-VPN access, mobile productivity apps, or integration with Microsoft Endpoint Manager

• Citrix Endpoint Management requires a local StorageZone controller to support Citrix Files with private data storage

• Citrix Endpoint Management needs certificate-based authentication configured on the Citrix Gateway to provide a single sign-on experience

• Citrix Endpoint Management requires enrollment profiles for Android Enterprise and ΓÇ£Allow users to decline device managementΓÇ¥ set to off.

• Citrix Endpoint Management supports using the Citrix Cloud service to authenticate managed devices on the following platforms:

  • Apple iOS
  • Android BYOD
  • Android Legacy Device Administration mode

• To manage the Citrix Cloud Endpoint Management Service use the console found under My Services

What are the requirements for Microsoft Endpoint Manager?

• Microsoft Endpoint Manager does support both cloud and on-premises deployments

• Microsoft Intune requires Azure AD Global Administrator or Intune Service Administrator permissions to deploy

• Microsoft Intune does not have a hierarchy for applying settings to determine if one policy clearly has precedence. If two policies exist for the same setting within Intune, then a conflict results.

• Microsoft Endpoint Manager supports iOS, Android, Windows Mobile, and Windows 10

• Microsoft Intune can be licensed in one of three ways:

  • Standalone Azure service
  • Enterprise Mobility + Security (EMS)
  • Microsoft 365

• Azure AD Premium licenses are required for the following features:

  • Some AD join operations
  • Windows AutoPilot
  • MFA device settings
  • Conditional access
  • Dynamic device groups

• Manage Microsoft Intune through the Azure Intune console.

Network Level Considerations

In this section you will find a list of items to help you focus on the appropriate design decisions specific to the Citrix App Delivery Controller (ADC).

Licensing Considerations

Citrix Application Delivery Controller (ADC) on Microsoft Azure is a L4-L7 virtual networking appliance. The Citrix ADC provides organizations secure access to applications and assets deployed in Azure. Citrix ADC on Azure provides a foundation for the network infrastructure without any physical limitations. Citrix ADC on Azure comes in two models: VPX (virtualized) or CPX (containerized). Citrix also provides an Ingress Controller based on Kubernetes Ingress. The Ingress Controller can automatically configure the VPX and CPX models based on a defined configuration.

To ensure enterprise-grade reliability and security, Citrix ADC uses advanced traffic management, observability, and comprehensive security features. Selecting the correct model and feature set is beneficial when it comes to planning your architecture. Some questions to answer about model selection and features might include the following:

What use cases are best for the VPX Virtual Appliance?

• You use virtual appliances on your hypervisor instead of physical appliances

• You need high SSL performance with no hardware acceleration

• You have a hybrid cloud scenario

• You need load-balancing on-premises and in public or private clouds

• You are replacing MPX or other hardware load-balancers with virtual appliances

• You need a multitenant infrastructure with full isolation

What use cases are best for the CPX Containerized Appliance?

• You need to support Kubernetes or OpenShift containerized applications

• You require load-balancing for microservices traffic within a Kubernetes cluster

• You want load-balancing as part of a DevOps application development pipeline

What instance sizes and prerequisites are recommended for the Citrix ADC VPX virtual appliance?

• The compatible networking models with Microsoft Azure are Citrix ADC VPX 10, 200 and 1000. Any Citrix ADC VPX licenses work, including Standard, Advanced, and Premium edition licenses.

• Models VPX1000 and higher require version 13.0 build 76.x or later AND Accelerated networking be enabled to reach the wanted performance level

• VPX virtual appliances can be deployed on any instance type that has two or more Intel VT-X cores and more than 2 GB memory. Currently, Citrix ADC supports only Intel processors with the following instance size recommendations:

  • Standard D2s v4 for VPX10 or VPX200
  • Standard D4s v4 for VPX1000 or VPX3000
  • Standard D8s v4 for VPX5000
  • Standard D16s v4 for VPX10000

Do I need a Citrix Ingress Controller?

• Citrix ADC CPX and Citrix Ingress Controllers are deployed from the Azure Marketplace and used for microservices deployments

• Azure Kubernetes Engine (AKS), supports deploying a Citrix ADC CPX as an Ingress Controller with either basic or advanced (CNI) networking

• Citrix Ingress Controllers are used for microservice communication with a Citrix ADC CPX

• Citrix Ingress Controller can be deployed in a standalone pod as

  • a Tier 1 ADC device to proxy North-South traffic, which supports traffic outside the AKS cluster to microservices inside the cluster
  • a sidecar container to an ADC CPX to load-balance North-South or East-West traffic, which supports microservices traffic inside the AKS cluster

• Citrix ADC CPX Express is a 20 Mbps container-based ADC that can run on a Docker container and supports up to 250 SSL connections simultaneously

• Citrix Ingress Controller is freely licensed and has no usage fees, you only pay for the Azure costs

ADC Licensing

Review your licensing options before you choose a particular deployment model so you are aware of the options up front. In some situations, you can run a Citrix ADC for only the costs of the Azure infrastructure. Some ADC licensing questions might include the following:

Can I use the Citrix ADC VPX as an ICA Proxy without buying a license?

• Citrix ADC in basic mode has the ICAOnly VPN virtual server parameter set to ON and works fully on an unlicensed VPX instance

• Citrix ADC in Smart-Access mode has the ICAOnly VPN virtual server parameter set to OFF and only supports 5 AAA session users on an unlicensed VPX instance

• Apply a Premium license to the Citrix ADC VPX instance to license more than 5 AAA sessions

• Citrix ADC VPX Express version 12.0.56.30 or later does not require a license file

• Citrix ADC CPX Express is a freely licensed CPX, you only pay the associated Azure costs

How is Citrix ADC licensed in the cloud?

• Citrix ADC on Azure is available with pay-as-you-go licensing through the Azure Marketplace subscription or using your own perpetual licenses

• Using your own perpetual license is referred to as Bring Your Own License (BYOL)

• BYOL requires the MyCitrix licensing portal to generate a valid license for Azure

• BYOL is the only licensing model available on Azure if you are not using the Azure Marketplace subscription

• License activation requires access to the public domain internet

Does Citrix ADC support check-in/check-out licensing model under the Citrix Application Delivery Management (ADM) service?

• Citrix ADC supports Check-in/Check-out licensing from Citrix Application Delivery Management (ADM), which has an automated license provisioning system

• Requires Citrix ADC VPX running 12.0 or later

• Requires Citrix ADM running 12.0 or later

• All licenses must be rehosted to Citrix ADM

• When Citrix ADC instances are removed or destroyed, licenses are automatically returned for reuse

Occasionally, the Citrix ADC VPX may come online with a default ADC license unexpectedly. To resolve this issue, do a warm restart before making any configuration changes to the ADC VPX instance to allow the Azure Instance Metadata Service (IMDS) to correct the licensing.

Scalability Considerations

Designing your ADC architecture and planning the deployment are the two key activities for the transition. Selecting the correct features and the best architecture model for your ADC deployments can be both time consuming and challenging. This section provides guidance about the Citrix ADC features and functionality to help you choose the best model.

What types of deployments are available and what are the best practices when deploying that type?

• Use multi-NIC and multi-IP design when you are deploying into production where high-availability requirements for redundancy or security exist

  • Using Citrix Solution Templates in the Azure marketplace is the recommended deployment method
  • Citrix recommends deploying the multi-NIC architecture using the “Citrix ADC” Citrix solution template from the Azure Marketplace\ Choosing this Citrix solution template gives you the following software plan options:
    • Citrix ADC VPX Bring Your Own License
    • Citrix ADC VPX Subscription License
    • Citrix ADC HA (Availability Zone) - SL
    • Citrix ADC HA (Availability Set) BYOL
    • Citrix ADC HA (Availability Set) - SL
    • Citrix ADC HA (Availability Zone) BYOL
    • Citrix ADC FIPS HA (Availability Zone) BYOL
    • Citrix ADC FIPS Standalone BYOL
    • Citrix ADC FIPS HA (Availability Set) BYOL
  • Integrates with Citrix ADM for GSLB (Traffic management) and Licensing
  • Best for the following use cases
    • Isolation of data and management traffic
    • Improved scale and performance of the ADC
    • Where applications require more than 1 Gbps of throughput
    • Web Application Firewall (WAF) deployments

• Use the single-NIC, multi-IP design for production environments with a single subnet or for non-production environments, such as testing

  • With a single NIC, you have 3 IP configurations:

    • ipconfig1 is management
    • ipconfig2 is client-side traffic
    • ipconfig3 is back-end server traffic

  • Ipconfig3 should not have a public IP address associated with it

  • Add IP addresses for all the configurations in the Azure portal first before configuring them in the Citrix ADC

  • Create an untagged VLAN for each data interface on the ADC VPX and bind the primary IP of the NIC. This procedure helps prevent MAC moves and interface changes in Azure from unexpectedly impacting your ADC.

• Use the single-NIC, single-IP for a Citrix ADC in standalone mode.

  • All functions, NSIP, SNIP, and VIP are tied to a single Citrix ADC IP address
  • Configure the resource group, network security groups, and virtual network before you provision the Citrix ADC VPX VM so the network information is available before provisioning
  • Only available in Azure and on Azure stack

• When deploying High Availability using Availability Sets (recommended)

  • The ADC VPX needs an HA independent Network Configuration (INC)
  • The Azure Load Balancer must be configured in Direct Server Return (DSR) mode

• When deploying High Availability using Availability Zones

  • Use the “Citrix ADC” Citrix Solution template in the Azure Marketplace with a software plan where "(Availability Zone)" is included in the name
  • Currently, not all Azure regions support Availability Zones, so check in your region before deploying this Solution template

What are the benefits of using Azure accelerated networking?

• Accelerated networking is not available on all instance types and the VMs must be stopped to before enabling accelerated networking on a NIC

• You must perform all configuration changes from the Citrix ADC VPX PV interface. Use the ADC show interface command to determine which physical interface is bound to PV

• Citrix recommends not performing any operations on the Citrix ADC VPX VF interface. If you must perform operations on the VF interface, Citrix only allows the clear stats or enable, disable, and *reset interface operations. VLAN binding is unavailable.

What methods are available for deploying Citrix ADC?

• Deploy through the Azure Marketplace. The Citrix ADC VPX virtual appliance is available as an image in the Microsoft Azure Marketplace.

• Deploy using the Citrix ADC Azure Resource Manager (ARM) json template available on GitHub.

• Deploy using Citrix ADM service.

Traffic Distribution

With DNS-based autoscaling, DNS is the layer that decides where the traffic is routed. The traffic manager uses DNS to direct the client traffic to the appropriate Citrix ADC instance that is available in the Citrix Application Delivery and Management autoscaling group. Azure traffic manager resolves the FQDN to the VIP address of the Citrix ADC instance.

With Azure Load-Balancer (ALB) as the traffic manager, inbound traffic goes first to the ALB and it decides where the traffic is routed. ALB manages the client traffic and distributes it to Citrix ADC VPX clusters. ALB sends the client traffic to Citrix ADC VPX cluster nodes that are available in the Citrix Application Delivery and Management autoscaling group across availability zones.

With both traffic distribution options, the Citrix Application Delivery and Management triggers the scale-out or scale-in action at the cluster level. When a scale-out is triggered, the registered virtual machines are provisioned and added to the cluster. Similarly, when a scale-in is triggered, the nodes are removed and de-provisioned from the Citrix ADC VPX clusters.

How do you deploy Citrix ADC VPX on Azure with Global Server Load Balancing (GSLB) and use Azure DNS Private Zones?

• When using DNS-based traffic management, each Citrix ADC instance in the Citrix Application Delivery and Management Autoscale group requires a public IP address.

• For DNS-based autoscaling, Application Delivery and Management waits for the specified Time-To-Live (TTL) period. After the TTL expires, it waits for existing connections to drain before initiating node de-provisioning.

• When using ALB-based traffic management, the public IP address is allocated to Azure Load Balancer. Citrix ADC VPX instances do not require a public IP address.

• The Citrix ADC requires either a DNS virtual server or a nameserver configured which is used by the Azure Load balancer for resolution

• For a Hybrid GLSB configuration (multi-cloud/data center)

  • A SNIP address or GLB Site IP address must be configured on each Citrix ADC node for metrics exchange between the nodes
  • The ADNS or ADNS-TCP service must be set up on the Citrix ADC nodes to process the DNS traffic
  • The Azure cloud security groups and firewalls must allow traffic on ports 53 and 3009
  • Support for GSLB Load-balancing solutions other than Citrix ADC is limited
  • Use the Multi-cloud GLB StyleBook for configuration of Global Load Balancing

Autoscale Guidance

An Autoscale group is a group of Citrix ADC instances that load-balance applications as a single entity. The number of instances in the ADC Autoscale group is based on the configured parameters, such as CPU usage. The Azure infrastructure (ALB or Azure traffic manager) sends the client traffic to a Citrix Application Delivery and Management autoscaling group in the availability set. Citrix Application Delivery and Management triggers the scale-out or scale-in action at the cluster level.

What are the requirements for integrating Citrix ADC with Azure Autoscale?

• Using Autoscale with Azure virtual machine scale sets (VMSS) with multi-IP deployments enabled for high-availability minimizes costs. Citrix recommends using Autoscale to reduce the amount of configuration and overhead necessary to monitor the server performance across VNets.

• An Azure Active Directory (AAD) application and service principal with contributor role on the affected resources are required to implement Autoscale

• With auto-scaling an IP set is created on clusters in every availability zone. After which, the domain and instance IP addresses are registered with the Azure traffic manager or ALB. When the application is removed, the domain and instance IP addresses are deregistered from the Azure traffic manager or ALB. Then, the IP set is deleted.

Citrix Application Delivery Management (ADM) Service

The Citrix Application Delivery Management Service (ADM) within Citrix Cloud provides a centralized location to manage your Citrix ADC deployments. These deployments include Azure cloud or on-premises versions of the following: Citrix ADC MPX, Citrix ADC VPX, Citrix ADC SDX, Citrix ADC CPX, Citrix Gateway, and Citrix Secure Web Gateway appliances. Citrix ADM is a cloud-based solution that manages, monitors, and assists with troubleshooting your entire application delivery infrastructure. Citrix ADM includes all the necessary capabilities to deploy, automate, and license Citrix ADC within an easy to navigate cloud-based console.

How does the Citrix ADM service work?

• Deploy through the Azure Marketplace. The Citrix ADC VPX virtual appliance is available as an image in the Microsoft Azure Marketplace.

• Deploy using the Citrix ADC Azure Resource Manager (ARM) json template available on GitHub

• Deploy using Citrix ADM service

StyleBooks

The most complex part of deploying an ADC is configuring it to work with your authentication system and your applications. Citrix offers StyleBooks to help ease the configuration experience. StyleBooks offer a way to simplify the complex task of Citrix ADC configurations. A StyleBook is a pre-configured template that users can use to create or manage Citrix ADC configurations. StyleBooks exist for most of the common applications and configurations. For instance, The SSO Office 365 StyleBook allows you to enable SSO for Microsoft Office 365 through Citrix ADC instances.

What are the Citrix ADM StyleBook applications templates and how do I use them?

• We recommend using StyleBooks for initial configurations if one is available. StyleBooks for Microsoft 365, Skype, Exchange, SharePoint, and ADFS are available

• Microsoft SharePoint StyleBook requires the following:

  • Sharepoint 2016 or later
  • Citrix ADM v12.0 or later
  • Citrix ADC v10.5 or later

• Microsoft SharePoint StyleBook supports the Load Balancing, Content switching, Responder, Rewrite, Compression, and Integrated Caching features of Citrix ADC

• When using SSL with the SharePoint StyleBook, verify that the Rewrite configuration parameter is enabled in the SharePoint Advanced Settings section of the StyleBook

• Citrix recommends you first select Dry Run to view the configuration objects that the StyleBook creates on the target Citrix ADC instances, If acceptable, then go ahead and execute the actual configuration.

Load-balancing with Azure Tag

For Citrix ADC VPX standalone and high-availability instances deployed on the Azure Cloud, now you can create load-balancing service groups associated with an Azure tag. The VPX instance constantly monitors Azure virtual machines (back-end servers) and network interfaces (NICs), or both, with the respective tag and updates the service group accordingly. The VPX instance creates the service group that load balances the back-end servers using tags. Whenever a VM or NIC with the appropriate tag is added or deleted, the ADC detects the change and updates the service group automatically.

How do I configure Load Balancing to use Azure Tags?

• Tags must be assigned to the VM instance or the VM’s NIC

• When using the Azure CLI to propagate tags, the secondary (standby) Citrix ADC must terminate the rain_tags process after a warm restart. This behavior prevents the old information from being used inadvertently

• The ADC VPX needs to be able to reach the tagged IP Address for the back-end server. For a tagged VM, this is the primary IP address, for a tagged NIC, it is the NIC’s IP address. If the VM is on a different VNet, then VNet Peering must be enabled.

• Save all configurations so they persist between VM restarts.

High Availability Considerations

Within the Azure cloud, the Citrix ADC virtual and containerized appliances have reduced feature sets. Some features, such as VLAN tagging, are no longer necessary because Azure performs the functionality at the infrastructure level. Understanding the limitations and the requirements are key to planning your migration. Using GSLB and Azure for the HSM Key Vault have other requirements that you should be aware of.

Azure Key Vault

Citrix ADC integrates with Azure Key Vault and stores its private keys in the Key Vault, which increases the security protection of the keys. Using Azure Key Vault simplifies the storage and management of keys. Azure Key Vault provides a central key management location for all enterprise ADC appliances across both Azure and the on-premises data centers.

Some questions to answer during the planning stages might include the following:

How does the ADC application integrate with Azure Key Vault and what are its limitations?

• Citrix ADC integration with Azure Key Vault requires the use of the TLS 1.3 protocol

• FIPS 140-2 level 2 compliance requires Azure Key Vault Premium pricing tier and the use of hardware security module (HSM) backed keys

• The ADC will access the Key Vault for each SSL handshake

• Access to the Azure Key Vault requires an Azure Enterprise application and service principal

• Citrix ADC use of Azure Key Vault has the following limitations:

  • Azure Key Vault limits the number of concurrent calls and the limits vary by request type and key type
  • Elliptic-curve cryptography (ECC) keys are not supported
  • HDX Enlightened Data Transport (EDT) and Datagram Transport Layer Security (DTLS) protocols cannot be used to communicate with Azure Key Vault
  • Clustering and admin partitions are not supported
  • The Azure application, Azure Key Vault, and HSM certificate-key pair cannot be updated in Azure after adding them to the Citrix ADC appliance
  • HSM certificate bundles are not supported
  • An HSM key cannot be bound to a DTLS virtual server
  • Neither the SSL Service or Online Certificate Status Protocol (OCSP) requests can use a certificate-key pair created with the HSM key
  • No error is generated when an HSM key and certificate mismatch occurs

GSLB

As businesses transition their workloads to the Azure Cloud, they need a hybrid model that allows DNS resolution in a secure manner. The Azure DNS Private Zone service is the key to this transition. With Private DNS zones, businesses can create a hybrid model that allows DNS resolution for both on-premises and Azure-based servers. The Azure servers can be connected to the on-premises data center via an ExpressRoute or VPN tunnel. Citrix ADC provides a seamless way for distributing traffic across both the on-premises and Azure workloads at a global scale. The Global Server Load Balancing (GSLB) feature provides that global scale and relies on the ADNS service within the Citrix ADC console.

This GSLB feature supports business goals including: migrating from on-premises to the Azure cloud, DNS-based failover, and blue-green environment testing. Both Round Robin and Location-based (static proximity) server routing methods are available. GSLB can be used for any service or host resolution, including StoreFront.

What are the requirements and limitations of using Citrix ADC for GSLB across both my on-premises and Azure cloud hybrid deployment?

• The ADNS service is a DNS server that runs on the Citrix ADC appliance. ADNS supports delegation of DNS name spaces, such that the Citrix ADC is the authoritative name server for the zone and all hosts within it

• Support for GSLB Private DNS zones is implemented using Citrix ADC appliances in the Azure cloud running the ADNS service

• Plan to use DNS forwarders for both virtual networks and data center networks

• All DNS queries are routed first to the local DNS forwarder to provide the best user experience

• GSLB DBS Service requires the following:

  • Citrix ADC version 12.0.57 or later and Microsoft Azure Load Balancer instances
  • Citrix ADC GSLB Service Group Feature Enhancements
  • GSLB Service Group entity: Citrix ADC version 12.057 or later
  • DBS feature components must be bound to the GSLB service group

What are the limitations of running Citrix ADC VPX instances on Azure?

• A secure tunnel between Azure and the on-premises data center must exist, typically across an ExpressRoute or VPN connection

• Assign a static Internal IP address to the Citrix ADC virtual machine to avoid issues caused by the IP address changing after a VM deallocation

What data center Citrix ADC functionality is not available in the Azure Citrix ADC?

• High availability does not work if the Public IP (PIP) address is associated with the VPX instance instead of an Azure Load Balancer

• The Azure architecture does not support the following Citrix ADC features:

  • Clustering, unless deployed via the Citrix ADM Autoscale feature
  • IPv6
  • Gratuitous ARP (GARP)
  • L2 Mode (Bridging); however, Transparent virtual servers with MAC rewrite (L2) will work for servers on the same subnet as the ADC’s SNIP
  • Tagged VLAN
  • Dynamic Routing
  • Virtual MAC
  • USIP
  • Jumbo Frames

• Public IP addresses do not support protocols where the port mapping is opened dynamically, such as passive FTP or ALG.