The Citrix Integration for Windows 365 Reference Architecture is intended for Citrix practitioners—including administrators, engineers, architects, partners, and consultants—who design, build, and maintain Citrix platform deployments for enterprises of all sizes.

Overview

Citrix Desktops-as-a-Service (DaaS) enables you to deliver high-performance virtual apps and desktops securely to any device. Managed by a Citrix-hosted service, this solution securely grants end-users access to Windows, Linux, and macOS (limited to desktop) applications and desktops from a central location, regardless of the operating system on their endpoints. The Citrix Integration for Windows 365 combines Citrix DaaS^™ with Windows 365 to provide a high-performance, secure, and easy-to-manage solution. This integration enables organizations to offer a seamless hybrid work experience, supporting a variety of use cases beyond standard Windows 365 features.

The Citrix Integration for Windows 365 Reference Architecture provides an overview of the Windows 365 service, a conceptual architecture of the Citrix DaaS^™ and Windows 365 integration, traffic flow diagrams for the integration, and use cases for the integration.

Windows 365 Overview

Microsoft Windows 365 is a fully managed desktop virtualization SaaS platform hosted in Microsoft Azure. Windows 365 provides dedicated, persistent, turn-key desktops for a flat-rate price. Windows 365 is available in four editions as detailed here:

¹ This edition does not apply to the Citrix DaaS + Windows 365 Reference Architecture. Any references to Windows 365 from now on will only pertain to the Enterprise or Frontline Dedicated editions.

² Please note that these editions are currently not supported by Citrix.

Windows 365 supports four identity and two device join types:

Microsoft Intune manages provisioning, software installs, updates, patching, and policy enforcement for all Cloud PCs.

Users connect to Cloud PCs over the Remote Desktop Protocol (RDP) through the Windows 365 Gateway, which is provided by the Azure Virtual Desktop (AVD) service. The AVD service also provides connection brokering and web client service.

Citrix Integration for Windows 365 Conceptual Architecture

The conceptual architecture for integrating Citrix DaaS with Windows 365 brings together Citrix’s high-performance application and desktop delivery capabilities with Microsoft’s Windows 365 platform, all under a unified identity and management framework.

 The Citrix Integration for Windows 365 supports both Microsoft Entra joined and Microsoft Entra Hybrid joined Cloud PCs. The supported architectures are illustrated at a high level here.

Microsoft Entra Joined Devices

Following the standard Citrix architecture layer method, the conceptual architecture for Microsoft Entra joined devices consists of five layers.

User Layer

The User Layer includes Citrix Workspace™ app endpoints on all supported devices, such as Windows, Linux, macOS, iOS, Android, and browser-based clients. Users access a single Workspace interface to view both Citrix-delivered applications and desktops, as well as access their Windows 365 Cloud PCs. Microsoft Intune handles endpoint management, configuration, application deployment, and compliance reporting, ensuring that devices meet security standards. Citrix real-time deviceTrust checks verify a device's compliance and context (such as location, security status, or the presence of unauthorized USB devices) to enforce granular, contextual access policies.

Access Layer

In the Microsoft Entra Joined architecture, Citrix Workspace and the Citrix Gateway Service provide the Access Layer. All users authenticate with Citrix Workspace, with Microsoft Entra ID set as the identity provider (IdP). At the same time, the Citrix Gateway service provides VPN-less access to their Windows 365 Cloud PCs alongside other Citrix desktop and application resources (virtual desktops, published applications, Web/SaaS applications). Connections to Windows 365 Cloud PCs use connector-less connections with the Citrix Rendezvous protocol, so Citrix Cloud^™ Connectors are not required within the Windows 365 Resource Locations.

Control Layer

In the Control Layer, Citrix DaaS orchestrates brokering, policy enforcement, resource aggregation, and monitoring for the Windows 365 Cloud PCs. The Citrix Windows 365 Connector is enabled in Intune, allowing a connection between the Windows 365 service and Citrix Cloud. The Citrix DaaS VDA Upgrade service is used to upgrade the Citrix Virtual Delivery Agent on each Cloud PC. From the user’s perspective, all resources, whether hosted in Citrix infrastructure or delivered via Windows 365, appear side by side in a single workspace. The Windows 365 platform is akin to other virtual desktop hosting platforms like Azure, AWS, and on-premises hypervisors, and is viewed by Citrix DaaS as a Resource Location.

Resource Layer

In the Resource Layer, the Microsoft tenant provides the core Windows 365 Cloud PCs. When a Windows 365 Cloud PC is launched from Citrix Workspace™, the user is seamlessly directed through the Citrix Gateway service and optimized with HDX™. This layered approach ensures that each platform is used for what it does best—Citrix for rich, virtualized applications and complex, multi-session environments, and Windows 365 for personal, always-available cloud desktops—while still delivering a unified and secure user experience.

Observability Layer

The Observability Layer includes all other layers, providing end-to-end visibility, diagnostics, and user experience analytics. Citrix DaaS Monitor offers detailed insights into Citrix sessions, logon performance, HDX™ bandwidth, and application responsiveness. This enables administrators to identify issues in real-time, analyze trends, and enhance performance across the entire Citrix environment. For endpoint-level telemetry, uberAgent® is installed on Workspace app endpoints to collect comprehensive performance and UX metrics, such as logon duration breakdowns, application load times, and resource usage—covering both Citrix virtual sessions and local workloads. These tools create a unified observability framework that links Citrix-hosted resources, Windows 365 Cloud PCs, and endpoint devices, supporting a proactive approach to troubleshooting and performance improvement.

Microsoft Entra hybrid joined Devices

Following the standard Citrix architecture layer method, the conceptual architecture for Microsoft Entra hybrid joined devices consists of the same five layers noted above. However, with the Hybrid Entra ID Join architecture, new options are available for the Access Layer, as multiple on-premises components are introduced.

Citrix Cloud Access Layer

Changes to the Microsoft Entra hybrid joined architecture using the Citrix Cloud Access Layer include:

• Within the Control Layer, an on-premises site has been added. This site includes Microsoft Entra ID Connect, which enables synchronization between Active Directory and Entra ID. Cloud Connectors are also deployed, enabling Active Directory integration with Citrix and Windows 365 Cloud PCs. Finally, Citrix Federated Authentication Service (FAS) servers are set up to provide Single Sign-On (SSO) access to Windows 365 Cloud PCs.

• Citrix Workspace and Citrix Gateway continue to manage user authentication; however, in the Hybrid Entra ID join model, any of the supported identity platforms by Citrix Workspace can be used. In this architecture, Hybrid Entra ID authentication is employed.

On-Premises Access Layer

Changes to the Microsoft Entra hybrid joined architecture using an on-premises Access Layer include:

• Within the Control Layer, an on-premises location has been added. This location includes Microsoft Entra ID Connect, which enables synchronization between Active Directory and Entra ID. Cloud Connectors are also deployed, enabling Active Directory integration with Citrix and Windows 365 Cloud PCs. By default, the Windows 365 Cloud PCs are connector-less, and the cloud connectors are only needed to connect Citrix Cloud to the AD domain. However, you can choose to have the VDAs register via cloud connectors if required. More details can be found in the product documentation. Lastly, Citrix Federated Authentication Service (FAS) is implemented to provide Single Sign-On (SSO) access to the Windows 365 Cloud PCs.

• The Access Layer has been moved to the on-premises location, and Citrix StoreFront and NetScaler Gateway are now used for user access, authentication, and enumeration of the Windows 365 Cloud PCs.

Citrix Integration for Windows 365 Configuration

Configuring the integration between Citrix DaaS and Windows 365 involves several components that administrators should be familiar with, including Citrix Cloud^™ and the Citrix DaaS service, as well as the Microsoft Windows 365 service and Windows 365 Cloud PCs.

Citrix Cloud handles the integration of the Citrix DaaS and Windows 365 services by registering and assigning Cloud PC licenses within Citrix DaaS. A Cloud PC registration token is generated to establish trust between the Cloud PCs and the Citrix Cloud services.

Microsoft Entra ID manages user and Cloud PC identities. It also enables authentication and authorization between Windows 365 and Citrix Cloud services.

The process of integrating Citrix DaaS and Windows 365 involves several workflows that Citrix administrators must initiate, as well as background service-related workflows.

Windows 365 Connect Flow

A one-time operation, Windows 365 Connect Flow provides consent for Citrix Cloud to obtain the necessary permissions. Following the Azure consent framework, an Azure application is registered within the customer's Entra ID tenant. A Global Administrator must grant these permissions, as Citrix Cloud will perform all tasks on behalf of administrators. When consent is granted, a new service principal is created using the Citrix application template with the required permissions. Citrix Cloud is granted an access token on behalf of the customer's tenant and is used to communicate with the Windows 365 service. Once consent is complete, Citrix Cloud stores the connection information in a database that includes the Windows 365 tenant ID. It lets the Windows 365 service know that a new Citrix customer is connected to the tenant. The Citrix Integration for Windows 365 supports connecting to multiple Entra ID tenants.

License Assignment Flow

The License Assignment Flow begins when the Citrix administrator assigns a Citrix license to a Windows 365 user or group. Citrix Cloud provides a registration token begins the Cloud PC registration process, which includes installation of the Citrix Virtual Delivery Agent (VDA), Cloud PC registration with Citrix DaaS, creation of the Machine Catalog and Delivery Group, adding the Cloud PC to the Machine Catalog and Delivery Group, assign the user to the Cloud PC, and create a Citrix policy enabling the rendezvous protocol for the Delivery Group.

It is worth noting that the Citrix DaaS Windows 365 license assignment is per-user rather than per-machine.

License Removal Flow

The License Removal Flow begins when a Citrix administrator removes a license assignment from a user or group. As a result, each Cloud PC assigned to that user is removed from Citrix DaaS, and Citrix Cloud notifies the Windows 365 service of the license removal, triggering the uninstallation of the Citrix VDA. After the VDA is uninstalled, the Cloud PC can be accessed via the Windows 365 portal or the Windows App using RDP.

Windows 365 Disconnect Flow

Citrix administrators initiate the Windows 365 Disconnect Flow to completely disconnect their Citrix Cloud tenant from the Windows 365 service. This is only possible when all Citrix licenses have been unassigned from users or groups. When executed, Citrix Cloud notifies the Windows 365 service to offboard the customer, and once confirmed, Citrix Cloud will remove the connection information from the Citrix customer.

Citrix Workspace URL Update

Running every hour, this task monitors any changes to the Workspace URL used for Windows 365 Cloud PC access. This ensures that the Windows 365 portal has the correct URL for Citrix Workspace if users use this method to access their Cloud PC.

Ghost Cloud PC Removal

This daily task checks and verifies that each Cloud PC in each Cloud PC Machine Catalog still exists within the Windows 365 service and removes the Cloud PC from the catalog if it no longer exists. When machines are reprovisioned, a real-time cleanup of the old machine is performed before the new machine is created.

Citrix Integration for Windows 365 Authentication

The Citrix Integration for Windows 365 supports Windows 365 deployments with Entra ID-joined and Entra Hybrid-joined Cloud PCs. For user authentication, this table provides details on the supported identity providers for both Entra ID join and Hybrid Entra ID join machine identity types.

√: Supported  X: Not Supported

Microsoft Entra Join with Citrix Workspace

When using Microsoft Entra join with the Citrix Integration for Windows 365, end users' Entra IDs are used for authentication into the Cloud PC.

In the diagram above, you see a typical Citrix Integration for a Windows 365 deployment with Microsoft Entra joined Cloud PCs. The user authentication flow to the Cloud PC is as follows:

1. The end user opens the Citrix Workspace app or a web browser and navigates to their organization's Citrix Workspace URL.

2. Citrix Workspace, configured to use Microsoft Entra ID as the identity provider, redirects the user's browser to the Microsoft login page (login.microsoftonline.com).

3. The user enters their Entra ID username and password.

4. Microsoft Entra ID processes the credentials, performs the authentication, and applies any relevant Conditional Access policies.

5. Upon successful authentication, Entra ID issues an authentication token (e.g., SAML assertion or OpenID Connect ID token) back to Citrix Workspace.

6. Citrix Workspace validates the received token and communicates with Citrix DaaS to enumerate the list of available Windows 365 Cloud PCs configured for the user.

7. The user selects the desired Windows 365 Cloud PC from the Workspace interface.

8. The Citrix HDX protocol initiates the connection to the selected Windows 365 Cloud PC.

9. The user is presented with their Windows 365 Cloud PC desktop login screen.

Microsoft Entra Hybrid Join with Citrix Workspace and Gateway Service

When using the Microsoft Entra Hybrid join option to integrate Citrix with Windows 365, all identity providers supported in Citrix Workspace are available. In this case, authentication into the Cloud PC is performed using Active Directory credentials rather than Entra ID credentials. Although Cloud PCs are assigned to a user's Entra ID by default, the machines are assigned to the corresponding Active Directory user so that connections can be brokered for those users.   

In this diagram, you see a typical Citrix Integration for a Windows 365 deployment with Microsoft Entra Hybrid joined Cloud PCs. As shown, Entra ID Connect is required to synchronize your Active Directory and Entra ID Directory domains, and it is essential for the hybrid join model. Additionally, a VPN or ExpressRoute is set up to provide connectivity between the Azure virtual networks (vNets) and the Citrix Cloud Resource location, allowing the Cloud PCs to join the Active Directory domain. A Citrix Cloud Connector^™ is necessary for Citrix Cloud to connect to the Active Directory domain.

1. The user authentication flow to the Cloud PCs is as follows:

2. Microsoft Entra ID Connect synchronizes the on-premises Active Directory and Microsoft Entra ID tenant.

3. The end user opens the Citrix Workspace app or a web browser and navigates to their organization's Citrix Workspace URL.

4. In this case, Citrix Workspace is configured to use Active Directory as the identity provider, so users enter their Active Directory username and password.

5. The credentials are securely transmitted via the established secure tunnel provided by the Cloud Connectors to the Active Directory domain controller to process the credentials and perform the authentication.

6. Upon successful authentication, the successful result is sent back to Cloud Connector, which relays the result to Citrix Workspace.

7. Citrix Workspace establishes the user's session and presents a list of available Windows 365 Cloud PCs configured for the user.

8. The user selects the desired Windows 365 Cloud PC from the Workspace interface.

9. The Citrix HDX protocol initiates the connection to the selected Windows 365 Cloud PC.

10. The user is presented with their Windows 365 Cloud PC desktop via HDX, ready for use.

Microsoft Entra Hybrid Join with Citrix Workspace (3^rd Party IdP)

For environments using an identity provider (IdP) other than Active Directory, the deployment described above still applies. However, in this case, the Citrix Federated Authentication Service (FAS) provides Single Sign-On (SSO) for sessions. In the scenario below, Okta is used as the primary IdP for Citrix Workspace.

Okta's role in this scenario is to serve as the primary authentication provider for Citrix Workspace. When a user logs into Citrix Workspace, Okta verifies their credentials and issues a token to Citrix Workspace.

However, Okta does not replace the underlying user directory that Windows 365 Cloud PC and its management use. The user account that the Cloud PC recognizes and the identity that FAS uses to issue a certificate for desktop logon still reside in on-premises Active Directory, which is then synced to Entra ID.

1. Microsoft Entra ID Connect synchronizes the on-premises Active Directory and Microsoft Entra ID tenant.

2. The end user opens the Citrix Workspace app or a web browser and navigates to their organization's Citrix Workspace URL.

3. Citrix Workspace is configured in Citrix Cloud to use Okta as its primary identity provider, and it redirects the user to the Okta login page. The user enters their Okta username and password.

4. Okta processes the credentials, performs the authentication against its user store (which might be Okta's own directory, an integrated Active Directory, or another source), and applies any relevant Okta policies.

5. Upon successful authentication, Okta issues an authentication token back to Citrix Workspace.

6. Citrix Workspace validates the received token from Okta, establishes the user's session, and presents the list of available Windows 365 Cloud PCs that the user is entitled to access.

7. The user selects the desired Windows 365 Cloud PC from the Workspace interface.

8. The Citrix HDX protocol initiates the connection to the selected Windows 365 Cloud PC.

9. In the background, Citrix Federated Authentication Service (FAS) leverages the user's successful authentication to Citrix Workspace to issue a short-lived, digitally signed certificate specifically for that user.

10. The Citrix Virtual Delivery Agent (VDA) on the Hybrid Entra ID joined Windows 365 Cloud PC uses this certificate provided by FAS to perform a "certificate logon" to the on-premises Active Directory Domain Services (AD DS).

11. This process automatically logs the user into their Cloud PC's Windows desktop session, eliminating the need for them to re-enter their credentials.

Microsoft Entra Hybrid Join with Citrix StoreFront and NetScaler Gateway

1. Microsoft Entra ID Connect synchronizes the on-premises Active Directory and Microsoft Entra ID tenant.

2. The end user opens the Citrix Workspace app or a web browser and navigates to their organization’s NetScaler Gateway URL.

3. The user connects to the NetScaler Gateway, which is configured to use Active Directory in this case as the identity provider, so users enter their Active Directory username and password.

4. Upon successful authentication, the NetScaler Gateway establishes a secure SSL/TLS tunnel (often DTLS for ICA traffic) from the user's device into your network. This tunnel secures all subsequent communication.

5. The NetScaler Gateway forwards the authenticated user's session information to the internal Citrix StoreFront server. This is often configured for pass-through authentication, meaning StoreFront recognizes the user as already authenticated by the Gateway and does not prompt for credentials again.

6. Upon successful authentication, the successful result is sent back to Cloud Connector, which relays the result to Citrix DaaS.

7. The Citrix DaaS control plane queries your on-premises Active Directory (via the Cloud Connectors) to determine the user's group memberships and entitlements. It then compiles a list of virtual applications and desktops that the user is authorized to access. This list is sent back to StoreFront (via Cloud Connectors). StoreFront then presents the users with the list of available resources, including Cloud PCs

8. The user selects the desired Windows 365 Cloud PC from the Workspace interface.

9. The Citrix HDX protocol initiates the connection to the selected Windows 365 Cloud PC.

10. The user is presented with their Windows 365 Cloud PC desktop via HDX, ready for use.

11. In the background, Citrix Federated Authentication Service (FAS) leverages the user's successful authentication to Citrix Workspace to issue a short-lived, digitally signed certificate specifically for that user.

12. The Citrix Virtual Delivery Agent (VDA) on the Hybrid Entra ID joined Windows 365 Cloud PC uses this certificate provided by FAS to perform a "certificate logon" to the on-premises Active Directory Domain Services (AD DS).

13. This process automatically logs the user into their Cloud PC's Windows desktop session, eliminating the need for them to re-enter their credentials.

Citrix + Windows 365 Business Continuity

The right approach to business continuity during a Citrix Cloud outage depends on how your Cloud PCs are joined:

For Entra ID joined Cloud PCs using connector-less VDA registration, users can leverage Service Continuity. Service Continuity caches Workspace subscription data locally in the Citrix Workspace app, allowing users to reconnect to existing sessions during a Citrix Cloud outage without any Cloud Connector involvement.

For Entra hybrid joined Cloud PCs, Service Continuity is not available. To provide resilience for your Cloud PCs, use Local Host Cache (LHC), which runs on the Cloud Connector and allows session brokering to continue locally during a Citrix Cloud outage. If you are deploying Entra hybrid-joined Cloud PCs and require brokering continuity during a Citrix Cloud outage, you must deploy Cloud Connectors in the Windows 365 resource location and configure VDA registration through them. Cloud Connectors in other resource locations provide LHC coverage only for machines registered in those locations. They do not provide LHC coverage for Cloud PCs in the Windows 365 resource location.

Monitoring Citrix + Windows 365

When integrating Citrix with Windows 365, the user experience (UX) relies on several control planes and hops, including Microsoft Entra ID, Microsoft Intune, Citrix Gateway service, Citrix Workspace, and the endpoint. Comprehensive, correlated monitoring is therefore crucial to:

• Detect and triage issues in real time (enumeration, brokering, logon phases, session failures) and verify impact across users/sites. Citrix DaaS Monitor offers a live troubleshooting dashboard for DaaS workloads, featuring failure surfacing and site health panels.

• Quantify UX and identify the root cause (such as network latency, policy issues, codec choice, or bandwidth constraints) at the HDX session layer; Installing Citrix uberAgent on the Windows 365 Cloud PC provides protocol-level and session configuration metrics that explain why a session feels fast or slow.

• Close visibility gaps caused by spanning Microsoft’s Cloud PC fabric and Citrix services; The Citrix Integration for Windows 365 connects Citrix Cloud with Windows 365 to implement HDX technologies and policies—observability must cover both sides.

What To Monitor

References

Tech Brief: Citrix for Windows 365

POC Guide: Citrix for Windows 365

POC Guide:  Integrating Windows 365 with Citrix Session Recording

Microsoft Windows 365 Architecture

Microsoft Windows 365 Service Description

Citrix customers have relied on Citrix Provisioning (PVS) technology for many years to provide best-in-class image management solutions for Citrix environments. Many of these customers have added Azure workloads to their Citrix solutions and want to integrate Citrix Provisioning on Azure with their on-premises solutions.  Many other Citrix customers who have not used Citrix Provisioning on-premises before are interested in what Citrix Provisioning on Azure will offer regarding operational agility and cost optimization.  This reference architecture is intended to provide both of these groups a blueprint for how and why to implement Citrix Provisioning on Azure.

Objectives of this document

This reference architecture provides a blueprint for the design of a Citrix Provisioning on Azure solution based on several overriding tenets or “Design Pillars”:

• Resiliency
• Security
• Cost optimization
• Operational Excellence
• Performance Efficiency

Citrix Provisioning on Azure can be architected in many ways. In the following sections, an earnest attempt will be made to define an example architecture based on a set of requirements and assumptions while explaining the reasons major decisions were made and where customers can make different design decisions based on differing requirements.

This document will provide information pertinent to deploying Citrix Provisioning on Azure.  It will not provide exhaustive documentation on all aspects of Citrix Provisioning, as there is a large base of information already documented for this purpose.  The following list of documentation should also be referenced for those new to Citrix Provisioning:

• Tech Brief: Citrix Provisioning
• Reference Architecture: Image Management
• Reference Architecture: Citrix DaaS – Azure
• Create Citrix Provisioning catalogs in Citrix Studio
• Official Documentation - Citrix Provisioning on Microsoft Azure
• Migrate workloads between Resource Locations using Image Portability Service

Advantages of the Citrix Provisioning Architecture

Customers love to see Citrix Provisioning in their Citrix Solutions for many reasons.  The highlights of this architecture include:

• Ability to manage non-persistent images quickly and efficiently
  • Full Image Versioning
  • Simple Image Replication
  • Near instant rollout and rollback
• Very Highly Available solution based on Active/Active Server Technology
• Fully automatable using PowerShell
• High-Performance Storage solution using Cache in RAM technology
• Easy to integrate into a hybrid cloud model using Citrix App Layering or the Citrix Image Portability Service
• Advantages of Citrix Provisioning within Azure
  • Azure provides standard, well-tested, high-performance networking on which to build the Citrix Provisioning infrastructure
  • Citrix Provisioning requires less storage than most other solutions
  • Citrix Provisioning provides nearly instant scaling of resources when used in Azure

Citrix Provisioning on Azure Design Considerations

As stated above, this architecture will be developed based on the above Design Pillars.  For an overview of how the architecture maps to these pillars, you can jump to the Design Pillars section of the document.

To illustrate design options and recommended solutions, two different environments will be discussed: one at scale and one of more modest size.

VDI at Scale

The first design to be addressed is VDI at scale.  This architecture is based on PODs and can be used for solutions from 5K desktops to 100K desktops or more.

The components of the solution include:

• One or more Azure Tenants
• An Azure Landing Zone or Hub Zone for Citrix workloads
• One Azure Subscription per 5K VDI Desktops
• At least two Azure Availability Zones per Azure Region
• One or more Azure vNets per Azure Subscription
• vNet Peering from workload vNets to Landing Zone/Hub vNets
• One Citrix Provisioning Farm per Azure Region
• One Citrix Provisioning Site per Availability Zone
• One Citrix Provisioning server for every 1500-2000 Targets
• N+1 Citrix Provisioning Servers Per Site
• Appropriate naming conventions
• One Citrix Cloud Resource Location per 10K Desktops per Azure Region

The following diagram illustrates the major components of the Citrix Provisioning on Azure architecture for VDI at scale.

High-Level Citrix Provisioning on Azure Architecture for VDI at Scale

In the following sections, each component of the architecture will be discussed.

Azure Tenants and Subscriptions

While using a multi-tenant Azure design for Citrix within Azure is possible, customers usually choose to manage all Citrix components within a single Azure tenant. From a Citrix Provisioning perspective, this is not a requirement.

For a large-scale deployment, multiple Azure subscriptions will be required. Typically, a separate subscription is used for the Azure Landing Zone, which can be considered the hub in a hub and spoke topology where the Citrix workload subscriptions are deployed as spokes. Currently, Citrix supports up to 5K targets per Azure subscription.  These limits are based on several factors, but power management is usually the limiting factor based on the number of API calls that can be made into the Azure infrastructure at the subscription level.   Citrix is constantly working with Microsoft to increase these limits, so always check the DaaS limits page to see if this has changed.

For customers that need to scale past 10K VDI desktops per Azure Region, multiple Resource Locations would be provisioned in the same region, conforming to the POD design shown here.

Landing Zone/Hub Site

The Landing Zone is where shared components will be deployed in this solution.  This includes services like:

• Citrix Cloud Connectors
• Active Directory Domain Controllers
• File Services
• Citrix Provisioning Servers

The Landing Zone will be provisioned with one or more vNets.  The critical aspect of this to the Citrix design is that there should be direct vNet peering between the Citrix worker vNets and the vNets used for Citrix Provisioning servers as the Citrix Provisioning servers need direct high bandwidth low latency network connections to the targets (VDAs)

Networks

The virtual networks used for Citrix workloads could be separated by Availability Zone (AZ) or stretched between Availability Zones.  In this architecture, they are stretched to make management more effortless.  Since the design is for 5K desktops per subscription, a /19 network can be used as it can accommodate up to 8K addresses. For customers that plan to grow very large in the future, and based on the fact that Citrix keeps increasing the number of desktops that can be managed in a single subscription, it may make sense to provision a /17 vNet that can accommodate up to 32K addresses so that the solution can support future growth without being rebuilt.

Within the vNet in each Availability Zone, a /20 subnet will be created to accommodate the 2.5K desktops deployed in that zone.

Citrix Resource Locations

A Citrix Resource Location defines the boundaries of a Zone in Citrix Cloud.  It is defined by a set of Cloud Connectors that act as gateways for information between the “Resource Locations” and Citrix Cloud, including the communications between VDAs and Delivery Controllers, StoreFront servers and Delivery Controllers, Citrix Cloud, and Active Directory, etc.  Cloud Connectors within the same Resource Location share the same scope of resources used to provide Local Host Cache functionality, which offers high availability services if the cloud connectors are disconnected from the Citrix Cloud Resources.  It is the Local Host Cache services that govern the limit of the number of desktops that can be provisioned within a single Resource Location.  This number is 10K VDI desktops or 1K virtual app servers with up to 25K multisession sessions.

Therefore, for every 10K desktops required in an Azure Region one Citrix Cloud Resource Location will be provisioned.  For more information on these limits, see the local host cache documentation.

The number of Cloud Connectors to provision in a Resource Location will depend on several factors, including:

• Access Method
  • Gateway/StoreFront
  • Gateway Service/Workspace
• Connection Method
  • Cloud Connectors
  • Rendezvous
• Risk Tolerance

Three Cloud Connectors per Resource Location are recommended, along with using Rendezvous for Gateway Service customers.  Three connectors allow for a failed upgrade without losing high availability for the Resource Location. If Gateway Service will be used and Rendezvous will not be used, then more Cloud Connectors may be required to handle the load of routing the HDX traffic from the Gateway Service to the VDAs.

Citrix Provisioning Farms, Sites and Servers

One of the primary considerations for this architecture is the Citrix Provisioning Farm and Site architecture.  While it is technically possible in Azure to stream targets across availability zones, here, we do not want to do that because, of course, it is likely that Availability Zones could be disconnected from each other, and high availability is one of our core tenets.  Citrix Provisioning will be designed with a Farm per Azure Region with separate sites per Availability Zone to keep the traffic local to each Availability Zone. By having one Farm in the Primary region and one Farm in the DR Region, configurations can be updated even if one of the Farm databases is down.  Of course, Citrix Provisioning will also be configured to “Enable offline database support” so that targets will keep working even if the database is down.

Citrix Provisioning is fairly flexible when designing the number of servers.  The most significant load on Citrix Provisioning servers is when many targets are booted at the same time.  If there is more load, some targets must wait while others boot.  Therefore, the tradeoff is between the cost of servers and the time it takes to boot targets at scale.  A good rule of thumb is to limit Citrix Provisioning server to between 1500 and 2000 targets per server.  In this design, we have included four servers for 5K desktops.  This allows for the typical case where there will be 1,250 targets per server; if there is a server failure, there will be 1,667 targets per server. 

The servers here will be configured with four cores and 32 GB of RAM.  The amount of RAM required is based on the number of vDisks that will be streamed.  The general rule of thumb is to provision 2 GB for the OS and 2 GB for every vDisk to be streamed.  That means that an Azure instance that is 4x32 can handle streaming up to 15 vDisks. This is more than enough for most Citrix Provisioning on Azure deployments.

Of course, in Azure, choosing the best instance type for your Citrix Provisioning server is critical. It is essential to look at not only CPU and memory but, more importantly, network and disk throughput. Citrix Provisioning servers on Azure require premium storage, so only instance types that support it should be considered.

If we compare the following two instance types, they have 4x16 instances and the same pricing, but the D4s_v5 has better network throughput.

If more memory is required, Azure has memory-optimized images, and the same is true here, where the newer instance type has more network throughput.

It’s worth noting that the pricing increases linearly in Azure as you go up in CPU and memory.  In our examples, the 8x32 instances cost exactly double what the 4x16 instances cost.

To decide on an instance type, determine how much memory your Citrix Provisioning servers will need based on the number of vDisks to stream. Then, find an appropriate instance with the best network performance. The following table lists memory requirements based on the number of vDisks streamed. These instances were chosen based on memory, price, and network throughput.

* Pricing based on 3-year reserved pricing in East US

Azure pricing can change based on the day and the Region.  All of these instances would be suitable for Citrix Provisioning servers, and checking for pricing and availability should be mandatory when provisioning your Citrix Provisioning servers. Pricing should be checked periodically throughout the life of the Citrix Provisioning services.

Storage costs are another factor to consider when designing the number of servers based on CPU and Memory.  Storage costs will depend on the type of storage used and the number of stores required.  See the storage section for more details.

Citrix Provisioning vDisk Stores

Citrix Provisioning supports vDisk storage on a local file system or a network share.  In this design, we recommend using disk stores deployed on local SSD disks attached to each Citrix Provisioning server.  This provides a very high level of redundancy because each server has its own copy of the vDisk store.  If a network share is used, then that share becomes a single point of failure for the Citrix Provisioning site.  While Citrix Provisioning supports Azure Files and Azure Netapp Files, both of which have good to very good redundancy, a vDisk could still be corrupted or deleted by accident, causing an outage.  This is less likely to be a significant issue if replicated local storage is used for the Citrix Provisioning stores.  If shared storage is used, it should be split into availability zones so there is at least redundancy between AZs in the design.

Citrix Provisioning vDisk Replication

In the section above, using local vDisk stores was recommended.  If that choice is followed, then the vDisks in the stores added to each Citrix Provisioning server must be synchronized.  Many articles and processes have been developed to synchronize vDisks over the years.  This includes using DFS commercial replication solutions and any number of scripted solutions that have primarily relied on Microsoft's robocopy utility to synchronize the stores.  The vDisk Replication Utility documented here can replicate vDisks between servers within the same site or farm and across sites and farms.  It will also add the vDisk into Citrix Provisioning in a different Farm.  That said, the various solutions can replicate vDisks between local Stores.

Smaller Deployment Site Options

What if only one to two thousand VDAs are required?

There are several options here.  The first option is to follow the same design as the large-scale design above, just lowering the number of Citrix Provisioning servers required for each Zone.

This provides a high degree of redundancy but with a higher cost as more Citrix Provisioning servers are provisioned than required for the number of VDAs to maintain high availability within each Availability Zone.

Another option is to use a single Citrix Provisioning site across Availability Zones with a single Citrix Provisioning server per Availability Zone. This option trades off the localization of network access against the cost of additional Citrix Provisioning servers.

The cost of the Citrix Provisioning servers when using an E4ds_v5 instance with 2TB of Standard SSD storage is approximately $340 per month, so adding two servers would add $680 per month to the solution.

Supporting a single Citrix Provisioning site will also be easier operationally than supporting multiple, and each organization must decide how much redundancy is required to meet its high availability needs.

Citrix Provisioning Database Options

There are many options for providing SQL services to Citrix Provisioning within Azure. While designing for SQL, remember that Citrix Provisioning has an option to “enable offline database support.” This allows the servers to keep streaming vDisks if the database goes down. It is still a best practice to provide a highly available database for Citrix Provisioning to maintain management operations using an HA design. 

The following SQL choices are available when using Citrix Provisioning in Azure.

• Azure SQL Database
• Azure SQL Managed Instance
• Microsoft SQL Server 2017, 2019, and 2022.
  • Standalone
  • Database mirroring
  • Always on failover with or without multi-subnet failover

Most of these are well documented elsewhere, but if your organization wants to move to a more cloud service-centric approach, Azure SQL Database may be a good choice for Citrix Provisioning on Azure as it can provide a cost-effective way to provision a highly available, local SQL instance that adds minimal requirements for maintenance and upkeep.

Azure SQL comes in two forms—one where you pay by cores and one by transactions (DTU).  If only the Citrix Provisioning database were provisioned in Azure, the solution that would make the most sense would be an Azure SQL Database, which provides a fully Microsoft-managed, highly available service for a single database.  Also, if you only use Azure SQL for Citrix Provisioning, using the DTU model for the service will probably make sense.  Check with your DBA team to determine the best approach for using Azure SQL in your organization.

Citrix License Server

Citrix Provisioning currently relies on an on-premises license and requires a Citrix License Server. If your organization already has an on-premises license server that will remain in the environment, it can be used for licensing in Azure.  Citrix Provisioning servers will continue to work without interruption for 30 days if they cannot contact the license server for any reason.  If the license server is only used in Azure and only for Citrix Provisioning, then the virtual machine instance used for the license server can be configured modestly.  Consider something like the D2_v4 instance type, a 2-core 4 GB RAM instance with a minimal monthly cost.

Always use the latest version of the license server.  Open port 27000 from the Citrix Provisioning servers to the license server.  The license server requires telemetry to be enabled so that the license server will require internet access open to port 443 for https://cis.citrix.com.

The license server should be backed up or replicated for recovery. If the server fails, there is a 30-day grace period for recovery.

Migration Options

Many organizations moving to Azure are already using Citrix Provisioning on-premises.  Depending on the technology you use to deploy Citrix Provisioning images, Citrix has a solution to ease the migration to Azure.  Several options exist for architecting the new solution in Azure if Citrix App Layering is used.  See the blog App Layering in Azure 2023 and Beyond for more details.

If you use a different method of managing Citrix Provisioning images, the Citrix Image Portability Service (IPS) can convert an on-premises image into an image appropriate for Citrix Provisioning in Azure. IPS is a scripted process that can be used for a one-time migration of vDisks into Azure or as a means to synchronize on-premises images to Azure regularly. See the standard documentation in the Image Portability Service and the APIs to Manage the Image Portability Service.

General Notes

Many pieces of information are essential to know when designing and deploying Citrix Provisioning on Azure that did not apply to the architecture sections of this document, but they are nevertheless critical.  These will be listed here:

• Citrix Provisioning on Azure Limitations (Not Supported)
  • Supported with 2203 LTSR, 2402 LTSR, Current Release
  • Windows Server 2012R2 and earlier Targets are not supported
  • Generation 1 VMs are not supported
  • vDisk Update Management
  • Printer Management
  • When Citrix Provisioning is integrated with a customer-managed Delivery Controller, removing VMs, catalogs, or AD accounts from the console is not supported.
  • Trusted Launch (Secure Boot and vTPM)
• Only BDM Boot Disks disks are supported
  • No PXE or ISO
• An Azure Feature Flag for “ReserveMacOnCreateNic” is required to be set on the Azure subscription.  This feature tells Azure to bind the Mac Address to the VM sooner.
• The Azure machine size used when creating the master VM must be compatible with that used when creating target VMs. Only Generation 2 VMs are supported. This includes the following:
  • The presence or absence of a temporary disk must be the same
  • The presence or absence of a GPU must be the same
• When using a Citrix Provisioning wizard to create targets, if the VMs have a temporary disk, creating new targets can use batch sizes of 200 VMs created in parallel.  However, if no temp disk is available on the targets, only 20 targets can be created simultaneously.  If no temp disk is available, set the following REG_DWORD to 10
  • HKCU\Software\Citrix\ProvisioningServices\VdiWizard\MAX_VM_CREATE_THREADS
• For targets provisioned using the new Studio provisioning, the default deployment size will be 10 targets in parallel, but this will soon be increased to 500.
• Master Image cannot include “Plan Information”
  • Meaning that a Citrix Provisioning Image cannot be made from a third-party Marketplace image
• If using Hybrid Entra ID/AD targets, also use Studio provisioning; otherwise, targets must wait up to 90 minutes to join the Entra ID after boot.  Studio provisioning will store the Entra ID join in the Identity Disk, making it available immediately.
• The Citrix Provisioning console only supports Active Directory integration for Admins.
• Both IPv4 and IPv6 are supported.  However, IPv6 is supported on 2311 and is newer for streaming only, and it can’t be used yet with studio provisioning.
• Accelerated Networking should be used on both the Citrix Provisioning server and Targets

Virtual Machine Costs Comparison

To help customers understand the differences in storage requirements between Citrix Provisioning in Azure and various configurations of MCS, we have outlined the options below for different types of provisioned systems.  The costs defined here were obtained using the Azure pricing when this document was written.  The costs below are based on 3-year reserved and pay-as-you-go pricing in Azure US East that can change at any time, but it should provide an understanding of the cost categories involved in the overall solution.  Since SQL would not be required if using MCS, we have also included base pricing for Azure SQL, assuming the least costly DTU-based pricing plan will be used.

The model compares VDAs deployed using different settings in Citrix Provisioning and MCS. Next, we will explain the main assumptions and overall differences between each deployment type. This Storage Cost Analysis V2.xlsxspreadsheet can be downloaded and used to perform what-if analysis by changing the assumptions.

Main Assumptions

• Pricing assumes Windows costs are paid using the Hybrid Rights Benefit
• Compute Gallery Images will use 80 GB of disk space
• 70% of VDAs will use 3-Year Reserved Instance Pricing while 30% will use Pay as you Go Pricing
• VDAs will be either D4D_V4 or D4_V4
• Citrix Provisioning Sites with 2K or fewer targets will use a single Citrix Provisioning site spread across Availability Zones
• Citrix Provisioning Servers use 3-Year Reserved Instance Pricing with E4ds_V5 Instance
• Citrix Provisioning servers will be provisioned with a 2 TB store
• A Citrix Provisioning server is deployed for every 1,667 targets.  This number is rounded up, and then one server is added per site for higher availability.

Deployment Types

The following sections discuss the differences between provisioning options.

MCS without Ephemeral Disk or MCSIO

No disk optimization solution is used in this MCS option, meaning the OS disk must be at least a Standard SSD, as defined in the model. MCS also provides an identity disk for each VDA that is always present and an E1 disk is used because it is the least expensive option, as standard disks have a minimum size of 32 GB versus 4 GB for SSD.

We assume that Compute Gallery disks are used for disk hydration at a 40-1 ratio and that the VDAs will be used for an average of 10 hours per day and shut down by Autoscale at other times.

The Compute instance defined here is the D4s_v4, which is the same price as the D4_v4 but allows SSD disks to be attached. From a performance perspective, the disk performance here will not be as good as that with ephemeral disks or MCSIO, but the cost is a little less than that of using ephemeral disks and about the same as MCSIO.

MCS with Ephemeral Disk

In this MCS option, an ephemeral disk is used for the OS Disk to increase performance and lower the storage costs. Ephemeral disks are on-host SSDs that provide very high IOPS for VDA storage. However, the instances that support ephemeral disks must provide temporary storage big enough to store the VDA's full image. Generally, these instance types are more expensive than instances that don’t have this requirement.  In our example, the instance that can support ephemeral disks costs $9 a month more than the same configuration without this support.

As mentioned earlier, we also have the identity (E1) and Compute Gallery disks. We are likewise assuming here that the VDAs will be used for an average of 10 hours per day and shut down by Autoscale at other times.

The Compute instance defined here is the D4d_v4, which is the same price as the D4_v4, but it has a 150GB temp disk and supports ephemeral disks.

MCS with MCSIO

In this option, we use MCSIO to provide excellent disk performance using less expensive storage in conjunction with a RAM cache.  Remember that this option will use 1-2 GB of RAM to obtain the higher level of performance provided.  Since the RAM cache works so well, standard HDD storage can be used for the OS and cache disks.  As with all the MCS options, this option uses an identity disk and the Compute Gallery for disk hydration.

Citrix Provisioning Provisioned by Citrix Provisioning

The cost makeup of Citrix Provisioning has a few more items to include in the analysis.  When deployed using the Citrix Provisioning process (meaning not using MCS as in the next section), there are:

• OS/Boot Disk (E1) – A 1 GB managed disk used to boot the VDA.  This disk is always provisioned whether the VDA is running or not.
• Cache Disk (E4)—A 32 GB managed disk used as a disk cache for the RAM cache with overflow to disk. This disk is always provisioned whether the VDA is running or not. Currently, the RAM cache uses Standard SDD, which is not configurable like it is in MCS.
• Citrix Provisioning Servers – See main assumptions
• Azure SQL

Citrix Provisioning, like MCSIO, uses cache in RAM with overflow to disk technology to increase the solution's overall performance.  In this model, the Citrix Provisioning servers are amortized over the cost of the VDAs.  We are using a 1667:1 ratio for Citrix Provisioning targets to servers with HA built into each Citrix Provisioning site.  If you have a very low VDA count environment, the cost ratio for the Citrix Provisioning server infrastructure will be higher.

Citrix Provisioning Provisioned by Studio

The makeup of costs when using MCS to deploy Citrix Provisioning is slightly different:

• Identity Disk (E1)—This disk stores the machine's identity and Domain Trust information. It is always provisioned, whether the VDA is running or not.
• OS/Boot Disk (E1) – A 1 GB managed disk used to boot the VDA.  This disk is always provisioned whether the VDA is running or not.
• Cache Disk (S4)—A 32 GB managed disk used as a disk cache for the RAM cache with overflow to the disk. This disk is always provisioned, whether the VDA is running or not.
• Citrix Provisioning Servers – See main assumptions
• Azure SQL

You can see here that there is a new disk to capture the machine identity, and MCS allows us to create a cache disk that is Standard HDD rather than Standard SSD.  These changes allow for a slightly lower overall cost.

Cost Comparison Summary

The main takeaway from this analysis is that the overall costs of the different available Citrix provisioning solutions are relatively similar.

Excluding the solution that uses ephemeral disks, the cost estimates range from $56.72 to $59 a month per VDA.  Compute makes up more than 93% of the cost, while the non-compute portion is relatively low in all cases, ranging from $2.12 to $4.40 per VDA per month.

The least costly option at scale is Citrix Provisioning, provisioned by Studio. However, all the options are similar at scale except for using ephemeral disks, which are less expensive for disks but cost $9 a month more for computing.

This does not mean you should not consider MCS with ephemeral disks, as it is a very high-performance solution that should still be considered if performance is critical.

Design Pillars

Citrix reference architectures are designed with respect to the following critical tenets or “Pillars”:

• Resiliency
• Security
• Cost optimization
• Operational Excellence
• Performance Efficiency

The options chosen to support the pillars within the design are discussed in the next several sections.

Resiliency

Resiliency is the cornerstone of any architecture that supports mission-critical operations.  Each infrastructure component is designed to support local and regional failure in this design. Incorporating multiple Availability Zones per region means that both VDAs and Citrix Provisioning servers are spread across physical data centers within the region. If desired, this configuration can be replicated to another Azure region.  Even within each Availability Zone, multiple Citrix Provisioning servers are deployed, providing N+1 availability.  Citrix Provisioning supports failing overall streaming targets automatically, from one server to another, if one fails without downtime to targets/VDAs.  The SQL database provided using Azure SQL is highly redundant, and Citrix Provisioning is configured to provide “offline database support” if the database becomes unavailable.  This feature allows vDisk streaming to continue if the SQL database is unavailable by caching the database locally, but management is not possible when running offline.

Citrix Provisioning requires storage for vDisks that must be accessed by the Citrix Provisioning servers. In this design, we recommend local storage for Citrix Provisioning because it becomes very highly available. Every server keeps a copy of all the vDisks, and there are then no shared components required to stream the vDisk. This requires more management as the vDisks must be replicated between servers, but the redundancy is worth the extra management.

Security

Using Citrix Provisioning in Azure allows organizations to use the more secure desktop computing model where non-persistent images are used for end-user workloads.  The advantage of this model is that the deployments are centralized and highly managed, and upon reboot, the workload desktops or servers are reset back to the known good configuration. Azure-specific security practices can be implemented at the subnet level to ensure that the desktop can only communicate from the “Citrix Subnets” up to the landing zone and not traverse vertically amongst the other workload desktops/servers.

Cost optimization

When migrating to Azure, designing the Citrix infrastructure with cost in mind is important.  For example, it’s typically much less expensive to deploy hosted shared desktops on Windows Servers than non-persistent VDI desktops or persistent VDI desktops, which tend to be the most costly choice.  In the financial modeling within the cost comparison section, we showed some decisions that significantly affect the overall cost of the solution.  These include:

• Using 3-year reserved instances for Citrix Provisioning infrastructure servers that will run 24/7
• Using 3-year reserved instances for a percentage of targets that will always be used, then using pay-as-you-go instances with Citrix Autoscale to manage the variable usage that can change from day to day or month to month.  In our model, this was a 70/30 split, but since reserved instances are, say, 60% cheaper than pay-as-you-go, the breakeven point in hours per day is approximately 9.5 hours of usage a day.
• Using Citrix Provisioning is less costly from a storage perspective than other designs because there is no large OS disk required while the target is running.  A smaller cache disk is used to provide higher performance at a lower cost.
• See the next section as operational excellence has an associated effect on cost

Operational Excellence

Organizations dedicated to using Citrix Provisioning say that the most significant benefit of using the technology is the speed and flexibility of changing the environment.  Citrix Provisioning allows for updates to an existing image or deployment of a new image very rapidly.  Targets must be rebooted to apply changes with the new vDisk or vDisk version.  On a similar note, adding targets to an environment is also a very quick and easy task, as in Citrix Provisioning, a target is just a deployment VM with an associated boot disk.  Also, Citrix Provisioning has been used for many years, and there is a lot of knowledge capital in the industry on managing it.

Performance Efficiency

Performance efficiency is the ability of a solution to adapt to changes in demand. Citrix Provisioning in Azure, along with Citrix AutoScale, provides a scalable solution that is difficult to match with an on-premises deployment, where capacity increases normally require capital purchases and very long lead times.

Conclusion

Citrix supports several different provisioning methods when using Azure workloads, including Citrix Provisioning (Citrix Provisioning), Machine Creation Services (MCS), and what we would call “manual” provisioning, which means using a provisioning method outside of Citrix where the VDA gets installed using a separate manual or automated workflow.  These are all valid ways to manage Citrix environments.  For those customers who love and see the many advantages of Citrix Provisioning, this reference architecture provides a primer on how to architect a Citrix Provisioning solution in Azure based on the five critical Design Pillars, and it intends to help customers deploy a secure, robust, available, performant environment to delivery Citrix workloads in Azure.

CompanyA is a food manufacturer located in the northern plains of the United States. CompanyA plans to acquire additional food manufacturers across different climates to continue growing and expanding into new types of food. As part of the acquisition process, CompanyA needs a repeatable strategy to integrate acquired company systems into a single, unified experience.

The integration across multiple independent organizations introduces many technical challenges. The challenges mostly focus on granting external users (CompanyA users) access to private resources (CompanyB applications) that separate, independent identity providers authorize.

Company A used Citrix Workspace as the cornerstone of its mergers and acquisition strategy. It is looking to use the same strategy for acquiring Companies C, D, and E.

Success Criteria

As part of the acquisition strategy, CompanyA needs a solution that can quickly and securely allow users access to CompanyA and CompanyB resources. To be successful, CompanyA defined a list of success criteria that forms the basis for the overarching design.

User Experience

The first aspect of a mergers and acquisitions solution is meeting the user’s needs. CompanyA identified the following user-related criteria for a successful design.

Security

The second aspect of a mergers and acquisitions solution is meeting security needs. CompanyA identified the following security-related criteria for a successful design.

Conceptual Architecture

CompanyA created the following high-level conceptual architecture based on the requirements defined by its acquisition strategy. The conceptual architecture meets all of the requirements and gives CompanyA the foundation to expand to additional use cases as identified in the future.

The architecture framework is divided into multiple layers. The framework provides a foundation for understanding the technical architecture of the mergers and acquisitions scenario. All layers flow together to create a complete, end-to-end solution.

At a high level:

User Layer: The user layer describes the end-user environment and end-point devices used to connect to resources.

• Users access resources from the Workspace app regardless of device, resulting in an identical experience across every form factor and device platform.

Access Layer: The access layer describes how users authenticate to their Workspace and secondary resources.

• Users continue to authenticate with their pre-acquisition primary identity.
• Users continue to use their pre-acquisition multifactor authentication solution. If the company does not currently utilize multifactor authentication, CompanyA provides phone-based TOTP tokens that support push-based authentication.

Resource Layer: The resource layer authorizes specific SaaS, web, and virtual resources for defined users and groups and their associated security policies.

• Regardless of a user’s originating company, the user must be allowed seamless access to any authorized resource hosted by other companies.
• To better protect data, CompanyA requires policies that disable the ability to print, download, and copy/paste content from the managed resource to and from the endpoint. CompanyA also requires restricting screen scraping\capturing applications and keylogging malware.
• Due to the unknown nature of the endpoint security status, CompanyA requires VPN-less access to resources using a managed enterprise browser with file encryption on unmanaged endpoints.

Control Layer: The control layer defines how the underlying solution adjusts based on the user's activities.

• With the policies in place to protect the users and company data, there are still risks. CompanyA uses the Security Analytics service to identify compromised users or insider threats and automatically take actions to maintain a secure environment.
• The platform unifying multi-directory authentication must be secured from external threats and attacks. CompanyA enables the integrated Web App Firewall and Bot Management to protect the authentication point in the environment.

Compute Layer: The Compute layer details how components are deployed on hardware, whether on-premises, in the cloud, or in a hybrid cloud.

• Citrix Workspace must be able to access every company’s identity provider, whether an on-premises Active Directory domain or a cloud-based offering from Okta, via a single Workspace site.

The subsequent sections provide greater detail into specific design decisions for CompanyA’s mergers and acquisitions strategy reference architecture.

Access Layer

Authentication

One of the challenges CompanyA experienced with previous acquisitions was integrating identity providers. Merging identity providers can take a significant amount of time. CompanyA utilizes a NetScaler to handle all authentication requests with the new strategy.

The authentication process works by the NetScaler evaluating the user’s company and then applying the correct authentication request. As CompanyA has standardized on Okta for primary authentication, the request is forwarded to Okta. Once Okta completes user authentication, Okta replies to the NetScaler with a token identifying successful authentication.

However, not every company that CompanyA acquires uses Okta. Instead, if the NetScaler identifies the user as from CompanyB, the user is asked for their Active Directory username and password. Those credentials are validated against CompanyB’s Active Directory domain. If CompanyB has already integrated a multifactor authentication solution, users will continue to use their registered tokens.

Because strong authentication is a critical first step to security, CompanyA wants a solution that is ready for organizations not using multifactor authentication. In this instance, after the user authenticates with their company’s Active Directory domain, the NetScaler uses the native time-based one-time password engine to provide multifactor authentication for the user. Once registered to the user’s device, the user can enter the code manually or use the push notification service. Push notifications require users to select “Yes” from their registered mobile device to fulfill multifactor authentication.

nFactor Policy

NetScaler plays a critical role in primary authentication. A nFactor policy is utilized to make authentication decisions.

To process authentication requests properly, the nFactor policy must know what company the user belongs to. Once the correct company identifier is selected, nFactor forwards the request to the correct branch of the authentication policy.

Once the nFactor policy is defined, CompanyA can continue to expand it to incorporate additional organizations it acquires in the future. The nFactor policy allows CompanyA to create additional flows utilizing authentication standards, including LDAP, RADIUS, SAML, client certificates, OAuth OpenID Connect, Kerberos, and more. The nFactor policy engine allows CompanyA to continue integrating additional acquisitions without a redesign.

Zero Trust Network Access

To provide access to internal resources like private web apps, virtual apps, and virtual desktops, CompanyA plans to use the Citrix Enterprise Browser with Secure Private Access service and Citrix DaaS. These two services utilize a zero-trust network access solution, a more secure alternative to traditional VPNs.

The Secure Private Access service and Citrix DaaS use the outbound control channel connections established by the cloud connectors and connector appliances. Those connections allow the user to access internal resources remotely. However, those connections are

• Limited in scope so that only the defined resource is accessible
• Based on the user’s primary, secured identity
• Only for specific protocols that disallow network traversal

Company A wants to ensure the security of the devices connecting to the environment. To this end, they will implement compliance checks of all devices' security posture, including antivirus status, OS updates, and other necessary compliance requirements.

Citrix Enterprise Browser will access all private web and SaaS applications. As a baseline, the following restrictions have been defined:

Resource Layer

Federated Authentication Services

Users authenticate to Citrix Workspace with a primary identity. The primary identity is based on the company’s identity provider. One of the challenges with mergers and acquisitions is access to secondary resources based on a secondary identity. For example, CompanyA allows certain users from CompanyB and CompanyC to access virtual Windows applications. To access a virtual Windows application, the user must have a user account (secondary identity) within the virtual resource domain. A CompanyB user’s account (primary identity) will not authenticate to a CompanyA resource (secondary identity). To translate credentials between a primary and secondary identity and provide single sign-on to virtual Windows applications, CompanyA uses the Federated Authentication Service within Citrix Cloud.

The Workspace Single Sign-On Tech Brief contains additional information related to the Federated Authentication Service.    

Web and SaaS applications must be configured to accept authentication information coming from either NetScaler or identity provider for users from Companies B and C to access.

Resource Security Policies

CompanyA wants to limit the risk of data loss due to an insider threat. Therefore, it incorporates numerous restrictions within the different application types to prevent users from copying, downloading, or printing data.

As a baseline policy, CompanyA defined the following (with the ability to relax policies as needed based on user and application).

Control Layer

Web App Firewall

When users authenticate to Citrix Workspace, they access a custom authentication form that supports the mergers and acquisitions strategy. The authentication form, hosted on NetScaler, is a public webpage that must be protected from bots and attacks. CompanyA uses the Bot Management and Web App Firewall components of the NetScaler solution to protect the public web app better.

The first line of defense is Bot Management. Bots can crash or slow a public web app by overwhelming the service with fraudulent requests. The NetScaler's bot management component detects a bot request and prevents it from inundating the system.

The second line of defense is the Web App Firewall. The Web App Firewall protects the policy engine that handles the submitted credentials from attack. These attacks typically include buffer overflow, SQL injection, and cross-site scripting. Web App Firewall detects and denies these attacks from impacting the authentication policy engine.

Security Analytics

CompanyA must identify and stop insider threats to the environment before the impact is too great.

To help protect the environment, CompanyA uses Citrix Analytics for Security to identify insider threats, compromised users, and compromised endpoints. In many cases, a single instance of a threat does not warrant drastic action, but a series of threats can indicate a security breach.

CompanyA developed the following initial security policies:

The Citrix User Risk Indicators document contains additional information regarding the various risk indicators provided to Citrix Security Analytics.

The Citrix Policies and Actions page contains information about the remediation steps Citrix Security Analytics can perform.

Sources

We want to provide you with source diagrams that you can adapt to help you plan a mergers and acquisition strategy. These source diagrams are found here: Ref Architecture - M and A.vsdx

In this guide, we walk you through designing a Citrix virtualization system on GCP. As the journey progresses, we discuss the implications of the decisions you need to make, and curating more reference resources along the way. This guide is a living document. Be sure to bookmark it and check back periodically to see how things change over time.

We start by reviewing the common design patterns for Citrix virtualization technologies on Google Cloud. Some think of these 'design patterns' as ‘reference architectures', but when we're working with infrastructure as code and cloud services, ‘design patterns' make a lot more sense.

Next we explore the Solution Components and Requirements. We lay out the solution prerequisites and give you an overview of what services/components are required to create a Citrix Cloud ‘resource location'.

We then revisit and dig more deeply into the design patterns, armed with a greater understanding of the services/components of a Citrix virtualization system on Google Cloud. Finally, we dive deeper into specific topic areas, including Virtual Delivery Agent (VDA) Design and Management, Citrix ADC/Gateway on Google Cloud, and Citrix StoreFront on Google Cloud.

Design Patterns for Citrix virtualization on Google Cloud

We recognize that different customers are at different stages on their journey to "the cloud". As such, we outline three design patterns that represent a spectrum from "we are all in" to "we will get there but it can take us a while". Observant technologists see the common elements between all three. They start to see how they can mix and match customer managed and cloud services to meet different business needs and environmental influences. We explore this modularity of subsystems when we revisit these three design patterns later in this guide.

The Cloud Forward Design Pattern

Organizations of all shapes and sizes are making the move to "the cloud" and subscription based managed services. For customers who are all in on "the cloud" (or interested in experiencing the best of what the cloud has to offer), the Cloud Forward design pattern is a great match. The Cloud Forward design pattern:

• Uses state of the art, cloud-delivered services from Citrix and Google.
• Is commonly used for new deployments, in addition to technology evaluation, proofing, training, and other use cases that value simplicity, flexibility, and speed of deployment.
• Requires no existing infrastructure or licenses, and can be built in less than 30 minutes using Google Deployment Manager templates such as the Citrix on GCP GitHub project.
• Provides high availability of all critical services.
• Creates a Citrix Cloud "resource location" - the foundation of the other two patterns outlined here.

All you need to get started is a GCP Project and access to a Citrix Desktops-as-a-Service (DaaS) subscription. Evaluation subscriptions to Citrix Cloud are available through Citrix and Citrix authorized resellers. Google also offers new customers a free trial which includes $300 of Google Cloud credit.

Note:

The GCP free trial does not include the use of Windows Server images, as noted in the Google Cloud Free Tier document. To use Windows Server images you must upgrade to a paid account in GCP. Your free credits still apply when you upgrade to a paid account as noted in the Upgrading to a paid account section Google Cloud Free Tier document.

This design pattern uses more than one of all key resources (➊) deployed in separate zones in a given Google Cloud region for high availability.

Citrix Cloud Connectors (❷) are responsible for communications to and from Citrix Cloud (❻), using outbound TLS connections to Citrix Cloud services over the Internet. Once installed on domain-joined Windows Server VM instances, the Cloud Connector software is automatically updated and maintained by Citrix Cloud.

Apps and desktops are provided by Windows or Linux VM instances, or both running Citrix's Virtual Delivery Agent (VDA) software (❸). The Citrix VDA software uses Citrix's sophisticated HDX technology to provide the best possible user experience with virtualized applications and desktops. VDAs register with Citrix Cloud Connectors, which are responsible for brokering HDX session connections to the VDAs. HDX sessions are proxied into the VPC through the Cloud Connectors by default, or optionally through the Citrix Gateway Service by configuring the 'rendezvous' policy. VM instances can be optionally backed by Google Cloud GPUs to create virtual workstations, in turn accelerating graphics intensive applications.

Citrix VDAs are most commonly deployed close to the infrastructure required by the applications being delivered (❹). As such, the application infrastructure is typically deployed or migrated into the same region as the Citrix VDAs.

End-users use the Citrix Workspace app (❺) (CWA) to connect to and interact with virtualized applications and desktops using Citrix's innovative HDX session remoting protocol. The CWA is available for almost any device or operating system, including Chrome OS, Windows, OSX, iOS, Android, and Linux.

This pattern uses the following cloud services (❻) from Citrix, which are secure and highly available by design:

• Citrix DaaS: provides session brokering, load management, single instance image management, monitoring, and cost/capacity management services.
• Citrix Workspace service: the user interface of the solution. This web service provides multifactor authentication, content presentation, and launching services for the Citrix Workspace app. This service consolidates access to virtualized applications and desktops, web, and SaaS applications, in addition to Enterprise file stores.
• Citrix Gateway Service: provides secure access to virtualized applications and desktops, in addition to Enterprise web applications, to devices on public networks.
• Citrix Analytics Service: uses advanced machine learning technologies to provide enterprise-grade security, performance, and user behavioral analytics and reporting.

This design pattern can also be paired with a Google Cloud VPN/Interconnect to extend existing Active Directory investments (❽) into Google Cloud or to provide access to traditional, on-premises, customer managed applications and infrastructure.

It ought to be clear that the architecture of the cloud forward design pattern creates a Citrix Cloud resource location. This is the common foundation shared across all three patterns presented here. The differences between the patterns lie in which technologies are used to service the five components of a Citrix virtualization system outlined in the following table. With the cloud forward design pattern, cloud services are used for all five components:

The cloud forward design pattern can be replicated, using the same Active Directory (or not) in different Google Cloud regions. This makes the pattern useful for deployments with geographically distributed applications and data. This pattern, for production deployments, is often extended by connecting the resource location in GCP to customer managed data centers using Google Cloud VPN, Google Cloud Interconnect. Such a private network connection allows you to extend key services (such as Microsoft Active Directory) up into Google Cloud. This can provide VDAs with access to applications and resources that have not yet been migrated. It can also act as a conduit to migrate apps and data up into Google Cloud. While not shown in the preceding diagram, the Citrix Workspace Environment Management service is commonly used, especially as systems make their way to production. The Workspace Environment Management service uses intelligent resource management and Profile Management technologies to deliver the best possible performance, desktop logon, and application response times for Citrix Virtual Apps and Desktops deployments. See User Environment/Settings Management later in this guide for more details.

The Hybrid Design Pattern

The Hybrid design pattern builds upon the Cloud Forward design pattern. It introduces customer-managed access layer components from Citrix (➊) to flexibly meet the needs of specific customer demographics and use cases. These customer-managed components include the following:

• Citrix ADC/Gateway(❷): deployed as virtual appliances on GCP, this component is often used for use cases requiring one or more of the following:
  • Advanced authentication scenarios, such as SAML/OAUTH 2/OpenID federation, RADIUS, smart card, and conditional access requirements.
  • Highly optimized and flexible session access for end user devices on public networks.
  • Advanced networking services such as content switching, web app firewall, integrated web caching, attack mitigation, application load balancing, and SSL offload.
  • Ability to direct specific users/devices to specific ‘stores' based on advanced, highly flexible, and contextually aware policies. Policy decisions can be based on user profile attributes, location, device type, device health, authentication results, and more.
• Citrix StoreFront(❸): The predecessor of the Citrix Workspace service, StoreFront is Citrix's ‘classic' provider of UI services. Installed on customer-managed Windows Server instances, StoreFront is often used for use cases requiring one or more of the following:
  • Extreme high availability, capable of surviving a broader range of failure scenarios, particularly when deployed in a highly available configuration.
  • Flexible session routing, with the ability to route internal user session traffic directly to VDAs while sending external users through Citrix Gateways.
  • Single sign-on from customer-managed, on-premises devices.
  • The need to provide multiple ‘stores' with different configuration properties to support diverse use cases on the same system.
  • The need for highly customized or branded, HTML based user interfaces.

With the Hybrid design pattern, Citrix access layer components are deployed in the customer's Google Cloud environment (➊). The components are typically deployed in pairs spread across multiple zones for high availability.

This pattern uses Citrix's ADC/Gateway VPX (virtual) appliances to securely proxy HDX sessions into the VDAs in the customer's environment (❷). Citrix ADC/Gateway appliances can be used with the Citrix Workspace service for simple session proxy services or complex authentication scenarios, or both (UI option A). It can also be paired with Citrix StoreFront (UI option B).

This pattern optionally uses Citrix StoreFront (❸) for UI services, allowing the system to meet the requirements for more complex use cases as outlined above. It pairs with Citrix ADC/Gateway, which handles authentication in addition to UI and HDX session proxy services.

To put the hybrid design pattern into the context of the five components of a Citrix virtualization system:

There are many other functional items you may also find important to consider before choosing between the cloud service or customer managed components. We provide you with a deeper dive into Citrix ADC/Gateway and Citrix StoreFront on GCP in later sections. You can use different combinations of technologies at each layer to achieve specific outcomes or meet specific needs - at the expense of simplicity.

For example: Citrix ADC/Gateway VPX appliances can be added to a system and used for Authentication or HDX proxy functionality while using Citrix Workspace for UI services. This gives the system the ability to support almost any identity and authentication strategy (including federation scenarios), plus the ability to use HDX's Enlightened Data Transport for the best session performance over suboptimal networks.

You can also introduce Citrix StoreFront to use for UI services, in parallel to or instead of Citrix Workspace. StoreFront requires Citrix ADC/Gateway for most use cases, but this combination would serve use cases with extreme high availability requirements, heavy UI customization requirements, and the ability to create multiple different ‘stores', with different properties, for different groups of users, device properties, physical locations, and so on.

The Cloud Migration Design Pattern

The Cloud Migration design pattern further builds upon the Hybrid design pattern. It allows customers with existing investments in Citrix technologies to systematically modernize their infrastructure, while seamlessly migrating workloads to the cloud. This migration can be done at a pace that doesn't disrupt existing/critical systems and use cases. It allows technologically conservative customers to ‘wade' into the Cloud, workload by workload, mitigating risk along the way on the customer's terms. It also allows the customer to systematically reskill staff on the latest, most capable technologies from Citrix and Google, and build their organizational cloud readiness at a manageable pace while using their existing investments in technology, infrastructure, knowledge, processes, and operationalization procedures.

One common example: an existing Citrix customer, with a substantial investment in a customer managed deployment of Citrix Virtual Apps and Desktops wants to begin running their app and desktop workloads on GCP. Perhaps they've also got multiple ‘Citrix farms' they currently manage on-premises. The customer has deployed Citrix StoreFront and most likely Citrix ADC/Gateway appliances to provide authentication, UI services, and HDX proxy services.

In this scenario, the customer is probably already using StoreFront's ability to aggregate apps and desktops from their multiple ‘Citrix farms' into one UI. To begin ‘moving in' to Google Cloud, they'd start by creating a Citrix Cloud resource location in a region of their choice. Assuming they're all on the same network, they can simply add the new ‘Citrix farm' to StoreFront and deploy their first virtualized workload on Google Cloud. This gives them the ability to run Citrix workloads in two environments, side by side - some on-premises, some on GCP - and migrate workloads to GCP as business priorities allow.

The cloud migration design pattern starts with the hybrid pattern described in the preceding section. The system built with the hybrid pattern becomes the new, state of the art environment where new workloads are deployed. The cloud migration pattern uses Citrix StoreFront(➊) or the Citrix Workspace (❷) user interface, or both to connect legacy Citrix environments (❸) to the new environment, as both UIs can present multiple brokering environments in a single view with a single login. This provides users with a single UI (❹) they can use to access both environments. This allows the customer to deploy new workloads onto Google Cloud (❺), while systematically migrating existing workloads to Google Cloud as the businesses opportunities and constraints dictate.

This same customer can also run Citrix Workspace side by side with StoreFront, and add the legacy ‘Citrix farms' to the Workspace UI using the Citrix Cloud Site Aggregation feature. Both UIs would provide access to the same resources for the same users. End-users can be gradually migrated to the Citrix Workspace service UI as business priorities allow.

The benefit of the Cloud Migration approach is that IT can migrate the app and desktop workloads from the legacy on-premises infrastructure to Google Cloud at a pace that suits them. Users can continue to access their applications and desktops in the same way, regardless of whether the workload is being delivered on-premises or from Google Cloud.

Using Site Aggregation, customers are also able to use Citrix Analytics, giving them insights into security, performance, and operations of both their on-premises and cloud-hosted infrastructure. This can help in the decision-making process of when a workload ought to be moved from on-premises to Google Cloud Platform. Citrix Security Analytics can also be used to ensure that as workloads become distributed across on-premises infrastructure and Google Cloud, the customer's security posture can be enforced.

Migration with Google VMware Engine

If you are considering the Cloud Migration design pattern, there's probably a good chance you're running Citrix on VMware today. For some customers, the option of extending their existing VMware based infrastructure to Google Cloud can be appealing. For these customers, this path promises to expedite workload migrations, using existing knowledge and process investments to get there sooner. With Google Cloud VMware Engine, customers can provision and run VMware Cloud Foundation (VCF) based Software-Defined data centers (SDDCs) on Google Cloud.

Citrix DaaS enables provisioning and image management of VDAs on VMware VCF-based public clouds. Before launch, Google Cloud VMware Engine underwent a comprehensive compatibility test to become Citrix Ready Verified. Both Citrix Provisioning platforms (MCS and PVS) were tested and functioned as expected. For more information, see Google Cloud VMware Engine on Citrix Ready.

When customers spin up a VCF based SDDC using Google VMware Engine, the SDDC (which includes compute, storage, and networking plus management) is peered to VPC networks on Google Compute Engine. This allows you to run workloads on the SDDC or Google Compute Engine, providing customers with options for various workloads. The following logical diagram depicts the relationship between Google Cloud, Citrix Cloud, and a managed SDDC instance:

Note:

Customers who have a firm requirement for full Citrix App Layering or PVS support ought to consider running their Citrix Cloud resource location on Google Cloud VMware Engine. Both Citrix App Layering and PVS are currently available on VMware-based platforms.

While a design and implementation of a Citrix virtualization solution on Google Cloud VMware Engine is outside the scope of this design guide, most of the concepts and components described in this guide still apply. For more information regarding setting up a Citrix Cloud resource location on VMware (Cloud Foundation), see Citrix DaaS documentation.

Solution Components and Requirements

Virtualization System Prerequisites

To build a Citrix virtualization system on Google Cloud, you need a minimum of two things to get started:

• A Google Cloud Project
• A Citrix DaaS subscription

Note:

The GCP free trial does not include the use of Windows Server images, as noted in the Google Cloud Free Tier document. To use Windows Server images, you must upgrade to a paid account in GCP. Your free credits still apply when upgrading to a paid account as noted in the Upgrading to a paid account section within the Google Cloud Free Tier document.

Got your pre-requisites lined up? Good! Now let's introduce you to what you're building.

Tip:

The Citrix DaaS documentation is relevant as this service is at the core of the solution. Be sure to give it a read before you start building, and keep it handy for when you need more information. It can be found on the Citrix docs site.

The Citrix Cloud Resource Location

The foundation of a Citrix virtualization system on Google Cloud is a Citrix Cloud construct called a "resource location". A resource location, in Citrix speak, is roughly analogous to a region on GCP. It's a well-connected grouping of resources, on a private network with an Active Directory, Internet egress (to utilize secure cloud services from Citrix and Google), and some applications and data you want to virtualize and securely deliver to any device in the world. Virtualized apps and desktops are delivered from VM instances on GCP called "VDAs". These are Windows or Linux VM instances running Citrix's VDA software which enables the desktop or application user interfaces to be remoted to client devices using Citrix's HDX display protocol. VDAs register with Cloud Connectors, which securely proxy communications with Citrix Cloud Services.

Tip:

One key rule of thumb for delivering virtualized applications to be aware of. You want to put the apps (running on the Citrix VDAs) near the data (infrastructure required for the apps). Not having apps and data local to each other can result in increased latency and slower applications, which can ultimately cause a degraded user experience.

Resource locations are typically built to be highly available, meaning your key services are spread across zones in the GCP region. Where applicable, more than one VM instance for key services are deployed for availability and manageability purposes. The key services you need to have in a resource location are:

• *Microsoft Active Directory
• *Citrix Cloud Connectors
• *A method for reliable Internet egress, such as Cloud NAT
• *Citrix VDAs (GCP VM instances with Citrix's VDA software installed underneath the applications being delivered)
• Other infrastructure, as needed, to support the applications being delivered

Let's further explore some of these services* in more detail as they're required to have a functional Citrix Cloud resource location on Google Cloud.

Microsoft Active Directory

All design patterns for Citrix virtualization systems on Google Cloud require Microsoft Active Directory. For a compelling user experience, Active Directory services ought to be available in every GCP region where you've got VDAs deployed, hence it's considered a core component of a Citrix Cloud resource location. AD is used for configuration management (group policy) in addition to authentication, though as we learn later, AD does not need to be the identity provider for the system. Many customers extend their existing AD into Google Cloud, though Citrix virtualization can flexibly integrate into various AD designs and servicing models.

When deploying Active Directory on Google Cloud, customers can build/maintain their own Active Directory Domain Controllers using Windows Server instances, use Google's Managed Service for Microsoft Active Directory, or a combination of the two. Active Directory trusts can also be used to connect two or more AD forests/domains depending upon the customer's needs.

For customers looking to minimize the administrative overhead required to build and maintain functional Active Directory services, the Google Managed Service for Microsoft Active Directory (Managed Microsoft AD for short) is an option worth considering. This service provides you with a fully functional Active Directory forest/domain without the overhead of building and maintaining Windows Server VM instances. The Managed Microsoft AD service is built on highly available, Google-managed infrastructure, and delivered as a managed service. Each directory is deployed across multiple GCP zones, and monitoring automatically detects and replaces domain controllers that fail. You do not have to install software, and Google handles all patching and software updates. With Google's Managed Service for Microsoft Active Directory, you can use native Microsoft administrative tools, manage Windows machines and users with Microsoft Group Policy. You can also join VM instances to it, and even setup Active Directory trusts with existing AD instances to support various complex Enterprise scenarios.

Customers who choose to use Google's Managed AD Service with Citrix virtualization technologies can expect these technologies to work, with a few important things to consider before doing so. For starters - you won't have Domain Administrator, Enterprise Administrator, or other 'super user' type access to the AD instance. You do, however, have full control of your own container at the root of the directory where you can create users, computers, groups, OU's, and group policies. You can also set up and use various types of trust relationships with other directories.

A few other things you CAN NOT do:

• Create AD objects in any of the default containers (such as /Computers): they're read-only. This brings up a common mistake some customers make when using Citrix's Machine Creation Services (MCS) provisioning technology: you must choose to create the machine accounts for your MCS managed VDAs in a container/OU that's writeable. If you don't choose such a location, MCS is not able to create the machine accounts.
• Install and configure some AD integrated features such as Certificate Services. As such, this impacts customers who plan to use Citrix's Federated Authentication Services (FAS) technology, which requires AD integrated Certificate Services. These customers must build and manage their own Active Directory on Google Cloud using Windows Server VM instances.
• Have local Server Administrator equivalence by default. In an 'out of the box' Active Directory installation, the Domain Administrators group is added to the local Server Administrators group by default. If you're using the Google Managed Service for Microsoft AD, you may have to create your own server administrators' group, add your own users to it, create and apply a group policy to add your group to the built-in Server Administrators group on member servers/workstations.

While trust relationships, site/service configuration, replication, and other AD related topics are not covered here, Citrix has provided extensive documentation on these topics applicable to all three deployment models.

Note:

For more information on Active Directory requirements for Citrix virtualization on Google Cloud, see Citrix Cloud Connector Technical Details. Besides covering supported Active Directory functional levels, this article also covers deployments scenarios for Cloud Connectors in Active Directory.

Important:

DNS name resolution is important for a properly functioning system. DHCP on GCP always uses Google's name servers. For VM instances to ‘find' and join your Active Directory instance on GCP, you want to implement private managed DNS zones, though not necessary if using the Managed Microsoft AD service. See Google Cloud DNS overview for more information.

DNS name resolution is also important when implementing Citrix's Rendezvous feature for HDX session proxy, including usage of EDT/Citrix adaptive transport. See Rendezvous protocol documentation for more details.

Citrix Cloud Connectors

Citrix Cloud Connectors function as a secure, cloud managed proxy for the Citrix virtualization system. Cloud Connectors are dedicated, domain joined Windows Server instances in separate zones within a region. It also functions as an offline session broker if Internet access is interrupted ("local host cache mode') - useful for mission critical deployments with extreme availability requirements. We discuss this function in more detail as we get into the Hybrid design pattern later on in this document.

Cloud Connectors are typically deployed as an N+1 resource, using VM instances spread across multiple zones in a given region. This enables a resource location to scale and facilitates the automatic update of the Cloud Connector software.

Note:

For more information on instance sizing for Citrix Cloud Connectors, see scale and size considerations for Cloud Connectors.

Cloud Connectors can sit on any VPC where they can reach Active Directory and the Citrix VDAs, and they need reliable Internet egress to function properly. One simple, highly available method for providing Internet egress is Google Cloud NAT, though many other methods are supported as well. For use cases with strict egress controls or auditing requirements, egress traffic from the Cloud Connectors to Citrix Cloud can be proxied.

Note:

For more information on ports and protocols used by Citrix virtualization technologies, see Communication Ports Used by Citrix Technologies. This document provides the foundation for the firewall rules you establish on Google Cloud.

Citrix VDAs

The last resource type you need to have a functional Citrix virtualization system are called VDAs - and they're the actual workload you're delivering. As mentioned earlier, these are Windows or Linux VM instances running Citrix's "Virtual Delivery Agent" software, which enables the desktop or application user interfaces to be remoted to client devices using Citrix's HDX display protocol. VDAs can be created and managed outside of the system using any provisioning mechanism you'd like. For example, you can use Google Deployment Manager templates, but for any type of scale, you want to use Citrix's MCS technology.

MCS automates the creation of ‘machine catalogs' - groups of identically configured VDAs built from one or more ‘golden master' VM instances. MCS uses snapshots of the persistent disk to capture the state of the operating system and application stack installed. It also uses the instance attributes of your ‘golden master' VM instance to create instance templates, which apply these attributes to VDAs under MCS management. MCS also automates the updating of the system disk on the VDAs using similar snapshots of the ‘golden master' disk, and orchestrates the Autoscale feature's efforts to automatically manage cost and capacity.

There's a lot more to know about VDAs, but we dig deeper later on in this guide. Can't wait? Head straight to VDA Design and Management Considerations.

Note:

See Communication Ports Used by Citrix Technologies to identify the firewall rules you establish on Google Cloud.

Additional note:

VDAs do not have to have Internet egress - and for certain use cases they don't by design. If, however, the VDAs do have Internet egress, the "rendezvous protocol" feature can be enabled via Citrix policy, allowing client devices (which run the Citrix Workspace app) and the VDA to securely connect via the Citrix Gateway Service. This improves the scalability of the Cloud Connectors, who are often responsible for proxying the HDX connections into the VDAs. The other option for proxying HDX connections over public networks - deploy customer managed Citrix ADC/Gateway instances at the perimeter of the VPC.

VDA Design and Management Considerations

The most dynamic part of a Citrix virtualization system is the VDA. Remember that VDAs are where the actual work is happening - the apps and desktops you provide users on a Citrix virtualization system run from VM instances on GCP. You want to make sure you get this layer right, but don't let perfection get in the way of progress! Do your homework up front. Set the expectation with users that the system will change over time. ...and build simple and effective processes to handle change: it's inevitable! With the power and flexibility of Citrix virtualization tech, managing change doesn't have to be a major burden.

In this section, we've attempted to logically break the topic up such that we can dive deep without losing context. We do our best to provide the details you need in each section and call out leading practices and recommendations along the way.

We start by examining the different VDA related options for delivering your mix of apps and desktops, and there are quite a few! We then dive into how to configure and use Citrix Cloud's VDA fleet and image management technologies, including MCS and the Autoscale feature. We then introduce user environment management (registry settings, drive/printer mappings, and so on) and user settings management (user profiles, personalization layers, home drives, and so on) options, dive into cost optimization and capacity management, and wrap up the section with more performance tuning considerations.

This is an ambitious amount of knowledge to distill - let's get after it!

Workload Delivery Options and Considerations

As you begin your workload delivery journey, it's important we start it on the right foot. That means touching on a couple non-VDA specific elements that need to be considered first. One of the most important conversations a good Citrix consultant has with a new customer is about the use cases you're going to be servicing. These conversations (more than one, because customer needs, business climate, and technology evolve over time) typically lead to the definition of reasonably well-defined groups of users and apps - we call them Delivery Groups. Most of the options we're going to break down in this section are re-evaluated for each Delivery Group and use case. It's common for customers to have quite a bit of variation and even overlap between Delivery Groups. At the end of the day, however, the most foundational element of each Delivery Group is the mix of applications, data, and services to be provided. Once you have that defined, you can begin to evaluate the decisions laid out in this section.

Important:

For each use case/Delivery Group, start by defining the mix of apps, data, and services needed, then work your way through the following considerations to decide what delivery options can best serve each.

Tip:

VDAs are managed in a resource grouping called machine catalogs. Machine catalogs are groups of virtual machine instances which serve a common use case for a group of users. They're typically based off the same ‘golden master' VM instance template, and inherit VM instance properties and a generalized copy of the persistent disk.

VDA Operating Systems

Windows or Linux

Now that you've defined the apps, data, and services required for each Delivery Group/use case, you can begin to consider what operating system is best suited for your VDAs. The most basic question: do you need Windows or Linux? This decision is often forced by the requirements of the application or set of applications you're delivering. If the app only runs on Windows, then Windows it is! The same obviously applies if the app only runs on Linux.

Business applications are often built upon Windows, so a large percentage of Citrix virtualized apps run on Windows based VM instances on GCP. Sometimes Windows is chosen because it's what the IT team knows, and the cost of spinning up operational competencies on a new OS like Linux is deemed too high, so Windows is used even if the application set can run on Linux. If, however, the app set can be run on Linux, it's worth considering - much of the complexity and a good portion of the costs (Windows OS and client licenses) can be avoided.

Server or desktop OS

If you can use Linux as the OS, the choice of ‘server or desktop' is relatively simple. You must pick a version that has a GUI, can run on Google Cloud, and is supported by the Citrix Linux VDA.

If you deploy Windows, the choice of server vs. desktop OS gets a bit more complicated. Both options share a common GUI, and both can present users with a virtual desktop. In fact, most Windows applications run on both Windows Server and Windows 10 desktop operating systems, though often application vendors won't call out Windows Server support in their documentation. The most major implication of Windows Server vs. Windows 10 desktop is licensing - and it's a large one.

Microsoft's licensing policies are restrictive when running Windows 10 (or any other ‘desktop' operating system) on public clouds. These policy-based restrictions can make it more expensive to run a Windows desktop OS on any public cloud, including GCP. For more information on Microsoft's licensing policies, consult your Microsoft licensing specialist, but the following gets you started on this complex topic:

• Microsoft Volume Licensing Product Terms (April 1, 2020) - for Windows Client, Windows Server, and Windows Services.
• Microsoft Office Licensing page - get the licensing brief.

If you're running Windows on GCP, Windows Server serves most use cases and application mixes, and you simply pay for the license usage along with instance usage. It's often more cost effective than a Windows desktop and ends up being the choice for many virtualization systems on Google Cloud.

Shared OS (multi-user) or single user ("VDI")

One common misconception is that Windows Server cannot serve desktop use cases, regardless of whether you are sharing the OS between multiple users or have one OS/VM instance per user. This misconception is false! When deployed in Multi-user mode (that is, RDSH role is installed), Windows Server can present users with a ‘hosted shared' desktop. Windows Server can also be used for "VDI" use cases, and while not as cost effective or scalable as the multi-user/shared OS option, it's a legitimate option for a single user desktop. We call this delivery model "Server VDI".

To summarize, the following combinations of options/operating systems can be used depending upon the use case:

Another common mis-conception is that Google Cloud's sole-tenant nodes (STN) are required to serve ‘desktop' use cases. Sole tenant nodes are required to comply with Microsoft's BYO licensing scenarios such as Windows 10 (desktop) OS. As mentioned above, Windows Server can be used to deliver a single-user desktop ("Server VDI") in addition to a multi-user desktop (Hosted Shared). Sole tenant nodes can also be used for Windows Server instances when you're bringing your own Windows Server licensing.

Most flavors of Linux are multi-user capable out of the box. As such, they can be deployed in either Hosted Shared or "Server VDI" models, with similar relative cost implications.

Note:

To help with the decision making process, the following decision tree compares Hosted Shared Desktops (Server OS multi-user desktops) to VDI Desktops. The tree doesn't explicitly differentiate between client VDI and server VDI models, but the decisions presented are valid for both. When a use case suggests VDI is the appropriate delivery model for your workload, Server VDI ought to be considered wherever possible for running on Google Cloud.

Published desktops or published apps

At the end of the day, the virtualized apps you deliver to users in a Citrix virtualization system run on VDAs. You have options for how you present them, which determines how users interact with them. You can present the user with, or "publish" individual applications and files. You can also present them with a desktop on which they interact with applications and data.

Example: a hosted shared desktop, being presented as a windowed app in Citrix Workspace app for Windows.

There's more to this choice (and many customers use both), but here's an attempt to summarize:

Published desktops (both hosted shared and VDI):

Published applications:

Pooled or persistent

This choice is another property of the machine catalog and is defined upon catalog creation. The hosted shared delivery model usually uses pooled/non-persistent VDAs, but both VDI models can use either pooled or persistent machine catalogs. With the pooled model, OS instances are reset by MCS on logoff or reboot of the VDA. This model ensures that users get a ‘clean' system image, which is in turn based on your template or ‘golden image' VM instance(s) and snapshots of it's persistent disk. They're referred to as ‘pooled' as a pool of VDAs are maintained and users are dynamically connected to an available/unused VDA in the pool. User settings and data can be managed several different ways. With pooled VDAs, they're handled such that the user gets the same configuration and experience regardless of which VDA they're logged into. See user environment/settings management in this document for more details on this topic.

Persistent machine catalogs contain VDA instances which are assigned to individual users, and they persist between reboots. This model is useful for scenarios where users need to install their own applications (such as developer environments) and use cases where necessary applications are not multi-user compatible.

Pooled instances tend to be the easiest to manage over time since Citrix's MCS can update the system disks attached to pooled instances with a few clicks. Capacity and cost management also tends to be more effective since an idle pool of instances can serve many users. Pooled instances are a bit less flexible than dedicated since changes to pooled instances don't usually persist between reboots. Technologies such as Citrix User Personalization Layer can be used to persist user initiated changes across sessions on different pooled VDAs, though it's only compatible with single user "VDI" use cases.

Persistent instances can be simpler to deploy, but tougher to manage over time since you have to handle OS/application patching, upgrading, and maintenance inside the VM. It can also be more expensive from a cost/capacity perspective as it is often tough to predict when a user will log on. This means that users must wait while their instance is started, or administrators must keep them running during time windows where each user is expected to log on.

Managed or unmanaged

Catalogs created and managed by MCS can contain persistent or non-persistent clones of template (or ‘golden master') VM instances. Machine catalogs can also be provisioned with another process or technology. Either way, you want to make sure they're created as power managed:

If you don't use power managed machine catalogs, key features such as Citrix Autoscale will not work, and you won't have help managing costs and capacity. Using MCS for VDA fleet provisioning and management brings a host of useful benefits to administrators, but ‘unmanaged' VDAs - those provisioned outside of Citrix - can also be used. We cover those benefits in Fleet and Image Management later in this guide.

GPU Acceleration

Certain types of applications deployed on VDAs can benefit from GPU resources if they're available to the VM instance. All three delivery models (hosted shared, server VDI, and desktop VDI, for both Linux and Windows) can use NVIDIA accelerated GPU instances for graphics workloads on Google Cloud. These Virtual Workstation GPUs can be attached to general-purpose N1 machine types for graphics intensive workloads such as 3D visualization, chip design, CAD/CAM, graphics and video editing, and include the required GRID license.

With the appropriate NVIDIA GRID driver installed on the instance, Citrix's VDA software detects GPU presence and configures itself appropriately.

Tip:

Citrix's HDX display protocol stack does lots of auto-detecting and adapting on the fly to provide the best possible user experience. However, it also tries to balance performance, responsiveness, and richness of the UX with bandwidth consumption. As such, graphics intensive workloads often benefit from some fine-tuning to get the balance right. See HDX Graphics Overview for more information. Note that Citrix does provide a policy template called "Very High Definition User Experience" (as outlined in Baseline Policy Design). This template can be used as the starting point for fine-tuning to your specific environment.

Cost Optimization and Capacity Management

When running VDAs on Google Cloud, you're paying for the compute, storage, and networking resources you use. This means there's a direct correlation between the amount of capacity you consume and the costs you incur. The choices you make and the practices you adopt have a direct correlation to the operational cost of the virtualization system.

First off, make sure you choose the right workload delivery options - the topics you just read through if you're reading this front to back. These can have a dramatic impact on the total cost of the solution! Here are a few other recommendations and topics to consider as you work to balance cost with capacity and optimize the user experience.

On-demand Provisioning

When you use MCS to create non-persistent machine catalogs in Compute Engine, MCS uses on-demand provisioning to reduce your storage costs, provide faster catalog creation, and provide faster instance power operations. With on-demand provisioning, Compute Engine instances are created only when Citrix DaaS initiates a power-on action. On-demand provisioning is used for non-persistent machine catalogs.

Note:

Some administrators find on-demand provisioning confusing initially, as VDA instances do not show up in the Google Cloud console until MCS powers them on. Also, since the instances receive a new virtual NIC and MAC address, it can take some time for Active Directory DNS entries to update/replicate successfully. VDA identity disks DO persist between reboots and creation/deletion events.

Rightsizing VDA Instances

Now that you've gotten some insight into the important decisions related to workload delivery options, let's dig into right-sizing your VDA instances. For VDI based delivery models, selecting the right instance type is straightforward. Assuming you've done some homework and have a decent understanding of the resource requirements of the OS, apps, and users of the VDI instances, you can simply map these requirements to the instance types available in the Google Cloud region of your choice. Don't worry if you don't have a perfect match between available instance types and workload requirements. Google Cloud supports custom instance types, which allow you to tweak the shape of your VDA instances as you go. Sustained use and committed use discounts still apply to custom instance types, so don't let that deter you from adjusting as needed to get the right size up front.

Also - Google Cloud's sizing recommendations feature can be used to identify adjustments to VDA shapes you may want to make over time.

Note:

One important thing to note - workload resource consumption can change over time. Sometimes events/activities reduce resource requirements - like when an administrator applies optimizations to the environment. Conversely, sometimes these requirements go up, like when an OS or app vulnerability is patched, or an update is applied. Find your baseline, but it's important to monitor consumption trends over time and adjust as necessary to find the optimal balance between user performance and costs.

When selecting the right instance size for hosted shared VDAs, things get a bit more complicated. What you're ultimately searching for is a moving target - the right balance of performance, cost, and manageability. To further complicate things, every workload is different. Variances between OS, applications, settings, tuning, and user expectations make it tough to nail down the right shapes for your VDAs. It also tends to change over time as well.

Fortunately, the tools and techniques for finding that ‘goldilocks' balance between performance, cost, and manageability are well known and thoroughly documented. One excellent article we'd recommend starting with is Citrix Scalability in a Cloud World 2018 Edition. This article is still relevant today as it discusses leading practices regarding instance selection based on performance, manageability, cost, available pricing models, and LoginVSI scalability testing. These concepts and considerations are still valid today, even though instance choices and pricing have likely changed since its initial publication.

Another article worth mentioning is Right-sizing Citrix on Google Cloud Platform. Although a bit dates, this article digs even deeper into the considerations and provides an example of how to find the most cost effective instance type based on single VDA scaling and your instance costs.

Finally, for extra insight into strategies for optimizing VDA costs, see this Autoscale tech brief on Citrix TechZone. It helps align your instance cost estimates with the capabilities of the Citrix Autoscale feature, including the use of vertical load balancing.

Speaking of Citrix Autoscale - read up on it and use it: with a bit of thought and clever design, you are able to cost optimize your VDA fleet while ensuring you've got capacity available to handle expected and unexpected fluctuations in system demand.

Speaking of demand patterns - you want to invest some time and resources into understanding the unique patterns of each workload. Expect them to change and evolve over time, and be prepared to adjust your capacity management strategy and tactics to accommodate.

Performance Tuning

Many factors can contribute to the perception of performance users get on your Citrix virtualization system. Besides selecting the right shape of VM instance for your VDAs, other key areas to investigate for performance optimization include the following:

User environment and settings management choices: Policies are your friend when managing and optimizing performance for your users. Policies control the configuration of Windows, applications, sessions, and much more. To further complicate things, there are multiple policy engines you can potentially use, each with their own impact on the user experience. Choosing the right policy engine and establishing consistent baselines are important, and fortunately are covered in depth in Baseline Policy Design. Also, the Citrix Workspace Environment Management service can be used to optimize the user experience and simplify management in a diverse environment.

Optimizing Windows: Windows is a general purpose operating system, and is designed to cover a broad variety of use cases, hardware types, and so on. Many of the default settings in Windows are unnecessary in a Citrix virtualization environment. Fortunately Citrix Optimizer is available to help, and includes many comprehensive templates you can apply to your VDAs to get the best possible performance and lowest overall resource utilization.

Antivirus tuning: Running antivirus software on Citrix VDAs and supporting infrastructure is a solid and recommended practice. However if incorrectly installed/configured, it can wreak havoc on the user experience. See Endpoint Security and Antivirus Best Practices for a great primer on how to get it right.

HDX Protocol Optimization: Citrix's HDX display protocol stack does many auto-detecting and adapting on the fly to provide the best possible user experience. It attempts to balance performance, responsiveness, and richness of the UX with bandwidth consumption. For some use cases (such as graphics intensive workloads, or low/poor quality network connections) often benefit from some fine-tuning to get the balance right. See HDX Graphics Overview for more information.

Also, Citrix provides several pre-built session policy templates which can be used to flexibly match settings to your specific use cases. These policies are configured and managed in Citrix Cloud Studio, and can be applied using various filters. These filters allow you to make sure the right policy is applied to optimize specific scenarios.

Choosing the right Pricing Models

Google Cloud offers various different pricing models customers can use for the different types of workloads you run there. Understanding the demand patterns for different use cases can help you choose the right model for each resource to balance cost and service availability/performance. In a Citrix virtualization system, customers commonly consider sustained use vs. committed use discount models for the resources that run on GCP. Sustained use discounts can vary between instance types (N1 vs. N2, for example) and some instance types (such E2) don't offer sustained use discounts. See VM instance pricing for more details.

The following is a simplified chart illustrating sustained use vs committed use discounts for N1 instance types:

Some resources are unique, highly scalable, and must be available for a Citrix virtualization system to function. As such, they're commonly run 24/7 and deployed N+1 for availability, and are great candidates for committed use discounting. This includes Active Directory, Citrix Cloud Connectors, Citrix ADC/Gateway VPX, and Citrix StoreFront VM instances.

For VDA instances, the choice isn't quite as simple, but the more clearly you understand your demand patterns, the clearer the choice is. It all boils down to how long the VDA needs to be powered on for. Consider the following chart (specific to N1 instance types), which is reproducible with a bit of back-of-the-envelope math:

This diagram shows that if a resource (running on an N1 instance type) will be on for over 50% of the time during a given billing cycle, you start saving money if you can apply 3 year committed use discounting. The break even point on a 1 year committed use discount is approximately 82%. If a resource is going to be powered on for more than that during a billing cycle, and 3 year committed use isn't available, then a 1 year committed use makes sense.

File Storage and Data Replication

Most Citrix virtualization systems on GCP require at least basic access to a Windows compatible file share to persist user settings, user data, and application data. Windows file shares are also used to store Citrix user personalization layers. When these shares are not available, the user experience and application functionality suffer. It is important to ensure that whatever solution you choose to provide, Windows compatible file shares are highly available and data is regularly backed up.

For multi-site deployments, reliable and performant data replication may also be necessary to meet availability, RPO, and RTO needs. This is especially true for environments where users can connect to desktops/apps in 2 or more regions, and application data/user settings must be available in the region where the apps/desktops run. The following section describes some solutions to consider for providing file storage and data replication services on GCP.

While non-Windows solutions for providing Windows file shares exist, most of these solutions cannot deliver the indexing capabilities required for search functionality inside a Windows desktop or applications such as Microsoft Outlook running on Windows. As such, most customers turn to Windows-based file server solutions, at least for storing user profiles and persistent application data. Fortunately, both customer managed and cloud service options are available for use when Citrix virtualization systems are run on GCP.

Customer Managed: Windows File Servers on Google Compute Engine

The first solution many customers consider for providing Windows compatible file services on GCP is building their own Windows file servers on Compute Engine to serve each resource location on GCP. Since Windows file servers are needed by various different types of applications and workloads, many IT shops can gravitate towards building and managing their own since this is something they know how to do. At the most basic level, the customer creates one or more Windows instances, attaches more persistent disks, joins the instances to their Active Directory, and finishes configuring Windows File Services.

This option, as you might imagine, provides customers with the most control and flexibility. While this is very appealing to certain types of customers and certain verticals, it also comes at a cost: the responsibility to size, scale, build, manage, patch, secure, and maintain everything from the Windows OS up. Customers electing to go this route ought to also ensure these file servers are highly available. This is often accomplished using file servers in multiple zones, and using Windows DFS-N/DFS-R, Windows failover clusters, or storage spaces direct. It's easy to end up in an unsupported configuration (per Microsoft) if you're not careful.

Note:

Customers considering this option ought to review Microsoft's support statement regarding using DFS-R and DFS-N for roaming profile shares and folder redirection shares.

Third Party

An alternative solution is using third party solutions such as NetApp Cloud Volumes ONTAP or the Cloud Volumes Services for GCP. Both solutions allow you to create compatible SMB shares that can be used for your storage needs. There are benefits to using third party storage solutions as opposed to managing your own Windows File Servers, such as less administrative overhead when managing storage. See File servers on Compute Engine for more information.

Citrix NetScaler VPX on Google Cloud

Deploying the Citrix NetScaler Gateway on GCP is different than deploying it on-premises, though in the end you're managing them yourself. Fortunately deploying Citrix NetScaler on GCP is thoroughly documented. We recommend reviewing the following resources before you solidify your design and begin implementation:

• Citrix NetScaler VPX on GCP in Citrix Docs: Provides a comprehensive overview of Citrix NetScaler on GCP, including supported VPX models, GCP regions, Computer Engine instance types, and other resource references.
• Citrix NetScaler VPX GCP Marketplace Deployments: All available Citrix networking deployment solutions available in the GCP Marketplace. Functional and relevant for Citrix Gateway deployments with Citrix Virtual Apps and Desktops and Citrix DaaS also.
• Citrix NetScaler GDM Templates: A GitHub repository for Citrix NetScaler GDM templates. This is an excellent reference for a repository that hosts Citrix NetScaler templates for deploying a Citrix ADC VPX instance on the Google Cloud Platform.

As discussed in Citrix NetScaler VPX on GCP on Citrix Docs, there are two primary deployment options available. They are:

• Standalone: Individual instances of Citrix NetScaler Gateway can be deployed and managed as separate entities. This is commonly used for smaller scale or POC deployments where high availability is not a requirement.
• High Availability: This is the most commonly deployed model for production environments: pairs of Citrix NetScaler Gateway VPX instances can be deployed using an HA configuration within the same zone or across multiple zones in the same region. We dig into this option more deeply later in this section.

Best practice:

When you deploy Citrix NetScaler Gateway appliances on GCP, we recommend using Premium tier (regional) external IP addresses. When using premium tier external IP's, traffic ingresses and egresses at the Edge network location nearest the user. Traffic then traverses Google's private network to get to the region where the resource is deployed. This provides better throughput, lower latency, and more consistent performance (lower jitter) as compared to Standard tier external IP addresses. For more information, see Google Cloud Network Service tiers.

NetScaler Standalone

While Citrix NetScaler VPX generally supports single, dual, or multiple NIC deployment types, Citrix recommends using at least three VPC networks for each NetScaler when deployed on GCP, with a network interface in each VPC for optimum throughput and data separation. When deployed to support Citrix Virtual Apps and Desktops, the management interface (NSIP) is typically attached to the "Private Citrix Infrastructure Subnet," the subnet IP (SNIP) is attached to the "Private Citrix VDA Subnet," and the Citrix Gateway virtual IP (VIP) to the "Public Subnet." The following simplified conceptual diagram depicts this configuration. It shows a single VPX instance in a single zone - this design pattern would be duplicated (likely in a second zone) for a High Availability configuration:

The following is a table showcasing the purpose of each NIC along with the associated VPC network:

Important:

Citrix NetScaler VPX instances with three NICs require a minimum of 4 vCPUs when running on GCP. See maximum number of network interfaces for more information.

NetScaler High Availability across Zones

As mentioned earlier, this is the most common deployment model for Citrix virtualization systems. This model uses a pair of Citrix NetScaler VPXs in a single region deployed across multiple zones. High availability (active/passive) can be achieved multiple ways. You can use a GCP HTTPS Load Balancer with the NetScalers configured independent of each other or by using Citrix NetScalerss HA configured in Independent Network Configuration (INC) mode. The latter option/architecture is expected to be popular for public cloud deployments, so we focus on that here.

While there are potential variants for a Citrix NetScaler Gateway VPX architecture on GCP, the following diagram depicts a three NIC Citrix NetScaler HA solution. This solution can be deployed by the Google Deployment Manager template with pre-configured VPC networks and subnets:

When using the Google Deployment Manager template, you must configure the VPC networks before deploying the Citrix NetScaler appliances. The three VPC networks ought to consist of the (❶) management network, (❷) public network, and (❸) backend-server network and appropriate subnets within each VPC network.

In the preceding diagram, we can see that each NetScaler has a different Gateway virtual IP (VIP). This is a characteristic of an Independent Network Configuration (INC). When VPXs in an HA pair reside in different zones, the secondary NetScaler must have an INC, as they cannot share mapped IP addresses, virtual LANs, or network routes. The NSIP and SNIP are different for each NetScaler in this configuration, while the Citrix Gateway VIP uses a Citrix NetScaler feature called IPset, or Multi-IP virtual servers. This feature can be used for clients in different subnets to connect to the same set of servers. With IPset, you can associate a private IP to each of the primary and secondary instances. A public IP can then be mapped to the primary ADC in the pair. In the case of failover, the public IP mapping changes dynamically to the new primary.

For more information on adding a remote node to an NetScaler to create an INC-based HA pair, see Citrix docs. For general HA deployment information for NetScaler on Google cloud, see Deploy a VPX high-availability pair on Google Cloud Platform.

This document is intended for technical professionals, IT decision-makers, partners, and system integrators. It allows the administrator to explore and adopt Citrix App Layering to aid with managing the delivery of images and applications to their end users. The reader must have a basic understanding of Citrix products, image management services, and application virtualization concepts. For more information on image management, refer to the reference architectures on Citrix Tech Zone.

Objective of this document

This document provides a technical overview and architectural concepts for managing and delivering applications using Citrix App Layering technologies. This document includes how app layering works and how to integrate it with Citrix virtual apps and desktops on different platforms.

Citrix App Layering Overview

Citrix App Layering is a flexible solution for providing a complex set of Windows applications to a diverse set of users on any non-persistently supported platform.

Citrix App Layering uniquely performs image management. The operating system and applications are split into different containers called “Layers.” Layers are created and updated independently, then compiled into “published images” to be distributed using any supported provisioning system.

Once the application libraries are created, different sets of images are deployed to many platforms using any combination of layers.

The primary goal of Citrix App Layering is to simplify Windows application management using a single interface. It allows the administrator to create and manage enterprise applications regardless of the underlying hypervisors or cloud infrastructure.

The OS and applications are separated into discrete, manageable units. Even if many images are required to control application access properly, each operating system and its applications are managed as a single instance. In this approach, updates don’t have to be performed on every image in the environment. Simplifies the environment while reducing management time, complexity, and costs associated with operating systems and app management.

The OS and applications are stored as a virtual disk that contains files and registry entries for a specific layer. There are two ways to include applications within virtual machines:

• Published image: This method combines the OS layer, a platform layer, and a set of application layers to create an image for provisioning systems like Citrix Provisioning or Citrix Machine Creation Services. App Layering can publish images to multiple provisioning systems and multiple hypervisors from the same set of layers.

Learn more about Citrix image management, refer to the reference architecture document.

• Elastic Layers: Here Application layers are dynamically attached to a virtual machine during the log-on based on AD group membership and application assignment. This allows for greater flexibility in application assignment by allowing dynamic delivery of applications to standardized images.

Separating the applications and personalization from the operating system provides a flexible and manageable solution to non-persistent image management.

Why Citrix App Layering

Citrix App Layering provides a cost-effective solution to manage a complex set of applications across multiple environments with different packaging requirements. Almost all applications that are embedded with kernel drivers, OS, and system service dependencies are compatible with App Layering technology.

There are many advantages to adopting the Citrix App Layering approach for image management:

• Simplifies master image management for PVS and MCS: Citrix App Layering is a single image management solution that supports both provisioning models used with Citrix and third-party hypervisors. Using App Layering both management and upgrades for images are simplified with no direct editing or reverse imaging of images.
• Azure support: Citrix App Layering supports Microsoft Azure and makes it simple for App Layering customers to migrate to an Azure platform.
• Decouple the apps and provisioning systems from the image: Citrix App Layering separates packaging from the image. In normal image management updates to an image are performed for each image separately. Using App Layering a layer can be part of many images. To update all the images the layer might be updated once, and the images regenerated. Upgrades to components like hypervisor tools, Virtual Delivery Agents (VDA), and Citrix Provisioning tools become easy.
• Supports complex use cases: Complex applications with kernel drivers, systems services, third-party drivers, and console access can all be supported using Citrix App Layering. Due to App Layering’s two modes for app layer deployment almost all applications are compatible.
• Non-persistence desktop User Layer: The User Layer is a writable Elastic Layer that provides users with a persistent type experience while using a non-presistent image. The user layer allows a user to install applications and save configuration settings that are outside the user profile.
• Reduces the number of required images: Elastic Layering can significantly reduce the number of required images by dynamically delivering applications to only assigned users at logon. Elastic Layers are compatible with both Citrix Virtual Apps and Citrix Virtual Desktops.

Citrix App Layering Use Cases

App Layering is an important addition to the Citrix technology portfolio that provides many benefits listed previously. Though it provides many benefits, App Layering is not meant to be used for all the use cases. In this section, several of the most relevant use cases are outlined to show the benefits of App Layering technology.

1-Too Many Images to Manage

Many Citrix customers must support a significant number of applications and a complex set of user requirements. When using an image-based provisioning technology, meeting these requirements often requires a high number of images to manage. These images have to support different user groups or different sets of conflicting applications. Often there is some overlap in the applications deployed to each image as well.

Citrix App Layering simplifies the management of this complex scenario.

App Layering allows the administrators to manage the operating system and applications as individual entities using layers. For example, Windows needs to be patched, the patch is made to the OS layer once and all the images that use the OS layer can be updated by the App Layering appliance at the same time. If Microsoft Office is used in 10 images, upgrading this Office is simpler by adding a version to the Office layer. Eventually all those 10 images are updated automatically.

When Elastic Layering is added, it is also possible to significantly decrease the number of required images. Elastic Layers allow admins to dynamically deliver applications during log-on. For applications that are not used by everyone, Elastic Layering allows the image to be created more generically while providing customization for each user.

2-Support for Multiple Hypervisors and Cloud Providers

Many organizations are moving to a hybrid multi-cloud environment mixing on-premises and cloud resources to enhance the user experience. Citrix App Layering provides image portability by supporting most hypervisors plus Microsoft Azure and Google Cloud by simply by creating a different Platform Layer for each desired environment.

Normally a mixed hypervisor and hybrid cloud configuration forces the administrators to maintain multiple different sets of images and applications in multiple different management platforms. With App Layering technology, the same OS and applications used on-premises can be pushed to the cloud or to another hypervisor with little to no extra work.

To understand App Layering cross-platform functionality, refer to the Citrix App Layering Cross-Platform Support section below.

3- Hypervisor Portability for Migration

Organizations often get “stuck” with a vendor’s technology simply because it is prohibitively expensive to migrate to new technology. Also, companies often merge with other organizations that have made different technology choices. One of the distinct advantages of App Layering is the ability to move from one platform to another simply by creating a different Platform Layer and migrating the App Layering appliance using import/export to the new platform.

This allows for a seamless migration, for example, from vSphere to Citrix Hypervisor or from an on-premises hypervisor to Azure.

4-Persistence for shared VDI Desktops

Many organizations have several users that require a high level of persistence on desktops. Including power users in any group, developers, engineers, architects, and so on. App Layering User Layers provide a significant amount of persistence on top of a pooled desktop architecture.

The User Layer is mounted on logon and any subsequent writes on the desktop are written to the User Layer. Most applications can be installed in the User Layer. The rules for what works in the User Layer are the same as for Elastic Layering. As long as the application does not install kernel drivers, third-party drivers, and the services that are dependencies to other services during boot, they will most likely work in the user Layer.

For use cases that require persistence, the User Layer is the best choice. For use cases to support just Microsoft Outlook OST files and index files, the User Layer may not be the best choice. The User Layer is intended to handle all writes to the VDI desktop after the user logs on. Other technologies like the Citrix Profile Management Outlook Container or FSLogix Profile or Office 365 Container handle the Outlook OST and indexes in a much more targeted manner so that the amount of I/O handled by the container is smaller. All of these solutions now handle managing the Outlook OST, Outlook streaming files and Outlook index files so that supporting indexing is no longer a reason to choose one technology over another.

Technical Overview of Citrix App Layering

Types of Layers

A Layer is a virtual disk containing the files and registry entries that are changed or added during packaging. Excluding the first version of the Operating System layer, layers are created by the App Layering appliance integrated with the hypervisor. An administrator creates a layer, the appliance dynamically provisions a packaging machine with a boot disk and a layer disk stored on the App Layering appliance and accessed using iSCSI. When packaging is complete, this disk is finalized and all the files and registry changes are written into the new layer and stored in the layer repository in VHD format.

App Layering uses these different types of layers:

• Operating System Layers
• Platform Layers
• Application Layers
• Full User Layers

OS Layer: The operating system like Windows 10 or Windows Server 2019. Built from a “Gold Image” VM in the hypervisor.

Platform Layer: The platform layer includes the software required to support a particular platform. Including the broker agent, provisioning system, and hypervisor tools if the hypervisor is different from the default hypervisor. The platform layer is also the highest priority layer and sometimes software is installed here so that it is compiled at the highest priority.

App Layers: The main layer type. Used for most application software.

User layers: An elastic (see next section) writable layer. User Layers are mounted at logon and once mounted almost all the desktop writes go to the user layer. This layer gives users the ability to significantly customize their VDI experience even though they are using a shared desktop model.

Citrix App Layering appliance

The Citrix App Layering appliance (appliance) provides both the administrative interface for App Layering and the engine for all App Layering processes.

The App Layering appliance is deployed as a virtual machine into the data center where application packaging and image publishing take place. The App Layering appliance is built on CentOS, configured with 4 vCPUs and 8 GB of RAM. These settings are not to be changed as the appliance is designed to work in that configuration.

The appliance is built with two disks. The first disk is a 30 GB boot disk for the operating system. The second disk is the 300 GB layer repository. This disk can be extended or expanded as necessary if more space is required.

During the process of layer creation and image publishing, the Citrix App Layering appliance saves virtual disk files in VHD format to its layer repository within the appliance. The appliance interfaces with an SMB share to support the appliance upgrade process and store Elastic Layers.

The appliance is used only to manage layers, images, and Elastic Layer assignments. Virtual Desktops and Virtual App servers do not interface directly with the appliance. When a layer is assigned elastically, the appliance copies the layer to the Elastic Layer share. The share holds these VHD files plus a set of JSON files that provide the layer assignments to users.

The Citrix App Layering appliance also hosts the App Layering management console. Deploying the App Layering appliance is the first step in the installation process. After installing the appliance, the management console is accessed to complete the installation steps.

Learn more about installing the Citrix App Layering appliance, refer to the product documentation.

The Citrix App Layering management console is a web-based application hosted on the App Layering appliance. The App Layering management console provides the interface to:

• Create and manage Operating System, Platform, and Application layers
• Create published image templates
• Publish and manage Layered Images
• Assign App Layering the administrator roles to users
• Manage the appliance and system settings such as task and log retention, security settings, and network file shares

Compositing Engines

Compositing Engines offload most of the packaging and publishing tasks that can also be performed by the App Layering Appliance. By offloading these tasks the packaging and publishing processes scale better and due to the advantages of the technologies used, the performance of the process is also enhanced.

A Compositing Engine is built by a Hypervisor connector as a Windows PE virtual machine that carries out a set of publishing tasks, then reboots itself into a packaging machine or published image. The Compositing Engine is used to create cached layer disks, create packaging machines and publish images. Currently all supported hypervisors have support for offload compositing connectors.

Compositing Engines have the following characteristics:

• Lightweight, ephemeral appliance running Windows PE
• Self-compositing
• Controlled via REST API
• Hypervisor disk formats supported (VHD, VHDX, and VMDK)
• UEFI support (Gen2)
• Secure boot on published images (only if no Elastic Layering or User Layers are used)
• iSCSI is used to attach disks hosted on the App Layering Appliance
• No Limit on the Number of simultaneous publishing jobs (there is a practical limit)

Advantages of Compositing Engines

While the use of Compositing Engines is a choice they offer such significant advantages that they should always be used. The significant advantage to the Compositing Engine is that it is running on a Windows device with direct access to hypervisor disks. This provides the mechanisms to support GEN2 machines in HyperV, native ESX VMDK formats with Thin Provisioning in vSphere and UEFI in both. Packaging and Publishing performance is enhanced because the large layer files are processed less and written directly into disks on the hypervisor by the Compositing Engine which attaches back to the App Layering Appliance to access the layers using iSCSI connections.

Note: Even the ability to support VHDX disks in PVS is made possible by integrating the composting engine connector with the PVS connector.

Connectors and Connector Configurations

The App Layering appliance integrates with hypervisors and provisioning systems using connectors.

There are two types of connectors - hypervisor and provisioning:

• Hypervisor connectors provide the mechanism to interface with a hypervisor. Currently, there are hypervisor connectors for XenServer, vSphere, Hyper-V, Nutanix AHV, Azure, Azure Gov, and Google Cloud Platform.
• Provisioning System connectors allow the admin to publish an image to the provisioning system. Currently, there is a Provisioning System Connector for Citrix Provisioning, and a Citrix Machine Creation Services on each hypervisor.

A single appliance can connect to any number of hypervisors or provisioning systems by having more connectors defined. Connector configurations define the settings required to integrate with the hypervisor or provisioning system. A configuration typically includes credentials for authentication, a storage location, a VM template, and any other information required to interface with the environment where the administrators are creating layers or publishing images. The administrator can create multiple connector configurations, each configured to access a unique location in the environment. Connectors are used for:

• Importing a Gold Image when creating an OS layer
• Creating layers and layer versions
• Publish layered images to a hypervisor or provisioning service.

Reference: Connector Configurations

Connector Types

This section defines the connector types available in Citrix App Layering.

Hypervisor Connectors

Hypervisor connectors are used when creating layers or importing a Gold Image to create the OS layer. When packaging dynamically, they create a packaging machine on the storage and host defined by the connector configuration. A hypervisor connection can also be used to create a virtual machine in the hypervisor.

Citrix Provisioning

Citrix Provisioning connector integrates with an App Layering Agent on a Citrix Provisioning server to publish an image directly to the Provisioning server as a virtual disk. The prerequisites for Citrix Provisioning are:

1. The connector service account must be a domain account with the administrator permission within PVS.
2. The connector service account must also be a local administrator on the Citrix Provisioning server.
3. A Citrix App Layering Agent must be installed on each Citrix Provisioning server defined in a connector. Only one agent can be defined per connector.

Reference: Citrix Provisioning

Machine Creation Services

The Citrix App Layering appliance can directly provision and publish layered images to hypervisors as virtual machines that are used as the Master Image for MCS. Connectors allow layered images to publish directly to the hypervisor. The MCS connector is almost identical to the Hypervisor connectors except that the MCS connector starts the published virtual machine which allows any scripts defined in layers to run on the Master Image before it is deployed by MCS. The Master Image shutdown and a VM snapshot taken as part of this process.

Published Image Templates

When using App Layering, OS, Platform, and Application layers are merged by the App Layering appliance to create published Images. Images are published and then created in a format required by the target provisioning system. For example, in case the administrators are publishing to Citrix Provisioning, the appliance creates a VHD or VHDX and uploads it to the defined provisioning server as a virtual disk. For Machine Creation Services on Citrix Hypervisor, the appliance creates a VHD, uploads it to Citrix Hypervisor and creates a Master Image VM using the VHD.

To define the configuration of a published Image, an Image Template is used. The image template defines which OS, Platform, and Application layers are included in the image. It also defines the connector used to publish the image, how large the resulting image is in GB, whether the image has Elastic Layering enabled or if it includes a User layer. The image templates are a point-in-time snapshot of the image configuration, they do not support versioning. However, Image Templates can be cloned and modified if slightly different versions of the same image are required.

App Layering Agent

The Citrix App Layering agent provides communications between the App Layering appliance and either a Citrix Provisioning server, or a Hyper-V server. The Citrix App layering agent details can be found in the Citrix documentation.

Reference: Install Agent

Active Directory

The App Layering appliance connects to Active Directory for both authentication to the appliance and assignment of Elastic Layers. When an administrator logs into the appliance it attempts to log on to Active Directory with the same credentials. If that logon works, the user is allowed into the appliance.

Access to directory services is required for the following purposes:

• Role-based access control (RBAC)
• Assignment of Elastic and User Layer

The App Layering appliance is compatible with SSL 3.0 Secure Socket Layer and TLS 1.1 and 1.2 transport layer security during the directory service's initial binding.

To create a directory junction, refer to the product documentation.

Layers

The following section describes the uses for each type of layer in more detail.

OS Layer

An OS layer is one that contains the Windows operating system. It is a leading practice to include any components that might be updated with Windows Update in the OS layer so that all are updated by Windows Update. All operating system roles and components such as .NET and Visual C++ runtime libraries are included as part of the operating system image for this reason.

It is preferable not to install end-user applications into the OS layer because all application layers made with a particular OS layer are tied to that OS layer. If an application is installed in the OS layer, then every image using that layer will have that application included. This process leads to a problem when the strategy is to make the OS layer universal. Separation of applications from the operating system is the key to limiting the number of OS images to manage. Even applications with drivers, services, and kernel devices are supported in application layers and do not need to be included in the OS layer.

During the creation of the OS layer there are a few points to remember:

• Check the supported operating systems from the link
• The OS layer is not connected to the domain. The domain join is part of the platform layer
• The hypervisor tools for the primary hypervisor are installed in the OS layer. The hypervisor where most packaging is performed
• The hypervisor tools for alternate hypervisors are installed into the Platform Layer for that hypervisor
• Install .NET components (from Microsoft) and updates in the OS layer
• If Microsoft Runtimes are required, they are installed in the OS layer
• Windows roles and features such as the RDS roles installed in the OS layer so that it can be updated with Windows Update
• In case local user or group must be added to the virtual machines, this task can only be done in the OS layer because the Windows Security Account Manager (SAM) captured from the OS layer

Reference: Create an OS layer

Platform Layer

A Platform Layer is a layer that contains configuration settings, tools, and other software necessary to run a published image on a particular platform. Two platform layer types can be created:

Publishing Platform Layer: This type of Platform Layer is used to include the software required by a target provisioning system, broker, and hypervisor. For example, a publishing Platform Layer to support Provisioning Services on vSphere for Citrix Virtual Apps have the Citrix VDA, and PVS Device Drivers installed. If vSphere is not the same hypervisor where packaging is performed, then this layer also has VMware Tools installed.

Packaging Platform Layer: Packaging Platform Layers are used if a packaging layer is required to support packaging machines. These layers are not often required, but there are several instances where one has to be necessary including:

1. If a layer has to be packaged on a different hypervisor than the standard. For example, if most layers are created on Hyper-V, but for some reason, a particular layer created within vSphere, a packaging platform layer with VMware Tools are used to support the packaging machine on vSphere.
2. If access to the packaging machine is required using VDA software. This layer is most often required when installing drivers for USB peripherals that must see the device to install properly. By creating a packaging platform layer with the VDA software installed and adding the packaging machine to a manually provisioned catalog in Studio, this type of access can be supported.

Creating a Platform Layer

For the creation of the Platform Layer a few points to be remembered:

• For Citrix Provisioning, the target device software is installed without running the Imaging Wizard and then rebooted
• Installation of Citrix VDA and Citrix Provisioning device drivers goes in the Platform Layer
• Domain join is performed in the Platform Layer - that means multiple Platform Layers can be created to support different domains
• The platform layer is also the highest priority layer, so some optional components can be installed including SSO software like Imprivata, hypervisor tools for alternate hypervisors, and the Citrix Workspace Environment Management Agent.

Learn more about the Platform Layer creation, refer to the article CTX225997.

App Layer

App Layers are used to package most applications. App Layers contain file system and registry objects for an application or group of applications. When creating or editing a layer, a packaging machine is dynamically created and all filesystem and registry changes are captured on that machine. The Packaging Machine contains the OS Layer and any included prerequisite App Layers. A second virtual disk called the Package Disk is attached to the packaging machine as the writable volume. That disk captures all the filesystem changes during packaging, and it also contains a registry hive (called an RSD hive) used to capture all the registry changes. When the packaging machine is finalized, only the Package Disk is saved as the layer. The Boot Disk is deleted.

Learn more about creating and editing App Layers refer to the product documentation.

Most of the time applications are installed with the same configuration that the administrator might normally use for the provisioning system that is supported. While creating App Layers, it is always important to work out to disable automatic updates. Because the administrator usually does not want applications to automatically update after deployment. Citrix has documented the layering process for a few common applications. These App Layering “Recipes” can be found at the following link.

Reference: App Layering Recipes

Elastic Layering

Elastic Layering is a method for dynamically deploying the application to a virtual session during user logon by mounting layers stored as VHD files on an SMB share and integrating them into the file system using the Citrix Composite File System (CFS). Elastic Layering is managed on the VDA by the Citrix Layering Service. The Elastic Layer Repository is an SMB share that contains both the layer VHD files and JSON configuration files which are used to define layer assignments for the Layering Service. The Layering Service reads the configuration files and then mounts layer VHD files using a Windows SDK call.

All applications are not supported by Elastic Layering technology because Elastic Layers are mounted during logon. The following application requirements exclude applications from being used within Elastic Layers:

• Applications with Kernel drivers
• Applications with services that are dependencies for other boot-time services
• Applications that modify the Windows Driver store like third-party printer drivers

Applications with these requirements can be layered but the layers must be included in the published image.

Configuration files (JSON)

As discussed previously, Elastic Layer assignments are defined in a set of .JSON files stored on the Elastic Layer Repository. These files are defined as follows:

• ElasticLayerAssignment.json: This file contains information about the user and group mapping to application layers. This file contains an entry for each group or user ID that has assigned applications and under the SID for that AD object the layer assignments are listed.
• Layers.json: This file defines the layers in the repository and metadata about the layer. This file is used to obtain the path used to mount the VHD.
• MachineAssociations.json: As the name suggests it defines machine association with any AD group, the format uses machine name patterns containing wildcards to associate a set of computers with a defined AD group. When a user logs on to a machine that matches the pattern, they receive the Elastic Layers assigned to that group. These settings are defined in the App Layering Management Console in the Users>Groups section.

User Layer

User Layers provide a more persistent experience for users while still supporting a shared desktop computing model. After a User Layer is mounted most system writes are redirected to the User Layer. This layer allows support for the following:

• Each user’s profile and data settings are stored in the User Layer
• User installed applications are supported in the User Layer as long as the applications conform to the rules allowed by Elastic Layers

User Layers are assigned one to one. One user can have only one user writable layer per OS layer per domain. The user can therefore only log on to one delivery group or pool with a desktop using the same OS layer and Platform Layer combination with the User Layer enabled. This layer is created as a virtual disk on a file share when the user logs on for the first time.

The Full user layer supports search index persistence between sessions. To support this, the Windows Search service is set to DISABLED (4) on VDAs when they boot. When the user logs on, the Ulayer Service changes the start type of the Windows Search service to START_ON_DEMAND (3). Before enabling WSearch, the ULayer service must also ensure the indexer's "crawl-scope" registry settings are correct. The crawl-scope is a set of registry keys that determine the areas of the user's data to be indexed. Input into the crawl-scope comes from defaults built into the base image, but can also come from the elastic layers and the user's persistence layer settings too. These inputs are processed at logon time to provide the complete set of crawl-scope locations, and there is a modest but measurable overhead to building this for each logon. To avoid this overhead, the ULayer service generates a hash string to represent the base image deployment (for example, the BIC instance) and the elastic layer assignments, and stores this string in the user's \Program Files\Unidesk\Etc\UserLayer.json file as "IndexerHash". On subsequent logons this string is compared to the recalculated IndexerHash and only when they differ is the crawl-scope rebuilt.

The default max size of a User Layer is 10 GB. This size can be altered by defining a quota for the User Layer share. It is also possible to override the default User Layer max size using a registry entry on the VDAs. To change the default max size, add the following registry override,

Elastic Layer Share and User Layer Shares

Elastic Layers are VHD files mounted by a client or server operating system over the VM network. Elastic Layers are mounted as read-only, and many machines can mount the same VHD file. The file server or share used for Elastic Layers are optimized for the read I/O.

User Layers are writable Elastic Layers. The User Layer is mounted read/write by only a single desktop. The file server or share used for User Layers are optimized for write I/O. When using User Layers performance of the storage is critical to the user experience. Flash storage arrays are highly recommended for the User Layer share.

The architecture for Elastic Layers is largely scalable. The Elastic Layer share repository path is defined in the VM HKLM registry for both VDI and RDS workloads. This design makes it possible to have an unlimited number of replicas to spread the load.

The Elastic Layer Repository and User Layer Shares are defined in the appliance. The Elastic Layer share is defined in the System>Settings and Configuration section. This path can be changed on VDAs using Group Policy by modifying the following registry values:

For large VDI implementations, this registry value allows multiple Elastic Layer repositories to be used for different sets of desktops. Learn more about changing the Elastic Layer repository in the registry without reimaging refer to the Citrix article CTX222107.

The location used for User Layers is assigned by the Active Directory group and therefore it is also highly scalable because as many shares as desired can be used. These assignments are defined under System>Storage Locations.

For more information on the architecture of the Elastic Layers share see the Availability, Backup, and Recovery section.

User Layer File Share

During the login process when an image is configured for User Layers, the User Layer VHD/VHDX file is created at the first time the user logs on after being assigned to the image. User Layer share settings are defined in the appliance by Active Directory Group. If a user is in multiple groups with assigned User Layer shares, there is a priority order to the share and their User Layer file created on the highest priority share. User Layer disks can only be used on one machine at a time.

User Layers are tied to both the domain and OS Layer for the desktop being logged into. The path to a particular User Layer as follows:

User Layers are write-intensive. It is recommended to use a file share that is optimized for writes. If the User Layer share is different from the Elastic Layer share, then user assignment is defined by AD user groups.

User layer assignment is defined in the System>Storage Locations section within the App Layering Console. Enter the share and the group associated with the share.

For more information on the architecture of the Elastic Layers share, see the Availability, Backup, and Recovery section.

App Layering Integration

The following section outlines the integration of Citrix App Layering with Provisioning Systems and Hypervisors.

Citrix App Layering and Citrix Provisioning Services

App Layering is easy to integrate with Citrix Provisioning (PVS). Integration is facilitated by installing the App Layering Agent on one or more PVS servers. The Agent provides the connectivity between the appliance and Provisioning Services.

The preceding diagram depicts publishing layered images using Citrix Provisioning. Images are centrally managed and distributed for deployment by publishing from the App Layering appliance to a Citrix Provisioning server. There are several architectures that can be created to support Citrix Provisioning. The most often used architecture is to integrate a single production PVS server that is used as the integration point. Virtual disks are then transferred to the PVS server store by the App Layering Agent when the image is published. The Agent uses the Windows BITS service to perform the transfer.

Consider using  a “DEV” Citrix Provisioning farm with a single server to interface with App Layering. This “DEV” server can also be used to stream test images. Once the image has been tested the virtual disk can be replicated to the Production Citrix Provisioning farm for further testing and deployment. or used by the Citrix Image Portability Service to migrate images to different platforms.

When an image is published to Citrix Provisioning, the image is named according to the Image Template Name with a date and time stamp for versioning. Virtual disks are managed in Provisioning Services the same way they might be managed without App Layering. Versioning is not used when App Layering is used. Instead, each time a new image is published a full virtual disk is created with a new date and time stamp. If an emergency change to the image is needed, Citrix Provisioning versioning can be used to quickly modify the image. Then the same change can be added to the appropriate layer after the issue has been quickly fixed.

Elastic Layering’s Impact on the Citrix Provisioning Cache Disks

Citrix Provisioning uses reserved memory and a locally attached cache disk to temporarily store local filesystem changes during a session. While using App Layering and Citrix Provisioning, the cache disks size must be larger than without App Layering. Whenever a file is opened, for write operation using App Layering the entire file is copied into the writable volume of the virtual machine so that it can be edited. Causes the entire file to also be copied into the disk cache when Citrix Provisioning might normally only copy the modified blocks into the cache.

The above diagram depicts writes to the cache disk with and without the App Layering filter driver installed in the Provisioning Server targets. To know more about caching with Elastic Layers, refer to the article CTX227454.

User Layer and Citrix Provisioning Cache Disk

When User Layers are being used, only the Provisioning cache disk is used before a user logs on.

In this case, the local writable partition disk is only used during boot up. Once a user has logged in, all new file modifications are redirected to the User Layer disk on the file share and not to the streamed virtual disk, which means that the Provisioning cache no longer sees I/O on the writable partition.

Citrix App Layering and Machine Creation Services

To publish Layered Images to Machine Creation Services a Machine Creation Services Connector created for the hypervisor being published to. The Connector configuration includes the service account credentials used to access the hypervisor, in addition to hosts, storage locations, templates, and so forth.

The connector is then used to publish a layered image as a virtual machine “Master Image” to the hypervisor.

The MCS connector starts the Master Image after it's published and runs any layer scripts that have been defined in any layers. After all the scripts are run, the Master Image will be shut down via script and the hypervisor will take a snapshot of the virtual machine. Once this process is complete, the Master Image can be deployed using Machine Creation Services. The naming of the virtual machine is similar to Citrix Provisioning. The virtual machine is named as the published image template name followed by a date and time stamp. When a new version of the image is published, it is a new virtual machine. The new virtual machine is then used to update the existing catalog to roll out changes. It is important when using MCS that the template used to create the Master Images must have been created from a real virtual machine. The machine has been booted in Windows and has the right time zone set. This task ensures that the virtual BIOS is properly configured. If this task is not done, the resulting booted image has its time-off by some hours based on UTC. The best way to create the template is to use a clone of the original gold image used to create the OS layer. This step ensures that the virtual hardware settings match from the OS layer through to the Master Image.

Citrix App Layering Cross-Platform Support

The Citrix App Layering architecture is designed to support many hypervisors and provisioning systems without creating layers specific to any one platform. As many organizations move towards multi-cloud or hybrid cloud environments, Citrix App Layering eases this migration.

App Layering supports multiple platforms on multiple hypervisors with the combination of connectors and Platform Layers.

App Layering Connector - The App Layering Connectors are developed in node.js and run from the App Layering appliance. The connectors provide integration to all supported platforms including both hypervisors and provisioning systems.

Platform layer – This layer is similar to an application layer except that it always has the highest priority and when publishing images cleanup “recipes” are run differently against platform layers than app layers. Platform layers are where software drivers are installed for a defined “Platform.” For example, when using Citrix Provisioning the VDA software and PVS software are both installed into the Platform Layer.

App Layering Connectors and Platform Layers are combined to support available platforms. Learn more about the multiple hypervisors and cloud-platform deployment details and configurations, refer to Plan Your Deployment.

App Layering Communication Flow

App Layering uses several connections and ports. For more information on the communications and requirements, see Firewall Ports.

App Layering appliance

The App Layering Management Console is an HTML5 based console accessed over TCP/IP on port 80 (HTTP) or 443 (HTTPS).

Management Appliance Access

In addition to HTTP and HTTPS, the administrators are able to access the App Layering appliance virtual machine directly using SSH on port 22. Access is not permitted on unsecured Telnet or FTP ports. SSH is used to log on to the appliance as "root" for full access and an “administrator” account is used to access a limited menu to configure network settings. In Azure, "root" and "administrator" are both disabled. Instead, the administrators are prompted during the appliance provisioning for a local administrative user account with a default suggestion of “CitrixAdmin”.

When exporting logs from the management console, a download link is generated and presented in the task details. Port 8888 is used to download logs.

Port 8161 is used for ActiveMQ management and configuration, but access to this port is only available from within the App Layering appliance.

Optionally, the App Layering appliance can check for upgrades and download them from Citrix servers over the Internet using HTTPS/443 or HTTP/80. www.citrix.com and cdn.citrix.com are accessed if an Internet connection is available.

Connector Configuration

Each connector type uses a different port. The current list of connectors is:

Connectors open as a separate webpage within the management console. Each connector has both an HTTP and HTTPS listening port. When a connector is opened, the administrator is redirected to an HTML5 based interface in a new tab. The administrator’s browser must be able to access the App Layering appliance on the ports listed above for each connector.

Miscellaneous Ports

The App Layering appliance talks to various network servers and services on their respective ports. The App Layering appliance connects to the following services on the following TCP ports:

The Agent Service

The Agent Service can be used for three separate functions:

• Integration with Citrix Provisioning
• Integration with Hyper-V
• Integration with a general scripting server

The Agent Service is accessed on the following ports.

At initial installation of the App Layering Agent, the installer opens a connection on port 443 to the App Layering appliance to register the Agent Server. The App Layering appliance stores the FQDN and short name for the Agent Service Host, and the registry on the Agent server contains a record of the App Layering appliance.

Once the Agent and the App Layering appliance have exchanged identities, the App Layering appliance communicates directly to the Agent service on a secure SOAP channel on port 8016. Most communications between the appliance and the agent work as follows:

The specifics of the actual command being sent can often be seen in the Connector log on the App Layering appliance or the applayering.agent.log file on the Agent Service server.

When the appliance is asked to generate a log bundle, it transmits a request to every Agent Service that has ever been registered on the appliance requesting logs from the agent. Each Agent Service generates its own log bundle and transmits it back to the App Layering appliance on port 8787.

The main function of the Agent Service is to transfer files to Citrix Provisioning or Hyper-V. When transferring files, the agent uses the Windows Background Intelligent Transfer Service (BITS) on the Agent Service Server to pull the virtual disk to the server and place it in the store or upload or download a VHD from Hyper-V.

The process of transferring a file works like this:

Normally, the file transfer runs on port 3009, which is unencrypted. This communication is done for performance reasons – the overhead of running on a HTTPS connection significantly impacts throughput. However, the Agent can be configured to force HTTPS and use 3509 instead.

When the Agent runs BITS, it provides BITS two things: the URL for the file download, and the destination file path. BITS is run as the user account that is configured in the Connector. Therefore, this user needs permissions to make an outgoing connection from the Provisioning server to port 3009 on the App Layering appliance; and also rights to write a file into the virtual disk store.

Hypervisors

Hypervisor Connectors use the following ports.

These ports are the same ports that the hypervisor browser-based management consoles also use. App Layering is using API calls through the hypervisor’s normal web service for communications with the hypervisor.

For vSphere file uploads, and downloads are not performed by communicating with vCenter instead they are handled by direct communication with an ESXi host. For this reason, the vSphere connectors require a host to be defined and the host firewall must be configured to allow access from the App Layering appliance on port 443.

Compositing Engines

Compositing Engines connect back to the App Layering Appliance for iSCSI connections on port 3260 and they make API calls to the appliance on port 443. The App Layering Appliance performs API calls to the Compositing Engines also on port 443.

Availability, Backup, and Recovery

App Layering can have several components where backups are appropriate, including the appliance, Elastic Layer Shares, and User Layer Shares.

Any availability strategy for App Layering must fit into the overall availability and recovery design for your whole workspace solution. Pools and delivery groups are already highly available because they are spread across hosts and pools of storage.

The components specific to App Layering are the appliance, Elastic Layers, and User Layers.

Appliance Backups

The App Layering appliance, as stated earlier, is not required for end users to have full use of App Layering, therefore, it is not a requirement to make the appliance highly available. However, it is a requirement to back up the appliance regularly. Appliance backups ensure that all the layers are available even if the appliance is somehow destroyed or corrupted. Any virtual machine back-up technology can be used for the App Layering appliance. It is also possible to use two appliances and the import and export feature to keep them in sync. However, this step is a manual process and does not back up many of the settings used in App Layering like Image Templates and Elastic Layer Assignments.

Availability and Disaster Recovery for Elastic Layers

Elastic Layers are files mounted on virtual desktops agents (VDAs) during log-on using a Windows in-guest mount from an SMB share. The Layering Service connects the layers on logon, but it never reconnects a disconnected VHD file. Therefore, it's critical to ensure that the file server used for Elastic Layers is highly available by using some type of clustered file system. Using multiple DFS-R targets is not sufficient for this use case, because if a target fails, the mounted VHD files cannot be remapped to another target until another logon happens.

For Disaster Recover, there are two models that can be used to handle Elastic Layers; a replication model or a dual appliance model.

Replication Model

In this model, Elastic Layer shares can be replicated using any file-system replication technology. This model can include technologies like DFS-R, Microsoft Storage Replica, Veeam, NAS vendor replication, Zerto, VMware vSphere Replication, and even robocopy.

Then the VDAs in the DR data center can be pointed to the Elastic Layer share in that data center via GPO. A GPO template to configure the repository location can be found here: CTX222107.

Dual Appliance Model

In this model, an appliance is installed into each data center. The import and export functionality provided by the Appliance is used to keep the two appliances in sync from a layer perspective. Here the administrators manage DR layers separately and build images in DR from a local appliance.

If this option is chosen, then the sync transfers over the WAN from the SMB share defined during the import and export operation. Then Layers must be assigned on the DR appliance, and the second appliance will then copy them out to the Elastic Layer share in DR. In this model, it is also possible to develop a solution that does not sync all layers but only desired layers. Since layers are chosen for export manually, only selected layers can be included in the process. Currently, the import and export process must be kicked off manually.

In the dual appliance model, connectors and permissions for elastic shares must be created on each side. The only objects that get imported are the actual layers themselves. However, it is possible in this model to have different layers in each site as needed. For example, if it is really an active-active site scenario.

Availability and Disaster Recovery for User Layers

User Layers are similar to Elastic Layers in that they are VHD/VHDX files mounted by Windows on logon using an in-guest mount. However, they are mounted as writable, and the files are locked by the Windows desktop. This task makes the options for backing up and replicating these disks much more difficult than normal Elastic Layers.

Due to this limitation, in case DFS-R or a robocopy script is used, the synchronization process has to be performed off hours (if there are times considered off hours). The process has to constantly check to see if the files are available to sync. For User Layers, which can be large files, robocopy might not work well as it might always copy the entire file rather than the blocks that have changed. DFS-R might be a better choice because it copies only modified blocks. However, replication at the storage level might be even better as it might synchronize more evenly as changes are made rather than waiting for the file locks to be removed.

There are other options that are supported here as well depending on the technology chosen to store the User Layer VHD files. If the file server supports the Volume Shadow Copy Service (VSS), then VSS snapshots are created to allow for backup and replication of the User Layers. By varying the frequency of the process User Layers can then also be rolled back to earlier points in time if corruption or mistake is made that adversely affects the user.

If a storage controller supports NDMP, this feature can also be used for User Layer backups. NDMP works at the storage level to back-up NAS storage directly to disk or tape utilizing NAS snapshots.

Due to the difficulty of replicating large disks and the expense for the bandwidth to do so, many customers choose to provide DR desktops for users without a replicated User Layer. In this model, there are two options. Users can be provisioned a separate delivery group in the DR site that is also User Layer enabled. They can then log on to the DR site and pre-configure that layer with what they need. Or users can be provisioned with a normal non-persistent DR desktop. In the latter case, it is often beneficial to mix Citrix Profile Management with the User Layer so that user settings can be replicated to the DR site.

Components of multi-site disaster recovery

The approach to multisite disaster recovery is similar to local recovery. For images, you must use a replication process. If you're using Citrix Provisioning, you can use Robocopy to copy vDisks to the secondary site. If you're using Machine Creation Services, you need a process to replicate virtual machines such as Veeam, Zerto, VMware vSphere Replication, or Site Recovery Manager. This also works to protect the ELM.

For Elastic Layers, SAN replication or a scripted copy both work. If you're using User Layers, then you need to replicate at the SAN/NAS level so that changed blocks can be replicated under the clustered file system used for the share.

This approach is better than having multiple connectors defined in the ELM and publishing directly to both sites because, when publishing, you must both compose the image and upload it to the store. If you use a process that replicates the already created image, this skips the composition process and is more efficient.

It's also possible to use two ELM appliances, one in each site. Then you can use the export/import functionality to keep those ELMs in sync from a layer perspective. You can then treat recovery separately and build images there from a local ELM.

If you choose this option, then the sync transfers over the WAN to the SMB share defined in Settings. The layers can then be synchronized to the SMB share used in the second site, using Robocopy with the /MIR switch. Currently the import and export process must be initiated manually.

You can also sync only desired layers rather than all layers. If you might like this option, contact your App Layering solution architect for more details.

In the Dual ELM model, connectors and permissions for Elastic Shares must be created on each side. The only objects being imported are the actual layers. However, it's possible in this model to have different layers in each site as needed.

References

The following resources are referenced for a better understanding of Citrix App Layering:

App Layering Product Documentation

How to Create a Platform Layer

Windows and App Layering Optimizations

App Layering Antivirus Guide

Understand the Office Recipe

App Layering Best Practices

Citrix Application Delivery Controller (ADC) Global Server Load Balancing (GSLB) is a DNS-based solution which describes a range of technologies to distribute resources around multi-site data center locations. This document describes the deployment topology and configuration architecture needed to set up GSLB between multi-sites where Citrix Virtual Apps and Desktops StoreFront servers are load-balanced by Citrix Gateway and Citrix ADC.

Fundamental Design Factors

The following includes fundamental design factors during an assessment and design phase that affects the formation of the design to cater for requirements. It highlights those considerations and provides background information and insight to support these.

• Multi-site Geo-dispersed Data center deployment with ADC - Customer operates Citrix ADC appliances deployed across data center sites (that is, data center 1 and data center 2). A Citrix ADC high availability pair deployment consisting of two appliances commonly shares physical peripheral hardware components placed within the same data center site. It is intended to protect against Citrix ADC services outages caused by Citrix ADC appliance or peripheral hardware component failures (that is, network switches, power distribution units, and so on). As Citrix ADC appliances are deployed to two different sites (that is, data center 1 and data center 2) not physically sharing peripheral hardware components (that is, network switches, power distribution units, and so on), the design caters for a deployment that uses Citrix ADC GSLB to provide for resilience and redundancy.

• Business continuity - For component resilience and redundancy, business requirements exist for the design to cater for single systems failure within and across data center sites without affecting services availability and performance. A disaster can involve a single data center failure or failure of individual services within a single data center site resulting in failing over services and client connections to another data center site. Citrix ADC GSLB is used to cater to network traffic distribution, high availability, and failover services across both data center 1 and data center 2 sites.

• Network traffic flow efficiency - The design incorporates network traffic flows involving multiple serial hops to access individual services within the customer infrastructure. To ensure network traffic flow efficiency and eliminate routing inefficiency, network traffic flows are designed to remain within each local data center site. As such, the design caters to primary traffic flows to use back-end systems within the same data center site, and secondary (backup) traffic flows use back-end systems within the opposite data center site.

Global Server Load Balancing

GSLB Feature Overview

With ordinary DNS, when a client sends a domain name system (DNS) request, it receives a list of IP addresses of the domain or service. Generally, the client chooses the first IP address in the list and initiates a connection with that server. The DNS server uses a technique called DNS round robin to rotate through the IPs on the list. It sends the first IP address to the end of the list and promotes the others after it responds to each DNS request. This technique ensures equal distribution of the load, but it does not support disaster recovery, load balancing based on load or proximity of servers, or persistency.

Fundamentally, GSLB based on DNS works the same way as standard DNS, with the exception that more logic is in place to determine what addresses to return. The logic in most situations is based on:

• The load and capacity of resources on the network
• The IP address or interface the query came from
• Previous requests made from the same IP or network
• The health state of resources

To ensure the various pieces of information are in place, the ADC system makes use of several ways to determine state so that proper decision making can occur:

• Via explicit monitors that check for availability of remote resources by accessing the resource itself
• Via Metric Exchange Protocol (MEP), which is a channel of communication between distinct NetScaler devices, and provides a mechanism for one ADC to provide state information about resources to another ADC
• Through SNMP based load monitors, which poll a remote resource for statistics such as CPU load, network load

Figure-1 A Typical DNS Flow to Application Access

When you configure GSLB on ADC appliances and enable MEP, the DNS infrastructure is used to connect the client to the data center that best meets the set criteria. The criteria can designate the least loaded data center, the closest data center, the data center that responds most quickly to requests from the client’s location, a combination of those metrics, and SNMP metrics. An appliance tracks the location, performance, load, and availability of each data center. It uses these factors to select the data center to send the client request.

GSLB Deployment

Deployment Types

Citrix ADC appliances configured for GSLB provide for disaster recovery and ensure the continuous availability of applications by protecting against points of failure in a WAN. GSLB can balance the load across data centers by directing client requests to the closest or best-performing data center, or to surviving data centers in the event of an outage.

The following are some of the typical GSLB deployment types:

Active-Active site deployment - An active-active site consists of multiple active data centers. Client requests are load balanced across active data centers. This deployment type can be used when you require global distribution of traffic in a distributed environment.

All the sites in an active-active deployment are active, and all the services for a particular application/domain are bound to the same GSLB virtual server. Sites exchange metrics through the Metrics Exchange Protocol (MEP). Site metrics exchanged between the sites include the status of each load balancing and content switching virtual server, the current number of connections, current packet rate, and current bandwidth usage. The Citrix ADC appliance needs this information to perform load balancing across the sites.

An active-active deployment can include a maximum of 32 GSLB sites, because MEP cannot synchronize more than 32 sites. No backup sites are configured in this deployment type.

The Citrix ADC appliance sends client requests to the appropriate GSLB site as determined by the GSLB method specified in the GSLB configuration.

Figure-2 Active-Active Site Deployment

Active-Passive site deployment - An active-passive site consists of an active and a passive data center. This deployment type is ideal for disaster recovery.

In this type of deployment, some of the sites (remote sites) are reserved only for disaster recovery. These sites do not participate in any decision making until all the active sites are DOWN. A passive site does not become operational unless a disaster event triggers a failover.

Once you have configured the primary data center, replicate the configuration for the backup data center and designate it as the passive GSLB site by designating a GSLB virtual server at that site as the backup virtual server.

An active-passive deployment can include a maximum of 32 GSLB sites, because MEP cannot synchronize more than 32 sites.

Figure-3 Active-Passive Site Deployment

Parent-child topology deployment - Citrix ADC GSLB provides global server load balancing and disaster recovery by creating mesh connections between all the involved sites and making intelligent load balancing decisions. Each site communicates with the others to exchange server and network metrics through Metric Exchange Protocol (MEP), at regular intervals. However, with the increase in number of peer sites, the volume of MEP traffic increases exponentially because of the mesh topology. To overcome this, you can use a parent-child topology. The parent-child topology also supports larger deployments. In addition to the 32 parent sites, you can configure 1024 child sites.

Entities

A GSLB configuration consists of a group of GSLB entities on each appliance in the configuration. These entities include GSLB sites, GSLB services, GSLB service groups, GSLB virtual servers, and ADNS services.

GSLB Sites

A typical GSLB setup consists of data centers, each of which has various network appliances that may or may not be Citrix ADC appliances. The data centers are called GSLB sites. Each GSLB site is managed by a Citrix ADC appliance that is local to that site. Each of these appliances treats its own site as the local site and all other sites, managed by other appliances, as remote sites.

If the appliance that manages a site is the only Citrix ADC appliance in that data center, the GSLB site hosted on that appliance acts as a bookkeeping placeholder for auditing purposes, because no metrics can be collected. Typically, this happens when the appliance is used only for GSLB, and other products in the data center are used for load balancing or content switching.

Relationships among GSLB Sites

The concept of sites is central to Citrix ADC GSLB implementations. Unless otherwise specified, sites form a peer relationship among themselves. This relationship is used first to exchange health information and then to distribute load as determined by the selected algorithm. In many situations, however, a peer relationship among all GSLB sites is not desirable. Reasons for not having an all-peer implementation can be:

• To clearly separate GSLB sites. For example, to separate sites that participate in resolving DNS queries from the traffic management sites.
• To reduce the volume of MEP traffic, which increases exponentially with an increasing number of peer sites.

These goals can be achieved by using parent and child GSLB sites.

GSLB Services

A GSLB service is usually a representation of a load balancing or content switching virtual server, although it can represent any type of virtual server. The GSLB service identifies the virtual server’s IP address, port number, and service type. GSLB services are bound to GSLB virtual servers on the Citrix ADC appliances managing the GSLB sites. A GSLB service bound to a GSLB virtual server in the same data center is local to the GSLB virtual server. A GSLB service bound to a GSLB virtual server in a different data center is remote from that GSLB virtual server.

GSLB Virtual Servers

A GSLB virtual server has one or more GSLB services bound to it, and load balances traffic among those services. It evaluates the configured GSLB methods (algorithms) to select the appropriate service to which to send a client request. Because the GSLB services can represent either local or remote servers, selecting the optimal GSLB service for a request has the effect of selecting the data center that should serve the client request.

The domain for which global server load balancing is configured must be bound to the GSLB virtual server, because one or more services bound to the virtual server serve requests made for that domain.

Unlike other virtual servers configured on a Citrix ADC appliance, a GSLB virtual server does not have its own virtual IP address (VIP).

ADNS Services

An ADNS service is a special kind of service that responds only to DNS requests for domains for which the Citrix ADC appliance is authoritative. When an ADNS service is configured, the appliance owns that IP address and advertises it. Upon reception of a DNS request by an ADNS service, the appliance checks for a GSLB virtual server bound to that domain. If a GSLB virtual server is bound to the domain, it is queried for the best IP address to which to send the DNS response.

DNS VIPs

A DNS virtual IP is a virtual IP (VIP) address that represents a load balancing DNS virtual server on the Citrix ADC appliance. DNS requests for domains for which the Citrix ADC appliance is authoritative can be sent to a DNS VIP.

Metrics Exchange Protocol (MEP)

The data centers in a GSLB setup exchange metric with each other through the MEP, which is a proprietary protocol for the Citrix ADC appliance. The exchange of the metric information begins when you create a GSLB site. These metrics comprise load, network, and persistence information.

MEP is required for health checking of data centers to ensure their availability. A connection for exchanging network metric (round-trip time) can be initiated by either of the data centers involved in the exchange, but a connection for exchanging site metrics is always initiated by the data center with the lower IP address. By default, the data center uses a subnet IP address (SNIP) to establish a connection to the IP address of a different data center. However, you can configure a specific SNIP, virtual IP (VIP) address, or the NSIP address, as the source IP address for metrics exchange. The communication process between GSLB sites uses TCP port 3011 or 3009, so this port must be open on firewalls that are between the Citrix ADC appliances.

Load Balancing Methods

Unlike traditional DNS servers that simply respond with the IP addresses of the configured servers, a Citrix ADC appliance configured for GSLB responds with the IP addresses of the services, as determined by the configured GSLB method. By default, the GSLB virtual server is set to the least connection method. If all GSLB services are down, the appliance responds with the IP addresses of all the configured GSLB services.

GSLB methods are algorithms that the GSLB virtual server uses to select the best-performing GSLB service. After the host name in the Web address is resolved, the client sends traffic directly to the resolved service IP address.

The Citrix ADC appliance provides the following GSLB methods:

For GSLB methods to work with a remote site, either MEP must be enabled, or explicit monitors must be bound to the remote services. If MEP is disabled, RTT, Least Connections, Least Bandwidth, Least Packets, and Least Response Time methods default to Round Robin.

Monitor GSLB services

When you bind a remote service to a GSLB virtual server, the GSLB sites exchange metric information, including network metric Information, which is the round-trip-time and persistence Information.

If a metric exchange connection is momentarily lost between any of the participating sites, the remote site is marked as DOWN, and load balancing is performed on the remaining sites that are UP. When the metric exchange for a site is DOWN, the remote services belonging to the site are marked DOWN as well.

The Citrix ADC appliance periodically evaluates the state of the remote GSLB services by using either MEP or monitors that are explicitly bound to the remote services. Binding explicit monitors to local services is not required, because the state of the local GSLB service is updated by default using the MEP. However, you can bind explicit monitors to a remote service. When monitors are explicitly bound, the state of the remote service is not controlled by the metric exchange.

Reference Architecture

Design GSLB

The following details the Citrix ADC instances network address configurations in terms of IP addressing and routing in data center site data center 1 and 2:

• NSIP – Citrix ADC Management IP address
• SNIP – Citrix ADC Subnet IP and ADNS Listener IP
• GSLB – GSLB Site IP
• VIP – Citrix ADC VIP for Citrix Gateway
• VIP Citrix ADC Load Balancing (LB) VIP for StoreFront

Figure-4 DNS and GSLB workflow

Figure 4 describes a DNS workflow from the client's application access request via DNS, which will be handled by GSLB entities. As a DNS request comes into the global DNS server, which delegates the request subzone to each ADNS IP as subzone name servers. Upon reception of a DNS request by an ADNS service, the appliance checks for a GSLB virtual server bound to that domain. If a GSLB virtual server is bound to the domain, it is queried for the best IP address to which to send the DNS response.

Figure 5 diagram illustrates its actual deployment architecture topology. It lists all necessary interfaces associated with designated ADC IP addresses accordingly (that is, NSIP, SNIP/ADNS IP, Gateway IP, Load Balance IP) overlays with GSLB topology and services.

Figure-5 GSLB Deployment Architecture

Those specific GSLB entities, as described in the earlier chapter, are:

ADNS Listener IP: An ADC IP that listens for DNS queries.

• The ADNS listener IP is typically an existing SNIP on the ADC appliance.
• For external DNS, create a public IP for the ADNS Listener IP, and open UDP 53 and TCP 53, so Internet-based DNS servers can access it.

GSLB Site IP / MEP listener IP: An ADC IP that is used for ADC-to-ADC GSLB communication. The communication, MEP transmits the following between GSLB-enabled ADC pairs: load balancing metrics, proximity, persistence, and monitoring.

• GSLB Sites – On ADC, you create GSLB Sites. GSLB Sites are the endpoints for the MEP communication. Each ADC pair is configured with the MEP endpoints for the local appliance pair, and all remote appliance pairs.
• TCP Ports – MEP uses port TCP 3009 or TCP 3011 between the ADC pairs. TCP 3009 is encrypted.
• The ADNS IP address can be used as the MEP endpoint IP.
• GSLB Sync Ports: To use GSLB Configuration Sync, open ports TCP 22 and TCP 3008 (secure) from the NSIP (management IP) to the remote public MEP IP. The GSLB Sync command runs a script in BSD shell and thus NSIP is always the Source IP.

Public IP Addresses: In summary, for public GSLB, if MEP and ADNS are listening on the same IP, then you need one new public IP that is NAT’d to the DMZ IP that is used for ADNS and MEP (GSLB Site IP).

• Each data center has a separate public IP.
• DNS is delegated to all public ADNS IP listeners.

Other Dependencies

The infrastructure for the solution provides a set of common components used by the entire solution.

• Network Time Services - Most components in the overall solution require integration with Network Time Services (NTP). The following table details the key NTP settings within Client infrastructure to be used with the Citrix Delivery Network deployment.
• Domain Name Services - Most components in the overall solution require integration with Domain Name Services (DNS). The following table details key DNS infrastructure within the Customer network to be used by the Citrix Delivery Network deployment
• Security and Authentication - Secure sessions are handled by Citrix Gateway. The following table details key decisions pertaining to SAN certificates for use in each production, acceptance, and test infrastructure.

Sources

The goal of this reference architecture is to assist you with planning your own implementation. To make this job easier, we would like to provide you with source diagrams that you can adapt in your own detailed designs and implementation guides: reference-architectures_adc-gslb.pptx

Citrix Product Documentation References

The deliverable provides guidelines for the implementation and configuration references. However, it does not provide step-by-step instructions on how to install or maintain the components discussed. Therefore, Citrix Consulting recommends Client design and operations teams involved in the design and deployment to review the following documents, articles, and guides prior to implementing the environment provided for production. These documents, articles, guides, and more are available from the online Citrix Knowledge Center, online Citrix Product Documentation, or online Citrix Community.

Citrix Online Product Documentation Citrix ADC

Citrix Online Product Documentation Citrix ADC VPX Virtual Machines

Citrix Online Product Documentation Global Server Load Balancing

Citrix Whitepaper CTX129514 – Secure Deployment Guide for Citrix ADC MPX, VPX, and SDX Appliances

Citrix Whitepaper CTX123976 – Citrix ADC Global Server Load Balancing Primer: Theory and Implementation

Citrix Knowledgebase Article CTX122619 – DNS and GSLB Primer

Citrix Knowledgebase Article CTX121713 – How to Delegate Subdomains in a Microsoft DNS or a BIND for Global Server Load Balancing on a Citrix ADC Appliance

Citrix Knowledgebase Article CTX110488 – Delegating DNS Subdomains to the GSLB Setup of the Citrix ADC Appliances

Citrix Online Product Documentation Load Balancing

Citrix Online Product Documentation SSL Offload and Acceleration

Citrix Community Blog Gateway Integration with StoreFront Lessons Learned

Citrix Community Blog StoreFront and Citrix Gateway GSLB considerations

Citrix Application Delivery Management (ADM) is a centralized management solution. It simplifies operations by providing administrators with enterprise-wide visibility and automating management jobs that are getting ran across multiple instances. You can manage and monitor Citrix application networking products including Citrix Application Delivery Controllers (ADC) MPX, VPX, SDX, CPX, BLX, Citrix Gateway, Citrix Web Application Firewall (WAF). You can use ADM to manage, monitor, and troubleshoot the entire global application delivery infrastructure from a single, unified console.

ADM also addresses the application visibility challenge by collecting detailed information about web-application and virtual-desktop traffic including application flow, security events, user-session-level information, webpage performance data, and database information flowing through the managed Citrix Appliances, and providing actionable reports. This approach enables administrators to troubleshoot and proactively monitor customer issues in a matter of minutes.

Citrix ADM Software virtual appliances can be deployed in several deployment modes and provide the flexibility to integrate within your existing Citrix networking design. The following are some of the deployment scenarios implemented by using ADM Software appliances.

• Single Server
• High Availability (Recommended)
• Disaster Recovery Mode
• ADM Agent Deployment (for adding remote Sites)

This ADM Reference document defines a set of architectural building blocks for delivering Citrix Application Delivery Management (Citrix ADM). The target audiences are technical professionals and architects seeking knowledge on how to key components to support the following objectives.

ADM Appliance Software Architecture

The Citrix Application Delivery Management (ADM) software uses a built-in data store to provide integration with the server, and the server manages all the key processes, such as data collection, NITRO API calls. In its data store, the server stores an inventory of instance details, such as host name, software version, running, and saved the configuration, certificate details, entities configured on the instance. Single server deployment is suitable if you want to process small amounts of traffic or store data for a limited time.

The following image shows the different internal and external subsystem components of a Citrix ADM appliance and the communication flow between the internal ADM server components and externally managed networking appliances and instances.

The Citrix ADM NITRO Service acts as a web server handling HTTP requests and responses sent to other subsystems within the appliance from the management GUI or APIs/SDKs, using ports 80 and 443. These requests travel via the Message Bus (message processing system) by using the inter-process communication (IPC) mechanism. Initially, the HTTP requests sent to the Control subsystem, which either processes the information or sends it to another, more appropriate subsystem. Each of the other subsystems including Inventory, StyleBooks, Data Collector, Configuration, AppFlow Decoder, AppFlow Analytics, Performance, Events, Entities, SLA Manager, Provisioner, Journal, and daemons (aaad/snmpd/ntpd/syslogd/monit/sshd/pitboss), have specific roles.

ADM Systems Design

Citrix ADM is a centralized management solution that simplifies operations by providing administrators with enterprise-wide visibility and automating management jobs that are getting ran across multiple instances.

To manage and monitor applications and the network infrastructure, you must first install Citrix ADM on one of the hypervisors. You can deploy Citrix ADM either as a single server or in a high availability mode. When using Citrix ADC Insight Center, you can migrate to Citrix ADM and avail of the management, monitoring, orchestration, and application management features in addition to the analytics features.

• Single-server deployment. In a Citrix ADM single server deployment, the database is integrated with the server, and a single server processes all the traffic. You can deploy Citrix ADM with Citrix Hypervisor, VMware ESXi, Microsoft Hyper-V, and Linux KVM.
• High availability deployment. A high availability deployment (HA) of two Citrix ADM servers provides uninterrupted operations. In a high availability setup, both Citrix ADM nodes must be deployed in active-passive mode, on the same subnet using the same software version and build, and same configurations. With HA deployment the ability to configure the floating IP address on the Citrix ADM primary node eliminates the need for a separate Citrix ADC load balancer.

The following diagram depicts the high-level ADM HA appliance deployment.

ADM Key System Requirements

Before importing a Citrix ADM appliance to your current platform (that is, Hypervisors), understanding the critical system licensing, hypervisor requirements, appliance image requirements, and ADC Build Integration limitations is a must.

Licensing Overview

Citrix ADM requires a verified Citrix ADC license to manage and monitor the Citrix ADC instances.

You can manage and monitor any number of supported instances and entities without a license. However, you can select and configure Analytics for an initial 30 discovered applications on the App Dashboard and view analytics data for 30 virtual servers without applying for extra licenses. To collect Analytics for more than 30 discovered applications (30 virtual servers), you must purchase and apply the desired licenses.

Full information on licensing is available in the Citrix ADM product documentation about licensing.

Supported Hypervisors

An ADM appliance deployed on-premises as virtual appliances can run on Citrix Hypervisor, VMware ESXi, Microsoft Hyper-V, and Linux KVM.

The following table lists the hypervisors supported by Citrix ADM.

Requirements for ADM appliance and agent Images

Citrix ADC instances deployed in remote data centers can be managed and monitored from Citrix ADM running in a primary data center. Citrix ADC instances sent data directly to the primary Citrix ADM that resulted in the consumption of WAN bandwidth. Also, the processing of analytics data utilizes CPU and memory resources of primary Citrix ADM.

Customers have their data centers located across the globe. Agents play a vital role in following scenarios where the customers can choose:

• To install agents in remote data centers so that there is a reduction in WAN bandwidth consumption.
• To limit the amount of instances directly sending traffic to primary Citrix ADM for data processing.

Requirements for Citrix ADM appliance

Requirements for Citrix ADM on-prem agent

Agents work as an intermediary between the primary Citrix ADM and the discovered instances across different data centers. Following are the benefits of installing agents:

• The instances are configured to agents so that the unprocessed data is sent directly to agents instead of primary Citrix ADM. Agents do the first level of data processing and send the processed data in a compressed format to the primary Citrix ADM for storage.
• Agents and instances are co-located in the same data center so that the data processing is faster.
• Clustering the agents provides redistribution of Citrix ADC instances on agent failover. When one agent in a site fails, traffic from Citrix ADC instances switched to another available agent on the same site.

The following is the minimum requirements for Citrix ADM on-prem agent:

The following figure shows Citrix ADC instances in two data centers and Citrix ADM high availability deployment using multisite agent-based architecture.

The primary site has the Citrix ADM nodes deployed in a high availability configuration. The Citrix ADC instances in the primary site directly registered with the Citrix ADM.

In the secondary site, agents are deployed and registered with the Citrix ADM server in the primary site. These agents work in a cluster handling a continuous flow of traffic in case an agent failover — the Citrix ADC instances at the secondary site registered with the primary Citrix ADM server through agents. The instances send data directly to agents instead of primary Citrix ADM. The agents process the data received from the instances and send it to the primary Citrix ADM in a compressed format. Agents communicate with the Citrix ADM server over a secure channel, and the data sent over the channel compressed for bandwidth efficiency.

Minimum Citrix ADC versions required for Citrix ADM feature

Diverse Citrix ADM features supported on different Citrix ADC software versions. Review the following table to make sure you have upgraded your Citrix ADC instances to the correct version.

Important Note: Integrated Cache Metrics are not supported in Citrix ADM with Citrix instances running version 11.0 build 66.x.

Environment Customizations and Sizing Recommendations

Sizing Settings

Citrix Application Delivery Management (ADM) storage requirement is determined based on your Citrix ADM sizing estimation. By default, Citrix ADM provides you a storage capacity of 120 GB. If you need more than 120 GB for storing your data, you can attach an extra disk (Max extra disk per ADM is 3 TB).

Notes:

• Estimate storage requirements and attach an extra disk to the server at the time of the initial deployment of Citrix ADM.
• For a Citrix ADM single-server deployment, you can attach only one disk to the server in addition to the default disk.
• For a Citrix ADM high availability deployment, you must attach an extra disk to each node. The size of both disks should be identical.
• If you had earlier attached an external disk of lower capacity, you must remove the disk before attaching a new disk.
• You can attach an extra disk of capacity higher than 2 terabytes. If necessary, the size of the disk can be smaller than 2 terabytes also.
• Citrix recommends using solid-state drive (SSD) technology for Citrix ADM deployments.

Prune Settings

To limit the amount of reporting data stored in your Citrix ADM server’s database, you can prune it. You can specify the interval for which you want Citrix ADM to retain network reporting data, events, audit logs, and task logs. By default, this data is pruned every 24 hours (at 00.00 hours). More details can found here.

Backup Settings

Citrix devices can be backed up automatically to the Citrix ADM server. Also, those backed-up data forwarded to an external server for historical trending and auditing. Refer to the link for more details.

ADM Deployment Scenarios

Single-Server Deployment

In a Citrix ADM single server deployment, the database is deployed and integrated with the server, and a single server processes all the traffic. You can use Citrix ADM with Citrix Hypervisor, VMware ESXi, Microsoft Hyper-V, and Linux KVM.

High Availability (HA) Deployment

An HA deployment of two Citrix ADM servers provides uninterrupted operations. In a high availability setup, both the Citrix ADM nodes must be deployed in active-passive mode, on the same subnet using the same software version and build and must have identical configurations. With HA deployment, the ability to configure the floating IP address on the Citrix ADM primary node eliminates the need for a separate Citrix ADC load balancer.

The following are the benefits of a high availability deployment with Citrix ADM:

• An improved mechanism to monitor heartbeats between the primary and secondary nodes.
• It provides physical streaming replication of the database instead of logical bi-directional replication.
• High availability configuration provides the ability to configure the floating IP address on the primary node to eliminate the need for a separate Citrix load balancer.
• It provides easy access to the Citrix ADM user interface using the floating IP address.
• Citrix ADM user interface is provided only on the primary node. By using the primary node, you can eliminate the risk of accessing and making changes to the secondary node.
• Configuring the floating IP address handles the failover situation, and reconfiguring the instances is not required.
• Provides built-in ability to detect and handle the split-brain situation

The following diagram depicts the ADM HA deployment.

Components of high availability architecture

In high availability deployment, one of the Citrix ADM nodes configured as the primary node (ADM HA Node 1) and the other as the secondary node (ADM HA Node 2). If the primary node goes down due to any reason, the secondary node takes over as the new primary node.

Disaster Recovery (DR) Mode - Reference Architecture

Disaster is a sudden disruption of business functions caused by natural calamities or human-caused events. Disasters affect data center operations, after which resources and the data loss at the disaster site must be fully rebuilt and restored. The loss of data or downtime in the data center is critical and collapses the business continuity.

The Citrix ADM disaster recovery (DR) feature provides full system backup and recovery capabilities for Citrix ADM deployed in high availability mode. At the time of recovery, certificates, configuration files, and a complete backup of the database are available in the recovery site.

The following table describes the terms used while configuring disaster recovery in Citric ADM

Note: The primary site and DR site communicate with each other through ports 5454 and 22, which are enabled by default.

The following image shows the disaster recovery workflow, the initial setup before the disaster, and the workflow after the disaster.

The image shows the disaster recovery setup before the disaster.

The primary site has Citrix ADM nodes deployed in the high availability mode, as shown in the previous section.

The recovery site has a standalone Citrix ADM disaster recovery node deployed remotely. The disaster recovery node is in read-only mode and receives data from the primary node to create data backup. Citrix ADC instances in the recovery site are also discovered, but they do not have any traffic flowing through them. During the backup process, all data, files, and configurations are sent and replicated on the disaster recovery node from the primary node.

After the initiation of the script at the DR site, the DR site now becomes the new primary site. You can also access the DR user interface.

Full information about the disaster recovery (DR) feature is available in the Citrix ADM product documentation article Configure disaster recovery for high availability.

ADM Agent Deployment

You can install and configure the agent, to enable communication between the primary Citrix ADM and the managed Citrix ADC instances in another data center.

You can install an agent on the following hypervisors in your enterprise data center:

• Citrix Hypervisor
• VMware ESXi
• Microsoft Hyper-V
• Linux KVM Server

The number of agents installed per site depends on the traffic being processed. Currently, Citrix has validated two agents per site for an agent failover scenario. Citrix recommends that you install at least two agents per site so that the traffic flows to another agent in case of an agent failover.

For communication purposes, the following ports must be open between the agent and Citrix ADM on-prem server.

The following ports must be open between the agent and Citrix ADC Instances.

References

Citrix Application Delivery Management Product Documentation

Citrix ADC Product Documentation

Citrix Analytics Platform

ADM Service Graph for cloud-native applications

Citrix ADM and Hybrid Multi-Cloud

Citrix ADM Service in Citrix Cloud

Appendix

Citrix Networking Appliance & Functionality Overview

VPX Overview

Citrix ADC is an application delivery controller that performs application-specific traffic analysis to distribute intelligently, optimize, and secure Layer 4-Layer 7 (L4–L7) network traffic for web applications. For example, a Citrix ADC bases load balancing decisions on individual HTTP requests instead of on long-lived TCP connections, so that the failure or slowdown of a server is managed much more quickly and with less disruption to clients. Its feature set can be broadly consisting of switching features, security and protection features, and server-farm optimization features.

The Citrix ADC VPX is a software-based platform that provides industry-leading delivery of applications over the internet and private networks. This virtual appliance can be deployed on hypervisors on-premises or cloud platforms. The VPX appliance is supported by Citrix Hypervisor, VMware ESXi, Microsoft Hyper-V, Linux KVM, Microsoft Azure, and Amazon Web Services (AWS).

The VPX provides the full functionality of the Citrix Networking product line, with throughput capabilities ranging from 10 Mbps to 100 Gbps. The performance is controlled by the platform license and can be increased on-demand by upgrading the platform license.

Citrix ADC VPX in Azure

Citrix ADC VPX in Azure is deployed in a Virtual Network (VNET) and is available from the Azure Marketplace in subscription-based, check-in/check-out, or Bring Your Own License (BYOL) editions. The recommended configuration includes three NICs: management, client-side, and server-side subnets. All on-premises Citrix Networking features are available in Azure, except for the following: clustering, IPv6, gratuitous ARP (GARP), L2 mode, tagged VLANs, dynamic routing, virtual MAC (VMAC), USIP, and jumbo frames.

More details on Citrix ADC VPX in Azure can be found here.

Citrix ADC VPX in AWS

Citrix ADC VPX in AWS is deployed in a Virtual Private Cloud (VPC) and is available as an Amazon Machine Image (AMI) from the AWS Marketplace in subscription-based or bring you to own license (BYOL) editions. The recommended configuration includes three Elastic Network Interfaces (ENIs): management, client-facing, and back-end subnets. All on-premises Citrix Networking features are available in AWS, except for the following: IPv6, gratuitous ARP (GARP), L2 mode, tagged VLAN, dynamic routing, virtual MAC (VMAC).

More details on Citrix ADC VPX in AWS can be found here.

MPX Overview

Citrix ADC MPX is a hardware-based, highly performant platform that provides industry-leading delivery of applications over the Internet and private networks, combining application-level security, optimization, and traffic management into a single, integrated appliance. All MPX appliances support Citrix nCore technology, which enables them to use their multi-core CPU systems for multi-gigabit performance and massive scalability for all application workloads.

An MPX can be integrated into any network as a complement to existing load balancers, servers, caches, and firewalls. It requires no additional client or server-side software and can be configured using the web-based GUI and CLI configuration utilities. Flexible Pay-As-You-Grow licensing helps customers protect their investment, avoid costly hardware upgrades, and reduce overall TCO.

SDX Overview

The Citrix ADC SDX platform delivers fully isolated network instances running on a single appliance. Each instance is a full-blown environment, which optimizes the delivery of applications over the internet and private networks. The SDX platform combines application-level security, optimization, and traffic management into a single, integrated appliance. The SDX appliance is architected such that each instance runs as a separate virtual machine with its own dedicated kernel, CPU resources, memory resources, address space, and bandwidth allocation. Network I/O is done in a way that not only maintains aggregate system performance but also enables complete segregation of each tenant's data and management-plane traffic.

A Citrix ADC can be connected to a network using various of methods such as one arm mode or two-arm mode. Citrix ADC requires multiple IP addresses to function on a network. The most important IP addresses are:

• NSIP (ADC IP): There must be only one NSIP assigned to each instance, used for management. NSIP addresses are not shared between a High Availability (HA) Pair.
• VIP (Virtual Server IP): Virtual Server IPs are used to host services on Citrix ADCs. Examples would be a Load Balancing Virtual Server, SSL VPN Virtual Server, and so on. VIP addresses are shared between a High Availability (HA) Pair.
• SNIP (Subnet IP): This IP address is used to access a particular subnet. This IP is used as the source address on the network when accessing resources on the particular subnet configured for use with the subnet IP. SNIPs are shared between a High Availability (HA) pair.

Citrix Application Delivery Management Analytics (Insight) Overview

Web Insight

Provides visibility into enterprise web applications allowing IT administrators to monitor all web applications using the Citrix ADC by providing integrated and real-time monitoring of applications. Web Insight provides critical information such as user and server response time, enabling IT organizations to monitor and improve application performance.

HDX Insight

Provides end-to-end visibility for ICA traffic passing through Citrix ADC. HDX Insight enables administrators to view real-time client and network latency metrics, historical reports, End-to-end performance data, and troubleshoot performance issues.

Gateway Insight

It provides visibility into the failures that users encounter when logging on, regardless of the access mode. You can view a list of users logged on at a given time. Also the number of active users, number of active sessions, and bytes and licenses used by all users at any given time.

Security Insight

It provides a single-pane solution to help you assess your application security status and take corrective actions to secure your applications.

SSL Insight

SSL Insight provides visibility into secure web transactions (HTTPS). It allows IT administrators, to monitor all the secure web applications being served by the Citrix ADC by providing integrated and real-time and historical monitoring of secure web transactions.

TCP Insight

TCP Insight provides an easy and scalable solution for monitoring the metrics of the optimization techniques and congestion control strategies (or algorithms) used in ADC instances to avoid network congestion in data transmission.

Video Insight

The Video Insight feature provides a secure and scalable solution for monitoring the metrics of the video optimization techniques used by Citrix ADC instances to improve customer experience and operational efficiency.

WAN Insight

WAN Insight analytics enables administrators to easily monitor the accelerated and unaccelerated WAN traffic that flows between the data center and branch WAN optimization appliances. WAN Insight also provides visibility into clients, applications, and branches on the network to help troubleshoot network issues effectively.

GDPR is a set of data privacy rules that apply broadly to both companies in the European Union (EU) in addition to any company globally that collects and uses data pertaining to EU residents. The GDPR went into effect on May 25, 2018 and includes several chapters which are further broken down into numbered “articles” or subsections that we will refer to in this document. These articles describe the specific requirements applicable to the handling of personal data.

GDPR seeks to

• Unify the different data protection regulations adopted by EU member states
• Protect the data privacy of EU residents
• Ensure that organizations handle personal data in a responsible and accountable manner from collection through to return or destruction

The GDPR applies to

• Organizations operating within the EU
• Organizations operating outside of the EU who offer goods and services to EU residents

Note: Privacy considerations for GDPR span the data lifecycle from collection, usage, storage, and secure disposal and retirement of data. It covers all personal data relating to your customers, employees, supply chain, partners, and anyone else about whom you collect personal information who resides in the EU. Personal data is any information relating to an identifiable person, therefore it is often referred to as personally identifiable information (PII). This can include information such as names, photographs, IP or email addresses, and medical information. For further detail on Citrix’s data lifecycle processes and practices, visit the Citrix Trust Center.

Two key GDPR articles are the focus of this document

• Access control (included in Article 25) - calls out measures which shall ensure that by default personal data is not made accessible to an indefinite number of persons.
• Encryption and data protection (included in Article 32) - calls out:
  • Pseudonymization and encryption of personal data
  • Ensure the ongoing confidentiality, integrity, availability, and resilience of processing systems and services
  • The ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident
  • Protection from accidental and unlawful destruction, loss, alteration, unauthorized disclosure of, or access to personal data transmitted, stored, or otherwise processed

How Citrix can help you with your GDPR compliance initiatives

Citrix Workspace simplifies the management of your systems and data by centralizing services in the data center or cloud as a digital workspace. The goal of this document is to describe how it unifies applications, data, and desktops into a digital workspace for your teams. One that allows you to better align with GDPR requirements around data management, data monitoring, and information auditing.

Citrix supports clients on their journey to GDPR compliance in 4 key ways

• By centralizing and enclaving applications and data
• By helping to ensure data is protected when shared or distributed
• By controlling who has access to data and resources
• By bringing IT together for application and data-specific security

Citrix Workspace - helping to enable GDPR compliance

Data oriented approach to GDPR requirements

Following the GDPR guidelines might be much easier for modern cloud companies than traditional enterprises. While most cloud companies have only a few centralized data sources where personal data is stored, traditional companies potentially have hundreds or thousands. These different data sources need to be assessed, reviewed, and updated to meet the latest data privacy standards.

These data sources can range from traditional SQL databases to emails, digital documents, or even physical documents. With today's often aggressive timelines, many enterprises face a challenge to properly prepare and update systems as required. It is important to understand that the GDPR doesn’t affect only active data sources, but also all backups, disaster recovery sites and physical printouts.

GDPR involves two key roles for data: the data controller and the data processor. The data controller is the entity who determines the purpose of the data and how the data is to be handled. The data processor handles or processes the actual data per the controller's guidelines.

The GDPR is all about increasing the maturity of the company when dealing with data and personal information. Citrix has always been a data and application oriented company, with a proven record of handling complex, often international projects that are dealing with thousands of applications.

The traditional consulting approach is focused on the business processes, identifying people and business requirements, access methods and slowly cascading down to infrastructure and data sources. However, the GDPR requires a more data centric approach. We recommend starting with identifying and assessing various locations where personal data is stored. Then move higher up the stack to make sure that data sources are properly secured. You can think about this as an inside out approach to security.

Define - Start by defining the criteria of personal data that is in scope of the assessment. This phase helps you define what to look for and how to prioritize data sources from a privacy perspective. This can include employees, customers, vendors, and any other relevant entities.

Assess - Analyze all the existing locations where data is stored. Identify the business requirements, data retention, and potential challenges to securing the data. Identify not only where the data is stored, but also how it’s being collected. Data segmentation is one of the most time consuming and critical phases of data consolidation projects. This phase requires a comprehensive approach, critical thinking, and well-defined methodology. Organizations need to consider data held in legacy systems. This is true even if there is a program in place to modernize or if the data is used only as a backup. It is important to understand that all these legacy systems are covered by the GDPR requirements as well and companies need to take a holistic approach.

Reduce - The goal of this phase is to identify if it is possible to reduce the number of data sources that need to be secured. For example, it is possible to consolidate data sources that are being used by business units. Instead of storing client data in multiple locations, a centralized location can be used to maximize effectiveness of security measures. Maybe the data is not even needed at all – the biggest privacy offender might not even be considered critical for the business units. It is also possible that applications are simply collecting too much data (“just in case”). The applications can be modified to stop collecting excessive information and the existing data can be erased. Instead of trying to secure all the possible locations of personal data, companies can take a different approach. Instead, ask when and where do they actually need to store data about customers and other parties. As GDPR compliancy is an ongoing process with periodic reviews, minimizing the number of included data sources can prove to be an effective long-term strategy.

Remediate - Identify if existing data sources and applications used to access them are following the GDPR guidance, or if changes are needed. If the data source includes personal data and is not secure, identify the possible approaches to solve the situation. A cross-departmental GDPR team can also identify, assess, and review not only the data itself, but also access methods, applications used and other factors. This includes items such as limiting user and third party access, revisiting requirements, and more specifically defining data security measures.

Review - GDPR compliance is an ongoing process and data assessments need to be performed regularly. It is therefore important to implement a robust, stable, and repeatable process that can be defended if it ever needs to be presented to auditors. The data sources security assessment needs to be performed and reviewed regularly.

With the large number of data sources that are in scope, the goal for most companies is to choose a few, robust, and proven architectures. This approach can help secure data sources that don’t initially meet GDPR requirements. Trying to create a tailor-made solution for each of the problematic data sources is unrealistic, unless a company has limited data sources. The result is often an environment where only a few applications are properly secured, while the majority is left unsecured, with the implementation project stalling by months or even years and going well over budget. GDPR also presents an opportunity to update privacy architectures across applications and data usage to support evolving global and regional privacy initiatives.

Complexity is considered one of the biggest enemies of security. You want to identify the minimum number of different architectures to secure most of the data sources identified as critical and storing data included in the GDPR scope.

The Pareto principle (also known as the 80/20 rule) is important during this data assessment. Companies need to try to minimize the effort required to secure most data sources. Most enterprises have hundreds or thousands of different applications and data sources that are used. They need to promptly identify the applications that contain critical data and don’t meet the GDPR requirements. Automated application assessment solutions can reduce the time required to analyze applications.

In the following sections, we present a few selected architectures that can provide a universal, secure, and proven solution to help secure any type of data. This ranges from web-based applications, through legacy client/server applications hosted on Windows or Linux to data stored in various documents or exchanged through emails.

Decision Flow for Data Types

Securing Windows and Linux Applications

Trying to secure traditional client/server applications, whether they are running on Windows or Linux operating systems, can prove challenging for various reasons.

The traditional approach was to secure each endpoint where these applications are installed. This involves management challenges, such as keeping all the endpoints up to date, encryption of the network traffic, data and workload encryption, implementing multifactor authentication (MFA) and encryption of the locally stored or cached data. In traditional IT architecture, defenses need to be set up around all endpoints, applications, and networks and the whole environment is only as secure as the weakest point. This traditional approach to security has often failed due to the introduction of new concepts including mobile workforce, expansion of security perimeter with cloud computing and BYOD initiatives.

Another common challenge for applications installed on traditional computers is to provide the same security functionality across the whole portfolio. It’s common to have a multigenerational IT portfolio residing on a single workstation. From Office-based applications (using Microsoft Access databases or custom plug-ins) through legacy Visual Basic to the latest professionally built applications. Making sure that applications with access to sensitive data support encryption, multifactor authentication and provide enough information for auditors has always been complicated.

Traditional Client/Server App Delivery

Citrix has a long tradition of providing a platform for the secure delivery of these client/server applications. This secure delivery is based on offloading the client application piece onto a dedicated set of servers (Citrix Virtual Apps and Desktops), specially designed, optimized, and secured for application delivery. By decoupling the application from the endpoint, it's possible to enable extra security features. The advantage of this approach is that security features are applied consistently. There is also no requirement for source code access and even applications that are no longer supported on newer platforms can be included.

Securing Applications to GDPR Standards

Article 25 - Access to Personal Data

There are multiple ways to limit or prevent users from accessing published resources. The most basic method is to simply hide the applications or desktops from users by enforcing Active Directory group membership. When publishing resources through the Citrix management console, the access is enforced on the Citrix Virtual Delivery Agent (VDA) machines hosting the workloads. Access and available functionality is further tweaked through Citrix's comprehensive policy engine.

The use of traditional user name/password authentication is decreasing with more secure multifactor authentication (MFA) increasing. Even for internal networks, more companies are enforcing MFA requirements to enhance security. With Citrix Virtual Apps and Desktops and Citrix Application Delivery Controller (ADC), MFA can be applied to any client/server application including even legacy applications that are hard to maintain. The ADC appliance provides an extensible and flexible approach to configuring MFA, from time-based one-time tokens, through smart cards, user or machine certificates to biometric authentication (through third party integration).

This access can also be configured based on various other factors. For example the endpoint a user is connecting from, the security state of endpoints such as antivirus or firewall requirements or the network where the user is connecting from. Context-aware policies can be applied, even enforcing a specific geo-location or using more advanced security measures, such as requirements for user or machine certificates to access certain resources. You can learn more about context-aware security through the zero-trust model in the following blog post.

Even more flexibility is available through Citrix ADC’s enhanced MFA feature call nFactor authentication. To learn more about different capabilities of nFactor authentication, refer to the following knowledge base article and one of the many deployment guides: https://support.citrix.com/article/CTX201949

The ability to deliver centralized access and authentication is critical in providing information about users connecting to applications. With Citrix Virtual Apps and Desktops, all access to resources is brokered through a controller with historical data saved in a centralized database. This data can be accessed from an ODATA API to provide integration with SIEM systems or provide copies for auditors if needed. To learn more about monitoring and reporting, refer to the doc on how to Monitor historical trends across a Site.

Aside from the monitoring and reporting of user access, all administrative changes and activities can be logged to a separate database. It is recommended to enable mandatory logging, where administrative activities are not allowed unless they are logged in the Configuration Logging database first. To learn more, refer to the Configuration Logging Documentation

Finally, for the most security-conscious environments, it is possible to create a separate set of user identities and automatically switch to them. This is done using the Federated Authentication Service. The approach can be used to further minimize the impact of lateral movement and contain any security breach.

Article 32 - Data Encryption in Transit

With Citrix Virtual Apps and Desktops, only screen pixels are transferred between the hosting server and the endpoint. Connection parameters are established during session initiation or reconnection. CVAD can ensure that traffic coming to and from the endpoint is always encrypted, even if the application itself doesn’t support encryption. This encryption can be enabled for any published application or desktop. For details on end-to-end encryption, refer to this document

Article 32 - Data Encryption at Rest

While Citrix Virtual Apps and Desktops can help with the encryption between user and application, the back-end itself remains out of scope for this solution. However, CVAD can be used to isolate unencrypted traffic and data during the transition period utilizing secure zones. This makes sure that all data is encrypted with a long-term process. Encapsulating this data in an isolated enclave can provide the required security while the migration project is underway. You can read more about secure zones in this blog post: “Unsinkable”: The Myth of Foolproof IT Security.

As for encryption on the endpoint, it is important to minimize data exposure at the endpoint and control data remanence. Data residing on the endpoint needs to be restricted, delivering only the minimum amount of data necessary. Virtualizing all access to personal data and then managing and protecting residual data, keystrokes and screen data is the Citrix approach. To learn more, read this blog post about the Citrix ICA client footprint.

Article 32 - Data Isolation and Protection

Contrary to traditional desktops on physical endpoints, server, and desktop OS images within Citrix Virtual Apps and Desktops usually have a much more restricted scope of operation. They are used to host a group of well-defined, centrally managed applications and desktops with predictable behavior and centralized configuration options.

There are different ways how data and applications can be protected and isolated from each other. With aggregation of resources from multiple servers, it is possible to create groups of separated servers to host different applications with different trust levels.

Even if applications are hosted on the same server, it is common practice to isolate and secure them. CVAD servers are used to host well-defined, centrally managed sets of applications. This means that these servers can support more restrictive security hardening than traditional workstations. Application whitelisting solutions such as Citrix's Workspace Environment Management (WEM) Application Security feature are much more useful with servers built for specific applications rather than general workloads. Allowing only specific white-listed executables is much simpler on these special-purpose built servers than general workstations.

Security is enhanced even more by application of granular Citrix policies. These policies provide control over many aspects of the workspace: available printers, ability to access network and local drives, or clipboard mapping among many more. A special template for “Security & Control” is included with all the best practices and recommended settings.

To learn more about hardening, refer to the VDA Hardening Guide.

Given that users often connect from untrusted devices and locations, extra security layers are often welcomed. Citrix App Protection Policies provides anti-keylogging and anti-screen-capturing while users are connected in virtual app and desktop sessions. Sensitive data, such as credit card numbers, is hidden from malicious programs attempting to intercept. Session Watermarking allows administrators to place customizable text overlays in sessions. Session Recording saves the entire video stream of a session to a file for later playback. These features help deter and remediate possible data exfiltration events.

Aggregation of Applications with Different Trust Levels

Securing Web and SaaS Applications

Web apps are architecturally different from client/server apps yet also similar in many ways – including suffering from security challenges. With web apps, a specific client is replaced with a single corporate standard or multiple general-purpose web browsers, with varying capabilities and dependencies. Although simplified, the same management challenges apply – keeping the browser up-to-date against vulnerabilities, encryption of traffic, and implementing multifactor authentication.

The demands for legacy application support and modern capabilities for SaaS have driven conflicting requirements. There are two types of web apps – the born-on-the-internet-apps and webified apps-custom and legacy web apps that support the business. The born-on-the-internet apps drive the requirements on security and architecture – load balancing, scalability, failover, and performance. Webified apps drive the requirements on supportability - browser plug-ins, extensions, and validating browser updates can break functionality.

The goal is for the end user to interact with web apps and manipulate data. This includes personal and sensitive data regardless if running in legacy environments or on SaaS apps pushing the limits of HTML5. Gartner recommends a two-pronged strategy. This is when an organization uses a legacy browser for running legacy applications, but also employs modern browsers for all other applications. That’s where Citrix helps – tying together the user experience and security requirements for hybrid or bimodal web environments.

For the enterprise, maintaining multiple versions of various browsers, plug-ins, and applets while also simplifying authentication and access is a pain point. It is addressable with Citrix Virtual Apps and Desktops or Citrix Secure Browser offerings. These solutions allow organizations to build a remote browsing infrastructure that separates internet and intranet web traffic from each other and the endpoint.

Traditional Web Applications Architecture

The second challenge is providing unified security features across the wide range of SaaS and cloud applications that typical enterprises are using. This effectively helps to give back some IT control – especially in BYOD and mobile environments. An example is providing a unified multifactor authentication solution across all apps instead of a fragmented user experience.

Citrix has a long tradition of providing a platform for secure delivery of web apps. This secure delivery is based on the Citrix Application Delivery Controller (ADC). Citrix ADC secures the session between the browser and the web app – by encrypting data in transit, maintaining strict access control, and data protection. This is largely based on its design as a reverse proxy that brokers connections coming from the browser to web app servers. And, with its position between the client and the server, extra security features can be enabled.

We’re going to cover how this architecture can help you secure the applications to follow GDPR standards. These technical measures fall largely under the requirements or Article 25 and 32 where controllers are required to “implement appropriate technical and organizational measures.”

Web Applications with Citrix ADC

Article 25 - Access to Personal Data

Authentication, Authorization, and Auditing are all core to controlling access to personal data. With Citrix ADC AAA proxy, it consolidates, extends, and enhances the traditional authentication schemes even in scenarios where the web apps do not natively support MFA. Citrix ADC supports authentication using user name/password, multifactor (MFA), time-based and one-time tokens (Citrix ADC has native One-time PIN support), smartcards, user or machine certificates and biometrics. While this is especially important for internet facing web apps, some organizations with a zero trust networking approach are moving to require MFA for “internal” access.

Citrix ADC's enhanced MFA, called nFactor authentication, takes into account capabilities such as SAML, client certificates, group extraction, and multiple passwords. Federation and SSO also provide an extra level of security and ease of use.

To learn more about different capabilities of nFactor authentication, refer to the following knowledge base article and one of the many deployment guides: https://support.citrix.com/article/CTX201949. On a related note, SMS-based MFA is not recommended as it has been deemed insecure by NIST.

To understand how Citrix protects personal data, refer to the Citrix Trust Center/Privacy Policy.

Logging, visibility, automation, and other capabilities are provided by Citrix Application Delivery Management (ADM). Refer to the product landing page for more details on Citrix ADM.

Article 32 - Data Isolation and Protection

Citrix ADC is a reverse proxy and as such it benefits from its location in the network architecture. Typically it is in a DMZ or security zone. From here it accepts the front-end user connection, creates a secure connection to the back-end server, and has full visibility into requests and responses. Also, Citrix ADC can change the logic of the web traffic on the fly without requiring updates to the back-end application. This includes encryption of not only the packet header but also the body as it does deep packet inspection and rewrite.

Citrix ADC can ensure that traffic coming to and from the browser is always encrypted, even if the web server itself doesn’t support encryption. This encryption can be enabled for any site proxied through the ADC. SSL offloading uses the ADC to perform the resource intensive SSL/TLS handshakes thereby offloading them from the back-end servers. For scenarios requiring end-to-end encryption, Citrix ADC can re-encrypt the connection to the back-end. This allows the ADC to inspect and apply security policies to the traffic. SSL bridging is available for when requirements demand that the ADC plays no part in terminating the connection. Using Citrix ADC with Citrix ADM allows administrators to keep central configuration and visibility of the cipher suites in use, helping prevent negotiation of outdated ciphers.

Citrix ADC Encryption Options

As a proxy between the browser and the web app, Citrix ADC protects the data flowing through it. That includes protecting from attacks against databases, attacks against the web app, and other users using its built-in application firewall. Citrix ADC protects against common web attacks including SQL Injection and cross-site scripting. You can read more about the Web App Firewall in our product documentation.

Protecting data also includes maximizing availability through Denial of Service (DoS/DDoS) attack protections. Combination attacks hit at all layers—so Citrix ADC provides Application layer defense (Layer 7), Transport layer defense (Layer 4) and Network layer defense (Layer 3). Citrix ADC not only provides a multi-layer approach to DDoS protection but it is coupled with a built-in IP Reputation service. It is an effective tool in identifying the IP address that is sending unwanted requests. Since most malware comes from compromised sites, you can use the IP reputation list to preemptively reject requests that are coming from the IP with the bad reputation. Citrix ADC's forward proxy, Secure Web Gateway, can filter out connections going out to the internet based on reputational risk. This enforces security policies on outgoing web traffic, while blocking access to inappropriate sites on a per user/group basis.

Citrix ADC Tokenization

Citrix ADC Pseudonymization

Pseudonymization is another control mentioned in Article 32. Conceptually, it’s a procedure by which identifying fields within a data record are replaced by one or more artificial identifiers, or pseudonyms. This makes storing personal data more secure in the event of a breach – by using data segmentation. An example is tokenizing or hashing sensitive data that Citrix ADC parses for web-application traffic. This means hashing personally identifying data while transmitted between a controller and a processor. This is done in PCI-DSS regulated environments. For example, for cardholder data, tokenization guidelines are specific for the Primary Account Number (PAN). Tokenization replaces the PAN with a surrogate value called a token. De-tokenization is the reverse process of redeeming a token for its associated PAN value. The security of an individual token relies predominantly on the infeasibility of determining the original PAN by knowing only the surrogate value. Applications may not need as much security protection as associated with the use of PAN. For GDPR, storing tokens instead of personal data is one alternative that can help to reduce the amount of personal data in the environment, potentially reducing the effort required to adhere to GDPR requirements.

Citrix ADC Data Protection

Securing Mobile Applications

Mobile devices, particularly with BYOD ownership, present many challenges to enterprises trying to secure data. Their use within the enterprise has driven the inception of technologies to securely manage mobile endpoints. Used beyond the borders of enterprise DMZs on any network, with apps from various sources, mobile devices present special risks to companies and their data.

• The GDPR data controller must secure personal data used by corporate mobile apps despite the fact they’re hosted on a user-owned mobile device.

• The GDPR data controller must ensure the confidentiality, integrity, and availability of the personal data during its use. It must also ensure that when a user exercises their right to erase their personal data that no artifacts are left behind and exposed to other apps, users, and so forth.

• Data controllers must support file sharing and collaboration securely between enterprise mobile apps and be able to erase files from the device in a moment’s notice.

• They must help protect the platform OS, mitigate the risk of malware, and enforce device security and pertinent policies to control device functions that make data vulnerable to loss.

• Data controllers must provide Unified Endpoint Management across multiple platforms including control of critical software patches that include updates to address vulnerabilities.

Traditional Mobile Application Architecture

Citrix Endpoint Management (CEM) is a market leading Unified Endpoint Management (UEM) component of the Citrix Workspace. It can help you secure and manage various mobile endpoint platforms ranging from iOS, Android, Windows, and Mac to rugged mobile devices and IoT devices. CEM also manages various mobile apps on endpoints and supports various delivery mechanisms including virtualized, web & SaaS, public app store, native enterprise mobile apps, and containerized mobile apps.

In the following sections, we discuss how this architecture can help you secure personal data on mobile endpoints.

The Citrix MDX Toolkit, the Citrix Endpoint Management MAM container technology, is a key part of the Citrix Endpoint Management solution to protect data. It provides end-to-end security maximizing protection of personal data, mitigating the risk of loss, by encrypting apps and data and managing the transfer of data through 70+ MDX Policies. These include functional areas such as Authentication, Device Security, Networking, Encryption, Access Thresholds, App Interaction, App Restrictions, and other app-specific policies. All of these are applied on a per-app basis designed to mitigate the risk of personal data loss.

MDX Technologies help provide end-to-end protection by managing encrypted data transfers between device and intranet data stores, in addition to between managed apps. Once these apps are installed, Secure Hub, a mobile app that provides access to desktops, apps and data, helps continuously enforce the desired policies. IT is always in control of the enterprise content on users’ devices. MDX also includes micro VPN, a per-app VPN that technology that integrates with Citrix Gateway. It is utilized seamlessly by managed apps to encrypt data traffic to and from the enterprise intranet.

Article 25 - Access to Personal Data

Citrix Endpoint Management provides various enrollment methods to validate user identity before initiating Mobile Device Management or Mobile App Management and then access to secure data. For example, a two-factor authentication solution can include One-time PIN (OTP) enrollment invitations along with Active Directory domain credentials. For environments with the highest security requirements, enrollment invitations may be linked to a device by SN, UDID, EMEI to uniquely identify the hardware.

Citrix Endpoint Management also provides a variety of multifactor authentication options to validate the identity of enrolled user devices. These include combinations of domain user name and password, RADIUS, Azure Active Directory, certificate, or derived credentials (a high security federal standard based on government issued personal identity verification cards). Certificate and domain authentication used with a CEM pin is a popular secure combination that provides a great user experience.

Article 32 - Data Encryption in Transit

Citrix Endpoint Management supports data encryption in transit through several methods such as:

• Containerized with embedded VPN when apps utilize the CEM SDK
• Platform-based utilizing a Citrix ADC VPN client
• Through policies to utilize native platform OS VPN functionality

The Citrix Endpoint Management SDK, or MDX technology, with micro VPN provides secure per-app VPN functionality to encrypt data in-transit between the mobile endpoint and intranet back-end. It works with Secure Hub and Citrix ADC to ensure MDX app traffic is directed over a dedicated encrypted VPN. It is unique Citrix technology that provides seamless encryption of data in transit.

For more information see this micro VPN FAQ; configuration of Android platform per-app VPNs using Citrix VPN for Android or iOS; or the configuration of platform per-app VPNs using native functionality.

Article 32 - Data Encryption at Rest

Citrix Endpoint Management (CEM) supports data encryption at rest through the CEM MDX with Citrix-provided encryption libraries, or through platform level encryption directly or indirectly with partner containerization solutions.

CEM can provide encryption at rest on any supported mobile device independent of platform encryption. The CEM secure app container technology, MDX, uses its own software applied data encryption using FIPS compliant algorithms minimizing the risk of data loss.

Device level encryption varies by platform. Apple's iOS features a file system with the OS information and user data written to flash memory. It also uses a factory-assigned device ID and group ID with the device user's passcode so only that passcode can unencrypt data on the phone or tablet. Android also provides encryption, although not every device manufacturer creates hardware that supports it and users can turn encryption off accidentally or deliberately with a factory reset on Android devices. Find more information about the MDX Toolkit, MDX policies, and integrating with MDX in Citrix documentation.

Article 32 - Data Isolation and Protection

Containerization enables mobile BYOD programs in corporate environments empowering users to use mobile endpoints as an enterprise device and personal device simultaneously by separating apps and data. It helps enterprises prevent malware, intruders, system resources or other applications from interacting with the application and any of its sensitive information. Citrix Endpoint Management enables containerized native mobile apps through MDX technology, and it also integrates with several partner container solutions providing further value by integrating many broad app and device management capabilities.

Securing Files with Workflows

Many of our daily workflows consist of the creation of files and collaborating on those files with others. Often those files contain personal data, such as name and address, Social Security numbers or credit card details. Unfortunately, there are many examples of data leakage occurring, from lost USB drives filled with files containing personal information to phishing attempts to file access on secured systems with employee permissions. Under GDPR it is necessary to secure and control this information in every step of the process. This includes storing these files inside a repository to internal and external collaboration and providing context-based access to the files. It also means monitoring for irregular activities and reporting on who has which permissions to which files.

Citrix Content Collaboration (ShareFile) provides a range of controls to help organizations become and remain compliant under GDPR. This starts by having a choice on the location where files are being stored. Options include inside one of the Citrix-managed StorageZones in different global regions or in a StorageZone managed by the customer in their own data centers or private cloud. Multiple locations can be used, allowing for the optimal location to store each individual file. By using the StorageZone Connectors technology to access existing repositories, such as network file shares or SharePoint document libraries, all file related activities are done through a single platform. This makes auditing the activities easier, in addition to making sure that the correct permissions are in place.

StorageZones Options

For more information on StorageZones including architecture details and deployment options, see the Content Collaboration Reference Architectures

Collaboration on files has not changed much over the years. Most of these workflows use email to send files to a group of recipients. Feedback is gathered from each of those recipients as separate emails to the thread and update the files before starting the cycle again. As such, multiple messages and copies of the same document are stored inside the email platform, which makes it more difficult to comply with GDPR policies. By using the ShareFile Feedback and Approvals workflow to collaborate on documents, all feedback and document revisions are stored in a single place, making it easier to comply with such regulations.

Many paper-based workflows in an organization contain personal data in some form. For instance, the workflow to hire people involves multiple steps where personal information needs to be recorded and shared. All this information needs to comply with GDPR regulations, centralizing and digitizing these workflows have a positive impact. ShareFile Custom Workflows is therefore designed to allow this personal data to be securely captured, securely stored inside ShareFile and, where needed, completed with an electronic signature. All information is stored together in a single location and is audited for who accesses and modifies this information, providing a practice that complies with GDPR.

Article 32 - Data Encryption in Transit

All connections between ShareFile clients and the Content Collaboration SaaS Control Plane, between ShareFile clients and ShareFile StorageZones, in addition to the Content Collaboration SaaS Control Plane and ShareFile StorageZones are fully encrypted. See CTX208317 and the Citrix ShareFile Security and Compliance FAQ for further details.

Citrix Files (ShareFile) clients for iOS and Android, which can be managed by Citrix Endpoint Management, also use the embedded VPN capabilities provided by the CEM SDK. See “Data Encryption in Transit” in the Securing Mobile Applications section of this document for more details.

Article 32 - Data Encryption at Rest

Citrix Content Collaboration offers a flexible architecture which provides customers the choice of where files are stored at rest. These repositories are called StorageZones and are managed by either Citrix or the customer.

For StorageZones managed by Citrix, hosted in either Amazon Web Services or Microsoft Azure, files are stored at rest with 256-bit AES encryption. The encryption key is a shared key for all files stored across every ShareFile tenant. Alternatively, this can be a customer-managed encryption key, configured in the Amazon Key Management Service. When the StorageZone is managed by the customer, per-file encryption can be enabled inside the StorageZone configuration. When enabled, files are encrypted with 256-bit AES encryption.

Files are not only stored at rest inside the repository in the data center or cloud, but also on the devices used by employees. As a best practice, it’s recommended to always use full drive encryption on Windows and macOS devices. On top of that, Content Collaboration allows controls for both corporate-owned and BYOD devices. It allows for a remote wipe of the user files, removing only the corporate files and not touching the personal files of the employee. When a remote wipe is initiated, the Citrix Files client sends back all file activity that has occurred offline between the wipe command and the actual wipe of the user data repository. This occurs when ShareFile logs on to the ShareFile SaaS application tier.

Similar safeguards are in place with the Citrix Files app for iOS and Android. All files at rest are encrypted by using the device keychain and encryption capabilities. When using a Citrix Endpoint Management-managed version of Citrix Files, the encryption key is stored inside Secure Hub. And because ShareFile provides a robust rendering and editing engine for Office files and PDF documents, there are many advantages. With the mobile Citrix Files clients, files don’t need to leave the applications for reviewing or editing. ShareFile offers multiple mobile device management options to secure the files, for instance by blocking access from jailbroken devices and blocking opening files in other applications. When using Citrix Endpoint Management, other advanced policies for more granular control are available.

Article 25 - Access to Personal Data

Authentication to ShareFile is either controlled by a user name and password (ShareFile credentials) or by using corporate credentials through a SAML Identity Provider. When using ShareFile credentials, the password for the user is subject to the password policy that has been configured. This password policy controls the requirements for the password in terms of complexity, history and how often it must be changed. The password is stored hashed and salted inside the ShareFile SaaS application tier for enhanced security.

SAML based authentication is commonly used for authentication to cloud services. Instead of authenticating directly to the enterprise directory, such as Active Directory, the authentication is done against an Identity Provider. This removes the need to expose the enterprise directory directly to ShareFile, but still allows users to authenticate with their enterprise credentials. The Identity Provider controls how the user must identify and authenticate itself, based on the context of that authentication attempt. This allows for extra security measures like multifactor authentication for authentication attempts from outside the corporate network and SSO based on the Windows authentication token for domain-joined devices.

Article 32 - Data Isolation and Protection

ShareFile integrates with market-leading Data Loss Prevention products for customer-managed StorageZones and Cloud Access Security Broker Services for any type of ShareFile StorageZone, enabling content-aware restrictions. Documents stored inside a ShareFile StorageZone are examined by the same policies that are already set up for other repositories. Based on those scanning results, files can be blocked for download or shared with others.

Sharing files is a key component of modern workflows. This makes controlling the access and permissions to documents containing privacy related information a priority, especially when the files are outside the direct control of your own security policies. With ShareFile Information Rights Management (IRM) watermarking, documents are protected via watermark with an online view only option that helps address unauthorized access including various image capture techniques. It also facilitates forensic investigations as needed to comply with GDPR.

ShareFile uses versioning to store different versions of the same file. This is not only convenient to review changes made to documents, but this can help when recovering from a malware or ransomware attack. By restoring the files to the state before the attack, data loss is minimized. Recovery time is reduced by automating the restore to previous versions by using the ShareFile PowerShell cmdlets.

For customers requiring all files to be archived for compliance purposes, ShareFile offers this capability. When a user deletes a file, or when the file is automatically deleted by a retention policy, the file is stored inside an archive instead of being fully deleted from ShareFile. Dedicated auditors can review the contents of the archived files, including access permissions, during an investigation.

Summary

Citrix Workspace simplifies the management of your systems and data by centralizing services in the data center or cloud as a digital workspace. It helps Citrix customers adhere to many GDPR requirements by helping to ensure that applications are centralized and enclaved, data is protected when shared or distributed, access to data and resources is controlled, and IT is brought together for application and data-specific security.

To learn more about security and compliance with Citrix secure digital workspace solutions, visit https://www.citrix.com/it-security/.

To learn more about Citrix’s approach to data management, including security documentation, privacy and security compliance, and vulnerability management, visit https://www.citrix.com/about/trust-center/.

Additional Links

Citrix Trust Center and Privacy Policy

Top 10 findings from Citrix environment security assessments

Citrix Ready Security Solutions

Citrix Global Partners

This document is intended for technical professionals, IT decision-makers, partners, and system-integrators. This document also allows the administrator to explore and adopt the Citrix Analytics service with other Citrix portfolio products. Citrix Analytics enhances the security of an organization’s Citrix environment by efficiently monitoring and managing the risk factors. The reader need to have a basic understanding of Citrix portfolio products and solutions.

Objective of this document

This document covers a technical overview, architectural concepts, and capabilities of the Citrix Analytics service. This document includes a tailor-made solution with other Citrix solutions that help administrators and users to understand and adopt in their Citrix environment.

Intelligent threat detection and mitigation

In today’s era, organizations are more concerned about the security and privacy of information, and many organizations are betting on best-in-class defensive security solution in their environment. One of the emerging technologies is an intelligent threat detection platform. Such a platform helps many organizations to aggregate, correlate, and analyze threat data from different sources to take relevant defensive actions.

In the dynamic environment of IT, the threat factors keep changing, advanced threats focused on public and private organizations are growing at a faster rate. The organization needs a complete security solution that mitigates and encounters any dangerous acts. The solution safeguards the organization's intellectual property, sensitive information, and financial data.

Machine learning and Artificial intelligence

Many organizations have faced (and continue to face today) various cyber-attacks. Intruders are adopting automation and scripts in their attacks and increase their speed and scale. The organization must mitigate and be able to respond in real-time at CPU speed to counter these kinds of aggressive attacks. Machine learning and artificial intelligence can help and enables the organization to counter the attacks and build defensive walls effectively.

Adoption of machine learning and artificial intelligence improves the security of the IT environment. Workforce errors can be mitigated and reduced. Machine learning and artificial intelligence can help in areas such as risk analysis, anti-malware, and anomaly detection.

Artificial intelligence can be applied to differentiate between normal and abnormal behaviors in the environment. Machine learning can be used to recognize these behaviors and provide a layer of security to network and software applications in the background. Machine learning uses stored logs/records and learns from the analyses to predict the data in the future.

The preceding diagram depicts a conceptual machine learning platform. In general, a machine learning platform caters for analyzing input data from data collection points (data sources). In later stages, it segregates data based on the type of applications. The platform keeps updating the models and profiles based on the data and the results. Machine learning techniques are applied in multiple ways that are specific to the requirement.

By applying machine learning techniques on more massive sets of data, an organization can be more efficient in their threat intelligence compilation and threat investigation. In this way, organizations can be more proactive in their approach to security threats and concerns.

Introduction to Citrix Analytics

Many organizations are facing cyber threats from all over the world. In real-time, it is challenging to identify insider threats as it may be even more damaging than external threats. Standard analytics often fails to expose those threats before severe damage to the system. The organization has to adopt user behavior analytics delivering proactive, secure insights. Standard analytics solutions primarily focus on security, and resolution does not provide visibility into the user session and information on user activities. Eventually, the IT team loses control over the performance and operations of the IT environment.

Citrix has developed a turnkey solution that works across the Citrix product portfolio. Citrix Analytics collects data across Citrix portfolio products and third-party products. Citrix Analytics allows administrators to detect, analyze, and proactively respond to security threats across Citrix environments.

Citrix Analytics enables administrators to handle user and application security threats, improve app performance, and support continuous operations. Citrix Analytics is available as a cloud service delivered through Citrix Cloud.

Citrix Analytics offerings

Security Analytics

Security Analytics provides visibility into user and application behavior. The administrator can distinguish between normal behavior and a malicious attacker. An inbuilt machine learning platform that proactively identifies and manages internal and external threats.

Performance Analytics

Performance Analytics provides visibility into user session details across an organization. Metrics collected by analytics engines help to identify issues that arise during a user’s login session.

Operations Analytics

Operations Analytics provides information on user activities such as websites visited and bandwidth consumption. Metrics that are received from data sources help to monitor networks and take corrective actions.

The preceding diagram depicts the Citrix Analytics service that is a cloud-based service which works across Citrix portfolio products and third-party products. It collects data from different data sources and detects abnormal behaviors of a user or any other entity. This process uses Machine Learning (ML) algorithms that continuously monitor the customer environment.

Data Governance and Data Sources

Data governance provides information regarding the collection, storage, retention of logs, and protects the gathered data by Analytics service. Security administrators can choose the logs that have to be monitored and take representative action based on the logged activity.

Most of the organizations have some form of data governance for individual applications or sections. Data governance is an essential task to perform during the project implementation. Few of the obligatory topics encompassed by data governance are:

• Data sources
• Data privacy
• Data transmission
• Data control
• Data retention
• Data storage
• Data quality and types

Reference: Data governance

Data Sources

Data sources are the services that send data to Citrix Analytics. Services that are running on cloud or in the on-premises locations that become a data source to Citrix Analytics by enabling certain functions within the product.

Services that are running on Citrix Cloud, including Content Collaboration, Endpoint Management associated with the Citrix Cloud account, are automatically discovered by Citrix Analytics. Other on-premises services such as Citrix Gateway and Citrix Virtual Apps and Desktops can be added as data sources to Citrix Analytics.

The preceding diagrams show external data sources such as Microsoft® Graph Security and Microsoft® Active Directory are part of Citrix Analytics data sources. Citrix Analytics captures data from these external data sources after successful integration.

Reference: Data Sources

Tech Concepts

This section discusses the architecture and services offered by Citrix Analytics.

Citrix Analytics for Citrix portfolio products

Citrix Analytics can be integrated with multiple Citrix and Microsoft® products. It collects metrics on users, applications, endpoints, networks, and data to deliver comprehensive insights into user behavior. Citrix Analytics, as of today, supports the following products:

• Citrix Secure Private Access
• Citrix Content Collaboration
• Citrix Gateway
• Citrix Virtual Apps and Desktops
• Citrix Endpoint Management

Citrix Analytics creates profiles of the users and applications across the network. Profile creation is only possible with information/data collected from the data source (user behavior information). This profile contains information about the devices, files, locations, and so on. To mitigate the threats in the network, the pattern generated by analytics provides high visibility and take necessary action. This service provides complete visibility over user behavior in the environment.

The preceding picture depicts the integration of Citrix products with the Citrix Analytics cloud service. Typically, end-users use their own devices to connect to Citrix Workspace and access required resources. Those services communicate with the Citrix Analytics Service hosted in Citrix Cloud. Also, customers can tie back on-premises Citrix Virtual Apps and Desktops environment to communicate with the Citrix Analytics service. Connecting on-premises resources requires either site aggregation or an on-premises StoreFront and Citrix Analytics agent installed on the Delivery Controller.

Citrix Analytics service receives logs directly from the data sources. Captured data remain in the databases for 13 months. Citrix Analytics uses these metrics for analysis through a machine learning platform and can perform actions when deviant or suspicious activities occur.

Citrix Analytics and Microsoft® products integration

Most organizations today rely on a diverse portfolio of security solutions that include endpoint protection, network firewalls, identity, access controls, cloud security, and so on. In the end, it tends to increase cost and complexity. Along with that, connecting multiple security tools and workflows becomes a challenging task for the IT team. To overcome these challenges integration process is simplified. Citrix Analytics integration with Microsoft® products helps unification of security and incident management which results in simplified reporting and analytics.

Citrix Analytics supports the integration of Microsoft® products that include Microsoft® Graph Security and Microsoft® Active Directory. Currently, it supports Azure AD Identity Protection and Windows Defender ATP from Microsoft® Graph Security. To enable this service on Microsoft® products, the customer must have enabled Citrix Analytics Service from Citrix Cloud. For more information, refer to the following link.

Enable Citrix Analytics on Microsoft® Graph Security

Microsoft® Graph Security API provides a standard interface and uniform schema to integrate security alerts, unlock contextual information, and simplify security automation. Microsoft® Graph Security aggregates data from multiple security providers, including Microsoft® Defender ATP, Office 365 ATP, Azure ATP, Microsoft® Intune, Azure Sentinel, and so on.

Microsoft® Graph Security API can easily be coupled with Citrix Analytics. The Microsoft® Graph Security acts as a data source for the Analytics service that transmits data from Security Providers. Currently, this solution supports the following security providers from Microsoft® Graph Security:

• Azure AD Identity Protection
• Windows Defender ATP

The preceding diagram depicts the Microsoft® Security Graph solution with Citrix Analytics to enhance overall analytics capabilities. The administrator can have clear insights of user behavior during the resource utilization that includes applications and end-user desktops. Citrix Analytics pulls the data from Microsoft® Graph Security hosted on Azure. Both the products work on the same platform that even makes it easier for integration.

Citrix Analytics UI provides processed data based on risk score indicators from high, medium, and low. Based on the risk score, value analytics can perform actions on that particular user. With the external feeds from Microsoft® Graph Security to Citrix, Analytics administrators can start to have a holistic picture of the access to resources by a user.

Reference: Integration of Microsoft® Graph Security

Integrate Analytics with Microsoft® Active Directory

The organization can connect Microsoft® Active Directory service with Citrix Analytics, resulting in an enhancement in the user profile in Citrix Analytics. User profile in Citrix Analytics encompasses imported information such as user information and user groups data. In case risky users are identified by Citrix Analytics, imported information like job title, organization, office location, email, and contact details help in gaining visibility of the user profile.

Security Analytics has provisions for monitoring privileged users. This functionality enables the administrator to monitor behavior anomalies of privileged users closely. For example, if a privileged user starts deleting files and folders excessively, the machine learning platform detects unusual behavior and triggers an alarm.

Reference: Privileged users in Citrix Analytics

The preceding diagram depicts the integration of Microsoft® Active Directory with Citrix Analytics. Before enabling this service by the administrator, Cloud Connectors are installed on-premises to pull the information. On successful integration, the Citrix administrator has to turn on data processing on the Citrix Analytics UI so that they can monitor and troubleshoot risks identified in the environment.

To learn more about integration with Microsoft® Active Directory, refer to this link.

Analytics

Countless use cases exist by adopting Citrix Analytics in a Citrix environment. The Analytics Service is divided into three areas:

• Security Analytics
• Performance Analytics
• Operations Analytics

Each offering is discussed in the following sections:

Security Analytics

Security Analytics aggregates data from endpoints for security monitoring and threat detection. Many deployments have a different set of tools that incorporate large and diverse data sets into the algorithm. As technology changes security analytics include adaptive learning skills. That helps to calibrate detection models, based on learnings and logic behind anomaly detection.

Citrix Security Analytics receives data from multiple data sources and displays actionable insights. Machine learning algorithms within Security Analytics detect and takes predefined actions on user behavior. Risk score indicators that are dynamic based on users, user behavior, endpoints, network traffic, and files.

Security Analytics supports integration with the following:

• Citrix Content Collaboration
• Citrix Endpoint Management
• Citrix Secure Private Access
• Citrix Gateway
• Citrix Virtual Apps and Desktops
• Microsoft® Graph Security
• Microsoft® Active Directory

To learn more about Security Analytics, refer to the following link.

User risk indicators

User risk indicators are user activities that look suspicious or can pose a security threat to the organization. User risk indicators span across all Citrix Products used in the deployment. The signs are based on user behavior and are triggered where the user’s behavior deviates from the norm. User risk indicators help in determining the user’s risk score.

User risk indicators occur based on the following categories:

• Access-based
• Data based
• Application-based

The preceding diagram depicts user risk score indicators. The indicators are based on user behavior and are triggered where the user’s behavior deviates from the norm.

Reference: Citrix user risk indicators

Policies and actions

A policy is defined as a set of conditions that must be met for an action to execute. A policy contains a single condition and one or more actions. The administrator can create single policy with multiple actions that can be applied to a user’s account.

Policies on Citrix Analytics help to perform actions on user accounts when unusual or suspicious activities occur. Once policies are applied, the action is triggered immediately after an unexpected event occurs.

As mentioned, the policy is a set of conditions. The risk score and risk score change are global conditions applied to a specific user for a particular data source.

Reference: Policies

Actions help to respond to suspicious events and prevent future anomalous events from occurring. The administrator can take action on user accounts that display unusual or suspicious behavior. It is up to the admin to apply action automatically, or manually depending on the conditions.

Reference: Actions

Performance Analytics

Performance Analytics is a powerful tool to find the root cause for end-user experience issues. Also, it quantifies user experience, and app performance gives users end-to-end visibility. Performance Analytics supports multi-site aggregation and reporting so that data represented from multiple sites or multiple sources in a unified display.

The user experience score is calculated based on latency, logon duration, reconnections, and failures. The exact root cause of the problem identified by keenly looking into the metrics. For example, logon duration includes: brokering, VM Start, HDX Connection, Authentication, GPOs, Profile Load, and so on.

The preceding diagram shows the UX score compilation by getting information from an end user’s latency, logon duration, failures, and reconnections. It allows the administrator to drill down further to find the root cause of the issues that users are experiencing. Performance analytics is available for both on-premises and cloud-based Citrix Virtual Apps and Desktops environments.

To learn more about user experience, refer to the following link.

Operations Analytics

Operations Analytics is a more specific term oriented to business analytics that focuses on improving existing operations. Operations Analytics involves the use of various data aggregation to get more transparent information for day to day IT services. Citrix infrastructure consists of multiple products that are an amalgamation of different sets of workloads, and the administrator needs to have insight into the environment. Also, many administrators fail to focus on improving and optimization of the Citrix environment.

To overcome such problems, Citrix has embedded Operations Analytics into an analytics solution. The analytics solution contains machine learning algorithms and delivers actionable insights into the operational data of customers Citrix environments.

For example, if an enterprise is using the Citrix Secure Private Access service, admins can use the operations dashboards to get insights into user operations and application operation data. The administrator has thorough visibility of the data consumption (download & upload), domains accessed, and other available metrics according to the data sources. These metrics help in procuring and providing resources and quickly responding to any operations issues.

In another way, operations analytics support the idea of enterprise resource planning. Operation Analytics aggregates information to enhance the proactive approach towards resource management.

The preceding diagram depicts Operations Analytics. It has two dashboards that are:

User Operations: Provides an overview of the user operations data based on transactions and data usage volume.

App Operations: Provides an overview of app operations data based on domains, categories, and download volume.

Reference: Operations Analytics

Citrix Analytics integration with Citrix Products

Citrix Analytics integrated with the other Citrix components labeled “data sources.” The following products supported by Citrix Analytics service and provides insights about user behavior in the Citrix environment.

• Citrix Secure Private Access
• Citrix Content Collaboration
• Citrix Endpoint Management
• Citrix Gateway
• Citrix Virtual Apps and Desktops

In this section, the integration of Citrix Analytics with other Citrix products discussed.

Citrix Analytics and Citrix Secure Private Access

Citrix Secure Private Access service enables the administrators to provide a cohesive experience integrating single sign-on, remote access, and content inspection into a unique solution for end-to-end Secure Private Access. Administrators can protect the organization’s network and end-user devices from malware and data leaks by filtering access to specific websites and website categories.

Citrix Secure Private Access and Citrix Analytics solutions gives clear insights into user behavior and monitors the entire network. The inbuilt capability of Citrix Analytics that uses machine learning helps to take corrective actions. Citrix Analytics uses similar metrics and collected by the Secure Private Access service. The parameters of activities of users, such as websites visited, and the bandwidth spent. It also detects malware and phishing sites.

The preceding diagram depicts a holistic view of Citrix Secure Private Access integration with the Citrix Analytics service. Citrix Secure Private Access and Citrix Analytics Services are hosted on Citrix Cloud. Data processing can be enabled by administrator with few clicks on the Analytics UI. Then Citrix Analytics starts capturing the data from Secure Private Access.

Security Analytics

The following are policies that an administrator can create to take actions based on a user’s activity.

• Attempt to Access Blacklisted URL: Policy that can enable to indicate when a user attempts to access a blacklisted URL

• Risky Website Access: This policy suggests that the user tried to access malicious, suspicious, or unsafe websites with high reputation ratings. (URL reputation rating is given for websites. The values range from 1 to 4. 4 is the riskiest website, and 1 is a clean site

• Unusual Download Volume: The volume of data downloaded by the user from an application or websites has exceeded the threshold defined implicitly by Citrix Analytics

• Unusual Upload Volume: The amount of data uploaded by the user from an app or sites has surpassed the limit set implicitly by Citrix Analytics

To learn more about Citrix Analytics integration with Citrix Secure Private Access, refer to the following link.

Citrix Analytics and Citrix Content Collaboration

Citrix Content Collaboration enables us to quickly and securely exchange documents, send extensive material by email, and safely handle document transfers to third parties. In the Citrix Workspace environment, the user can access all of their files from the Citrix Workspace app. To track the user’s behavior and activity, it is cumbersome. In a small scale or large-scale Citrix environment, threats hidden behind the screen are tough to discern and take precautionary measures.

Citrix Content Collaboration can be monitored and troubleshot with the help of Citrix Analytics. Citrix Content Collaboration service hosted on Citrix Cloud nimbly integrates with Analytics service by enabling the data sources.

The preceding diagram depicts the integration of Citrix Content Collaboration and Citrix Analytics service. Security Analytics as part Analytics service support monitoring and troubleshooting of user behavior and activities.

Security Analytics

The following are policies that an administrator can create to take actions based on the user’s activity.

• Excessive file downloads: This policy indicates an attempt to download data that exceeds the AI-based threshold for this period

• Excessive file/folder deletion: This policy indicates an attempt to delete a disproportionate number of files or folders, which exceed the AI-based threshold for the user

• Excessive file sharing: This policy indicates that the share links created and shared have exceeded the threshold set by the machine learning algorithms for this period

• Excessive file uploads: This policy indicates an attempt to upload data that exceeds the AI-based threshold for this period

• Excessive login failures: the user makes multiple failed login attempts

• Ransomware activity suspected (File Replaced): This policy indicates an effort to replace existing files with encrypted versions, resembling a ransomware attack

• Excessive access to sensitive files (DLP alert): indicates an attempt to access files deemed confidential more than the threshold defined in the Citrix Content Collaboration Data Loss Prevention (DLP) policy

• Unusual login access: This indicator is triggered when the user has suspicious access to Citrix Content Collaboration account identified by the Analytics AI engine, based on user’s usage locations and behavior patterns

• Ransomware activity suspected (Files Updated): This policy indicates an attempt to update existing files with encrypted versions, resembling a ransomware attack

In case any unusual behavior or suspicious activity detected by Citrix Analytics service, then it can protect the data by disabling the user and expire all links.

Citrix Analytics and Citrix Endpoint Management

Citrix Endpoint Management is a solution for managing endpoints, offering mobile device management (MDM) and mobile application management (MAM) capabilities. With Endpoint Management, the administrator can manage device and app policies and deliver apps to users.

Usually, the end-user might change device or install one of the blacklisted apps. Such incidents trigger an alarm in the Endpoint Management environments. Most of the time, it may go unnoticed to Citrix administrators or the possibility of manual errors.

In such cases, when an administrator has to manage thousands of endpoints, it becomes a burden. This process tends to increase much working hours managing the devices. To overcome these problems, Citrix Endpoints Management can be integrated with Citrix Analytics service by the administrator. That helps in detection of unmanaged device, blacklisted app installed device detection, and applying actions on such device can be automated.

The preceding diagram depicts Citrix Analytics integration with Citrix Endpoint Management (Citrix Cloud service). From the Analytics UI with just a few clicks, the Endpoint Management service can be monitored and detect any suspicious activities on endpoints. This monitoring service starts on any device Android, iOS, managed, or unmanaged, so that the administrator will have a clear view of devices such as jailbroken, blacklisted apps.

Security Analytics

The following policies that an administrator can create to take actions based on the devices and applications.

• A device with blacklisted apps detected: This policy indicates the detection of a device with blacklisted applications on the network using Citrix Endpoint Management service

• Jailbroken or rooted device detected: This policy indicates the detection of a jailbroken or rooted device on the network using Citrix Endpoint Management service

• Unmanaged device detected: This policy indicates that an unmanaged device discovered by Citrix Endpoint Management service in your network

When any of the conditions are met, the administrator has an option to give a notification to the administrator and user. The administrator can also lock that particular device.

Citrix Analytics and Citrix Gateway

Citrix Gateway service provides secure remote access solutions with a diverse Identity and Access Management (IdAM) capabilities. Gateway service helps in delivering a unified experience into SaaS apps, heterogeneous Virtual Apps and Desktops, and so forth.

Citrix Gateway’s End Point Analysis (EPA) scan policies help to detect user access-based threats and reports in the UI. Similarly, Citrix Gateway detects all the user logon failures, which can be primary, secondary, or tertiary authentication failures. Also, authorization failures distinguished from the Citrix Gateway.

Citrix Gateway metrics on user logon activity are collected by Citrix Analytics to perform an automated task that would not be practical for an administrator to do manually.

The preceding diagram depicts Citrix Analytics integration with Citrix Gateway Service. The administrator has to enable data processing so that Citrix Analytics starts capturing the data from the Gateway Service.

Security Analytics

The following policies can be created by the administrator to automate the monitoring activity.

• Authorization failures: This policy indicates that the Analytics AI engine identified that the user has attempted to access a resource without sufficient permissions

• EPA scan failures: This policy shows an attempt to access the network using a device that has failed Citrix Gateway’s End Point Analysis (EPA) scan before or after authentication

• Logon failures: This policy indicates that the Analytics AI engine identified multiple primary authentication failures based on the user’s usage and behavior patterns

• Unusual login access: This policy suggests an attempt to logon to Citrix Gateway from significant locations that deviated from the user’s access pattern

With the help of Citrix Analytics, when any of the conditions crop up, then the administrator can take action on that user by enabling “Log off user” from the Analytics UI. Else administrators can examine that user’s activity with the help of selecting “Notify administrator.”

Citrix Analytics and Citrix Virtual Apps and Desktops

Citrix Virtual Apps and Desktops are virtualization solutions that give IT control of virtual machines, applications, licensing, and security, while providing anywhere access for any device. Citrix Virtual Apps and Desktops allow end-users to run applications and desktops independently of the device’s operating system and interface. Similarly, administrators have the advantage of managing the network and control access from selected devices or all devices.

Citrix Analytics for Virtual Apps and Desktops framework gives an insight into user activities. Based on user behavior, alarm, or notifications are triggered when the user’s behavior deviates from the normal behavior. App-based risk indicators are triggered when users attempt to access an unauthorized application over a specific period. Data based risk indicators are generated for unusual data upload and download with a large volume.

The preceding diagram depicts Citrix Analytics integration with Citrix Virtual Apps and Desktops. Based on the user's attempt to access resources with a device that has unsupported operating systems, new equipment, an alarm triggered with notification. The inbuilt machine learning platform keeps feeding the latest update into its database, and profiles are updated. In the future, any anomaly in the environment can be easily detected and mitigated without admin intervention.

Security Analytics

The following policies can be created by the administrator to automate the monitoring activity.

• Potential Data Exfiltration: This policy indicates an attempt to exfiltrate data from Citrix Workspace to an external device or location

• Access from New Device(s): This policy indicates an attempt to log on to Citrix Workspace from a new device

• Access from a device with unsupported OS: This policy indicates an attempt to access the network launching receiver from a device with an unsupported OS version or unsupported browser version

• Unusual Application usage (SaaS): This policy indicates that the user used applications at achieved times that deviate from the user’s usage and behavior patterns identified by Citrix Analytics

• Unusual Application usage (Virtual): This policy indicates that the Analytics AI engine identified usage of applications at great times based on the user’s usage and behavior patterns

Citrix Analytics can perform actions on the user’s account when the preceding conditions are encountered. During the policy creation the administrator can enable the action when any conditions are met.

Reference: CVAD Risk Indicators

Performance Analytics

Performance Analytics gives an insight into user session details in the Citrix environment. The data collected by Citrix Analytics helps to monitor and troubleshoot issues that arise during a user’s login session.

The following are metrics that Citrix Performance Analytics provides to the administrator.

• User Experience: This offering provides insights into the user and session performance parameters — a profound view of all sites in the organization within a concise dashboard. The user experience score helps to segregate the users based on their experience namely Excellent, Fair, or Weak

• User Sessions: The User Session section of the User Experience dashboard displays an important session metric for the chosen period and Site. The administrator can view Total Sessions, Total unique users, and Session failures data

• Session Responsiveness: This data represents the ICA Round Trip time. The information used to quantify user experience. The Session Responsiveness has Active Session, Session classification, and Session classification trend. If any session is facing network issues, the session responsiveness trend data helps to identify such problems in the network

• Session Logon Duration: Logon duration is the app or desktop availability after the user clicks in the Citrix Workspace app. Total logon time includes phases such as Brokering, VM Start, HDX Connection, Authentication, Profile Load, Logon Script, GPO, and Shell Launch. This section has Total logons, Session classification, and sorted by Delivery Groups

Reference: Performance Analytics

Citrix Analytics and on-premises Citrix Virtual Apps and Desktops

Many organizations having on-premises Citrix Virtual Apps and Desktops environments can take advantage of the Citrix Analytics Cloud Service. Citrix Analytics supports and discovers data sources automatically.

To enable Analytics on Virtual Apps and Desktops Sites administrator can adopt one of the following methods to onboard on-premises Virtual Apps and Desktops Sites to Citrix Analytics:

• Onboard Sites using Workspace
• Onboard Sites using StoreFront

Onboard Virtual Apps and Desktops Sites using Workspace

Citrix Analytics automatically discovers the sites once added to Citrix Workspace. The Virtual Apps and Desktops site card displays the number of discovered Sites and users. To add Sites in Citrix Cloud the customer needs to have a Workspace subscription. The administrator has to do Site aggregation before proceeding with onboarding on Citrix Analytics.

The administrator has to turn on data processing on the Site card. Once the Citrix Analytics start receiving the events, collected data can be viewed based on the selected time. A policy agent installed to configure the policies and this step is not associated with data transmission from the data sources. To know more about policy agent and installation, refer to the following link.

Onboard Virtual Apps and Desktops Sites using StoreFront

Another method that StoreFront can become a data source to Citrix Analytics is by aggregating applications and desktops from Citrix Virtual Apps and Desktops Sites into a single store for users. User events captured by Citrix Analytics process through to get actionable insights into user behaviors. There are a few prerequisites that the administrator has to configure before enabling this method. On the networking front, StoreFront deployments must have TCP port 443 open for outbound internet connection. In case environment uses proxy servers on the network, administrator has to allow a particular port of communication.

From the Citrix Analytics service, the administrator has to connect to an on-premises StoreFront deployment. To enable this feature, import the configuration settings. That makes Citrix Analytics receive the data from StoreFront.

To learn more on Citrix Virtual Apps and Desktops as a data source, refer to the following link.

Onboarding Citrix Performance Analytics

Performance Analytics is a comprehensive monitoring solution. Performance analytics helps in monitoring and viewing the usage of Citrix Virtual Apps and Desktops Sites in the organization. To enable these capabilities, the administrator has to configure on-premises sites with Citrix Analytics from the Citrix Director console. This feature requires Director version 1909 or later, Delivery Controller and VDA version 1906 or later.

Reference: Configuring on-premises Sites with Citrix Analytics for performance

Key Benefits of Citrix Analytics

Citrix Analytics is an intuitive analytics service that allows administrators to monitor and identify inconsistent or suspicious activity on the networks. A turn-key machine learning platform that provides actionable insights into user behavior based on indicators across users, endpoints, network traffic, and files.

An organization can solve business challenges and performance-related issues by adopting analytics solutions built-on a machine learning platform. There are various benefits that organizations and administrators can realize by selecting Analytics Service in Citrix brownfield deployment or green field deployment.

• Detect and analyze threats based on user behavior Detect and prevent ransomware attacks by taking security measures. Monitor and analyze user access, and authentication behavior helps to stop malicious activity and prevent any data loss. Automated platform with the help of ML and AI algorithms, any anomalies are detected

• Health visibility Advanced analytics helps to identify issues proactively. Application Performance Analytics distills app data and many real-time performance metrics into a single-digit App Score that shows responsiveness of an app. Admins and app owners can drill down into the infrastructure associated with specific apps to troubleshoot issues

• Tighter external security Citrix Analytics generates a threat index based on violation type, rate of attack, location, and client details

• Improved internal security Citrix Analytics collects a wide variety of data from many sources surrounding user activity, access behaviors, and network and data usage patterns. The automated machine learning platform helps to mitigate any unusual occurrences. Information on UI helps administrators to understand the activities/events on the environment

• Central management and automation Citrix Analytics collects different metrics from multiple resources, that help an administrator to monitor all Citrix portfolio products from a single UI

• Cloud service Citrix Analytics is a SaaS offering from Citrix Cloud that has security, performance, and the operational dashboard. The UI provides a summary and categorization of the user risk profiles, user experience details, and so on in the Citrix environment

Summary

Many of the global security challenges are not addressed by existing traditional threat protection tools alone. Most of the attacks come from external actors that possibly defended, but strikes from within the perimeter are even more menacing. While defending external threats, many organizations fail to recognize the impact on performance and identifying the root cause.

Citrix Analytics that comes with built-in machine learning enhanced with artificial intelligence holds great promise in addressing many security challenges in a Citrix environment. Citrix Analytics is not just about the security of a Citrix environment. It gives visibility into performance with user experience score and user operations visibility in terms of domains visited, data consumption, and so on. Integration of Citrix Analytics in green field or brownfield deployments helps the administrator to have greater control of the Citrix environment and reduce IT costs.

Sources

Goal of this reference architecture is to assist you with planning your own implementation. To make this job easier, we would like to provide you with source diagrams that you can adapt in your own detailed designs and implementation guides: reference-architectures_citrix-analytics.pptx

References

The following resources referenced for a better understanding of Citrix Analytics:

Citrix Analytics

Security Analytics

Operations Analytics

FAQ

When Covid-19 occurred, it forced all of Worldwide Co. employees to work remotely. Worldwide Co. office users typically worked from a corporate-owned PC associated with their cube/office. Worldwide Co. quickly deployed Citrix DaaS to allow users to securely access their work PC from home using Remote PC Access.

During the 2020 pandemic, Worldwide Co. realized that employees in certain roles were equally or more productive working from home than at the office. Therefore, they wanted to ensure their environment allowed for these new permanent remote workers.

Although many employees became permanent remote workers, a group of employees have roles requiring onsite, office work. However, Worldwide Co. wants to provide the office-based employees with the flexibility of working remotely as needed.

This reference architecture shows how Worldwide Co. planned their Citrix DaaS environment.

Success Criteria

Worldwide Co. defined a list of success criteria that formed the basis for the overarching design.

Conceptual Architecture

Based on their requirements above, Worldwide Co. created the following architecture. This architecture will not only meet all of the above requirements, but it will give Worldwide Co. the foundation they need to expand to other use cases as they are identified in the future.

The Citrix DaaS architecture is divided up into layers. This framework provides a foundation to understand the technical architecture for the most common virtual desktop/application deployment scenarios. All layers flow together to create a complete, end-to-end solution for an organization.

At a high-level:

• User Layer: This layer describes the end-user environment and end-point devices that are used to connect to resources.
  • External Users: Access Citrix Workspace to gain access to Azure Virtual Desktop hosted in Azure.
  • Internal Users: When in the office, continue to use their physical PC. When working remotely, they access Citrix Workspace and Remote PC Access to connect to their office-based physical PC.
• Access Layer: This layer describes details surrounding external and internal access to the Citrix environment.
  • Citrix Workspace: A complete digital workspace solution that allows you to deliver secure access to the information, apps, and other content that are relevant to a person’s role in your organization.
  • Gateway Service: This cloud-based service provides secure remote access with Identity and Access Management (IdAM) capabilities, delivering a unified experience to SaaS (Software as a Service) apps and virtual apps and desktops.
• Resource Layer: This layer defines the virtual desktops, applications, and data provided to each user group.
  • Remote PC Access: A traditional, local Windows desktop, assigned to a single user and can be physically accessed locally or accessed remotely.
  • Azure Virtual Desktop: virtualized Windows 10 multi-session operating system for users to be able to access their desktops and applications remotely.
• Control Layer: This layer describes details surrounding the components used to support the rest of the environment.
  • Citrix DaaS: This cloud-based service manages the authorization and brokering to Azure Virtual Desktops and Remote PC Access.
  • Workspace Environment Management Service: This cloud-based service uses intelligent resource management and Profile Management technologies to deliver the best possible performance, desktop logon, and application response times.
  • Citrix Analytics for Performance: This cloud-based service tracks, aggregates, and visualizes key performance indicators of the Citrix DaaS environment.
  • Citrix Analytics for Security: This cloud-based service assesses, detects, and prevents risks in real time. it gives you proactive security protection without complicating the employee experience.
• Host Layer: This layer describes the hardware components, private, public, and hybrid cloud that are used for the Citrix environment – hardware, storage, and virtualization details.
  • Physical PC: They use the physical PCs they already own but allow users to access these remotely when needed
  • Azure: to reduce their data center footprint, they deploy new virtual desktop resources on Azure.

In the sections below we go through each of the previous architectural components and how they meet Company XYZ’s requirements.

Detailed Architecture

User Layer

Aligning the user requirements with an appropriate virtual desktop is the initial step in creating a complete, end-to-end solution. Worldwide Co. defined the user requirements below.

Office workers typically work from the office with their corporate owned PC. When the pandemic happened, they needed a way to securely work from home while still using their PCs that were in the office. Worldwide Co. realized that office workers can be productive as remote workers and want to provide the flexibility of working remote. They continue to use their PCs locally when they work at the office and access them remotely through Citrix DaaS Remote PC Access when they work from home.

Employees are predominantly remote employees. Worldwide Co. doesn’t want to provide corporate owned devices, instead they want to provide these employees the choice to use whatever device they want. This can include devices such as personal laptops, smartphones, or tablets. Because Worldwide Co. wants to minimize their data center footprint, they have chosen to deploy Azure Virtual Desktop with Citrix Virtual Apps and Desktops service for this set of employees.

Access Layer

Providing access to the environment includes more than simply making a connection to a resource. Providing the proper level of access is based on where the user is located in addition to the security policies defined by the organization. Worldwide Co. chose to do the following:

• Minimize Data Center Footprint:
  • Gateway Service: Worldwide Co. decided to deploy Gateway Service to align with their goal of reducing their data center footprint. Gateway service allows them to provide secure remote access for their external users, without having to deploy and maintain any physical hardware, public IP address, or firewall rules. They also don’t need to worry about architecting for redundancy since Citrix takes care of that for them—Gateway Service operates in multiple regions worldwide with integrated redundancy. Gateway service minimizes the infrastructure required which provides administrators the agility to scale rapidly when needed (M&A, DR, new users, or contractors). More information on the Gateway Service can be found here.
  • Rendezvous Protocol: Worldwide Co. also turned on the Rendezvous protocol which allows the HDX session to bypass the Citrix Cloud Connector and connect directly to the Citrix Gateway Service. The Rendezvous protocol reduces the load on the Cloud Connectors, which helps reduce the data center footprint.
• Multifactor Authentication: Worldwide Co. decided to implement multifactor authentication to protect their intellectual property. They have chosen to do this through the Time-based One-Time Password microservice within Citrix Workspace. They chose TOTP because it allows them to meet their security needs without having to deploy or maintain other third-party systems. Additional information on TOTP and Workspace Identity can be found here.
• Optimal Routing- Gateway Service: Because the Gateway service is globally distributed, it allows the user to connect via the fastest access point which creates the best user experience.
• Secure Resources: All users authenticate with Workspace and Gateway service provides vpn-less access to their physical PCs and cloud-hosted VDI desktops. While this Reference Architecture only shows the users accessing virtual apps and desktops, Workspace gives organizations the flexibility to provide end users with SaaS and web apps all from one unified location. Also, it provides SSO so users don’t have to constantly reauthenticate over and over again.
• Business Continuity: Worldwide Co. also has taken advantage of the latest Service Continuity functionality within Citrix Workspace. Service Continuity further expands the Citrix Virtual Apps and Desktops service resiliency in case there is an outage with any of the following:
  • Citrix Workspace Portal
  • Citrix Cloud platform
  • Citrix Identity Provider Service
  • Citrix Virtual Apps and Desktops control plane
  • AWS and Azure platform

Worldwide Co. opted for this approach instead of Local Host Cache because Service Continuity doesn’t have any on-premises requirements. Essentially it uses long-lived connection tickets for workspace and connects users to their VDAs as long as there is a network connection between the endpoints and the VDA. More information on Service Continuity can be found here.

Resource Layer

Users need access to their resources, whether those resources are desktops or applications. Resources are configured within resource locations that are managed by Worldwide Co. The configuration of the resources must align with the overall needs of the user groups. End users expect an experience that is similar or superior to a traditional PC environment. Resources can be located on-premises, in private cloud, public cloud or in a hybrid approach. This is seamless to the end-user. Cloud Connectors are located within each Resource Location to connect the resources with Citrix Cloud. Worldwide Co. chose to do the following:

• Minimize Hardware costs:

  • Remote PC Access: Remote PC Access allows users to access their office-based, physical PCs.

  

  Users access through their own personal device and access through the Workspace App. After authentication, users would have access to their physical Windows desktops. Worldwide Co. followed the best practices found here (/en-us/tech-zone/design/design-decisions/remote-pc-access.html) for their Remote PC Access VDAs.

• Minimize Data Center Footprint: Worldwide Co. has chosen Azure as their other resource location. This allows them to quickly spin up new resources as needed without having to add new infrastructure. It gives them the flexibility to scale quickly and easily.

  

  Worldwide Co. used the following Design Decision guide when considering which instance series to deploy. Ultimately, they have chosen a D13_v2 instance with standard HDD disks and a 2GB MCSIO cache with a Windows 10 multisession OS. Worldwide Co. has chosen to have these be domain-joined to their on-premises Active Directory via Azure Active Directory Domain Services and users’ accounts in the organization’s on-premises Active Directory. More information can be found here.
• Optimize the images proviced to the end users: Worldwide Co. has chosen to use the Citrix Optimizer to optimize their VDAs. Information on the Citrix Optimizer can be found here.
• Adaptive session: Worldwide Co. used the baseline policies, however they turned on “Adaptive Transport”. Adaptive Transport allows the session to respond to changing network conditions. With remote workers, adaptive transport allows them to have an optimal user experience. They have also taken advantage of other HDX technologies to improve the overall experience.

Control Layer

With Citrix DaaS, the delivery controllers, SQL Database, Studio, Director, and Licensing are the core components in the Control layer. These components are provisioned on Citrix Cloud by Citrix during the activation of the Virtual Apps and Desktop Service. Citrix handles the redundancy, the updates, and the installation of these components. This allows the environment to always have the latest features and security patches. More services within Citrix Cloud can be enabled to support the requirements of Worldwide Co. Worldwide Co. chose the following:

• User Experience Reporting: Worldwide Co. chose to enable Citrix Analytics for Performance which allows them to quantify the end user experience and proactively address any issues. This information can be seen across both of their resource locations. More information can be found here.
• Optimal performance and app respoonse time: Worldwide Co. wanted to avoid a situation where a single user can monopolize CPU resources, which negatively impact other users (noisy neighbor syndrome). Therefore, they used the Workspace Environment Management service to enable CPU Management settings. More information on CPU management can be found here.
• Real-time Detection of Security Risks: Worldwide Co. wanted to guard sensitive data and to keep assets and employees secure in real time without complicating the employee experience. More information on real-time security analytics can be found here: Citrix Analytics for Security.

Worldwide Co. chose to domain join their Azure Virtual Desktops to organization’s on-premises Active Directory via Azure Active Directory Domain Services and keep their users’ accounts in organization’s on-premises Active Directory. The Active directory is synced with the Azure AD in the customer’s Azure subscription using Azure AD Connect. This setup allows the user’s identity to be authenticated from the synced Azure AD.

Host Layer

Administrators have the flexibility to deploy on-premises, in a public cloud, or in a hybrid approach. Worldwide Co. has chosen to do the following:

• Optimize Cloud Cost:
  • Autoscale: Worldwide Co. chose to deploy Autoscale to optimize cloud costs. Autoscale allows you to intelligently use, allocate, and deallocate resources. More information about Autoscale can be found here. Worldwide Co. will initially use the following schedule-based Autoscale parameters based on the typical workday:

To accommodate more users, Worldwide Co. also enabled load-based scaling with the following parameters:

• Azure sizing: Worldwide Co. chose to deploy D13_v2 instance with standard HDD disks and a 2GB MCSIO to provide the best user experience at the lowest cost. An in-depth analysis on the scalability of Citrix DaaS on Azure can be found here.

This document is intended to help Citrix partners and customers understand the most critical design decisions necessary to successfully deploy Citrix virtualization technologies on Amazon's public cloud. It is not meant to be a "How-To" reference - Citrix considers those Deployment Guides, and they are now delivered and maintained separately from Reference Architectures. In this document, we use the Citrix Architectural Design Framework to organize and present the leading practices, recommendations, and design patterns which are used by Citrix and select Citrix consulting partners.

Overview and Executive Summary

Citrix virtualization and networking technologies have been successfully serving the needs of enterprises large and small for nearly three decades. There are many ways in which these technologies can be licensed, deployed, integrated, and managed. This flexibility allows Citrix technologies to serve various different use cases, business types, integration requirements, and operational models. This paper is written for the Citrix customer or partner who's considering or planning to deploy these technologies on AWS' public cloud infrastructure.

For both existing customers looking to modernize their infrastructure and new customers looking to deploy Citrix virtualization and networking technologies, there are several key high and low level decisions which must be made along the way to help facilitate a successful deployment. To help customers and partners understand these decision points, we have introduced and defined some specific terminology to set the appropriate context, then used this context to highlight the critical decisions and implications to consider as you plan your deployment.

We start by defining two primary technology adoption models: Customer Managed and Cloud Services. We then break the components of a Citrix virtualization system down into multiple subsystems, and categorize them by adoption model:

We take a stand for cloud services, recommending that most organizations use or plan to use cloud services where feasible. We don't offer this stand blindly - while we do believe cloud services, in the end, offer overwhelmingly positive benefits for our customers, we recognize that not all organizations/deployments are able to use cloud services for all subsystems today. Sometimes, use case requirements (with technical attributes or shortcomings in a currently available/specific cloud service) need adopting a combination of cloud services and a customer managed component: we focus on these in this paper. In other cases, non-technical items (politics, budgetary/contractual considerations, cloud readiness deficiencies, regulatory/compliance considerations, and such) may discourage the usage of cloud services. In these instances, we recommend working with your Citrix partner/sales/engineering team to help overcome them. Through the rest of this paper, we go to great lengths to identify key capabilities, features, or attributes that help customers decide which adoption model to use for which subsystem and when.

Next, we define and examine three common deployment scenarios, highlighting which adoption model is used for each subsystem:

• Greenfield/Cloud only - uses cloud services for all Citrix virtualization system subsystems, plus AWS public cloud services.
• Hybrid (not to be confused with a 'hybrid cloud') - the most common deployment model, the hybrid model uses DaaS for session brokering and administration, with both customer managed and cloud service options for the remaining subsystems.
• Lift and Shift - as the name states, this model uses existing, customer managed CVAD, StoreFront, and ADC/Gateway and either migrates these components to AWS as is, or installs them into AWS as part of a workload migration to AWS public cloud services. While this is a valid deployment model for certain specific use cases, it comes with substantial caveats.

Finally, we use the well documented Citrix Architectural Design Framework to organize and present the key design decisions to be considered when deploying Citrix virtualization technology on AWS. We keep our focus on "what's different about Citrix on AWS" for clarity, providing links to other resources for more detailed information as needed.

We ultimately recommend that most customers use the Hybrid deployment model from day one, using the CVAD service for session brokering and administration. This provides the customer with the key capabilities necessary to cost-effectively run a Citrix virtualization system on AWS, substantially reduces the cost and complexity, provides access to the latest features and capabilities available, and simplifies the migration to and usage of other cloud services in the future. Either cloud services OR customer managed components can be used for the remaining subsystems (depending upon the customers' specific needs), though we recommend customers are clear as to why they're using customer managed components and have a plan to move to cloud services in the future once the cloud services meet their specific needs.

For more insights into leading practices for Citrix on AWS, readers can reference the following Cloud Guidepost articles:

• Leading practices for Citrix Cloud on AWS - Part 1
• Leading practices for Citrix Cloud on AWS - Part 2
• Leading practices for Citrix Cloud on AWS - Part 3

Key Concepts and Deployment Scenarios

In this section, we describe some key concepts and deployment scenarios before we dive into specific considerations for each layer of the Citrix Architectural Design Framework.

Technology Adoption Models

Citrix DaaS technology can be 'consumed' or implemented many different ways. The most common methods can be described generally as Customer Managed and Cloud Services. A third model is also worth mentioning - Partner Managed. We describe this model here for clarity, but from an architectural design perspective, the first two are the most relevant.

Why are we discussing technology adoption models in a reference architecture? The choice of adoption or 'consumption' model has a substantial impact upon the system design, capabilities, cost, and delineation of management responsibilities. This section will define and explore these models, and provide some general guidance to help customers make informed choices as they design a Citrix virtualization system.

Customer Managed

For many years, businesses wanting to consume technology had no choice but to buy, install, configure, and maintain the entire technology stack required to build a Citrix virtualization system. Citrix's virtualization software was only available as installable binaries. Customers who bought Citrix's virtualization software would download these binaries (often in the form of a downloadable ISO disk image or self-extracting executables) then install, configure, and maintain the software themselves. The Citrix software (and networking hardware) was most commonly installed into/on infrastructure the customer owned and maintained, in data centers they also owned and maintained.

Conceptually speaking, a Citrix virtualization system is made up of various different Citrix components, many of which we describe in detail in this paper. They also require different layers of third-party components which must be provided before you can do anything with the Citrix bits. Ultimately, they all come together to create a functional Citrix virtualization system. For the sake of clarity, this document refers to this technology adoption model as "Customer Managed". We use this term to describe various different components in a Citrix virtualization system, including the requisite third party components in the layers underneath, next to, and 'above' the Citrix components. This model can also be called "Self-Managed."

Today, customers have compelling alternatives to a customer managed adoption model, yet some still adopt elements of their technology stack using this model for various reasons. While this model provides customers with the most control over each component, it comes at a cost: the customer takes on the responsibility to manage and maintain the component, including securing, operating, patching, upgrading, and maintaining high availability. This model is also commonly deployed for 'air gapped' systems (those without any Internet access, and hence are limited in their ability to use cloud services, which are commonly and securely accessed over public networks).

Here's an example of the architecture of a Citrix virtualization system that's using 100% customer managed components deployed on AWS using basic AWS IaaS services such as Elastic Compute Cloud (EC2) and Virtual Private Cloud (VPC) networking. We are discussing some of the details of this architecture in later sections of this document, though you can immediately notice the similarities to the much simpler greenfield/cloud only deployment model:

Diagram 1: 100% Customer Managed, Lift/Shift deployment using AWS as IaaS only.

Cloud Services

Over the last 15 years, technological advancements across many different fields have given rise to hyper scale public clouds, sophisticated cloud services, microservice architectures, DevOPS/Agile delivery frameworks, subscription licensing models, and 'evergreen' software and systems. These advancements have revolutionized the way technology is acquired, adopted, delivered, and maintained across almost every industry in the world.

Today, many of the components or layers that comprise a Citrix virtualization system are available "as a Service." In contrast to the Customer Managed adoption model (where customers buy technology as a corporate asset and build/maintain systems themselves), customers "subscribe" to various services, and the service provider takes on the responsibility for delivering and managing these services. These services are often consumed over public networks (that is, the Internet) leading some to call this "Cloud Service" or "Web Service" adoption model. In this paper we're going to refer to this type of adoption model as "Cloud Managed Services," or simply the "Cloud Service" model.

Citrix offers many of its traditional products 'as a Service', using its platform partners' latest technological advancements to simplify and streamline adoption, accelerate the pace of innovation, improve quality, and deliver more incremental value to their customers over time. Citrix calls this service delivery platform "Citrix Cloud," and it represents the current and future state of the art from Citrix.

Here's an example of the architecture of a system that's using 100% cloud service components for a Citrix virtualization system on AWS. We are discussing the details of this design in a later section of this document:

Diagram 2: 100% Cloud Services on AWS with AWS Managed Services

Partner Managed

While many organizations choose to build their own Citrix virtualization system, some customers seek to get out of the business of managing IT so they can focus resources and attention on serving their own customers and markets. To serve these customers, Citrix works with integration partners who use Citrix technologies to provide a 'finished goods' service to their customers.

Defining and differentiating the different integration partners/types and offerings available are outside of the scope of this document. However, Citrix partners face the same choices customers face when designing a Citrix virtualization system. The Citrix partner can choose to use one or more services from Citrix Cloud, or they can choose to build and manage some components of the system for their customers' specific needs. As such, the guidance provided in this document is relevant to both the Citrix partner and their customers, just for different reasons.

Citrix Virtualization System Components

To understand the implications of the design decisions we detail later in this document, we're going to put the components of a Citrix virtualization system into higher level 'buckets' we'll then use to guide your decision-making process. Every Citrix virtualization system, regardless of how it is deployed and licensed, needs these components available to function and provide the best, most secure user experience. It is not uncommon for customers to mix and match self-managed components and cloud services, especially if they've got complex use case requirements, third party integration requirements, or extreme control or availability needs.

The following table highlights these key components for clarity. Details and recommendations on when/where you'd use one option vs another is covered later in this document:

Leading Practices and Recommendations

Whether you manage the Citrix virtualization system yourself or you use Citrix or an authorized partner to do it, consider using cloud services wherever possible. For use cases/environments where the cloud service doesn't meet your needs, customer managed components can be used. That said - Citrix encourages customers to be clear on why they are deploying self-managed components, and be prepared to migrate to cloud services once the cloud service meets their specific needs. The cloud services provided by Citrix through Citrix Cloud are evolving rapidly. Over time you can expect them to provide all the functionality required to serve all but the most complex use cases. Citrix Cloud services ultimately minimize the amount of infrastructure the customer is responsible for managing and maintaining. Citrix Cloud also provides highly available, pre-integrated services, and ensures customers always have access to the latest, most secure, and feature-rich services.

Common Deployment Models for Citrix Virtualization on AWS

As a cloud provider with the most functionality, largest community of customers, unmatched experience, and maturity, AWS sees a wide range of customers from various industries moving systems and infrastructure to their clouds. Over time they've seen common deployment scenarios/migration patterns develop. In this section, we explore these patterns/scenarios, discuss when and where you may want to consider using them to bring a Citrix Virtual Apps and Desktops workload to AWS, and provide some recommendations for which patterns to consider for common migration scenarios.

The three most common scenarios for delivering Citrix Apps and Desktops on AWS are:

• Greenfield/Cloud Only deployment, using Citrix Cloud services with "resource locations" on Amazon EC2 (Amazon Elastic Compute Cloud) service. This scenario is commonly used when customers prefer to go to a subscription model and outsource control plane infrastructure and management responsibility to Citrix, or they're looking to experience/evaluate the capabilities provided by Citrix Cloud services.
• Hybrid deployment/workload migration to AWS, using Citrix Cloud services for session brokering and administration, Workspace UI or StoreFront for content aggregation/session presentation/session launching, and can also use customer managed Citrix ADC/Gateways for HDX session proxying, complex authentication scenarios, or both.
• Lift and shift. With this scenario, customers essentially move or redeploy their self-managed Citrix infrastructure into AWS, treating the deployment on AWS just like their existing customer managed deployment. With this scenario, customers use Citrix ADC/Gateway and Citrix StoreFront to aggregate resources from on-premises and AWS hosted sites. This facilitates the migration of workloads to AWS, though customers can keep their on-premises workloads around and simply add another site in AWS. The new site can be used for new workloads or to support disaster recovery (DR) and failover use cases. This model is characterized by the use of customer managed components for session brokering and administration, UI services, authentication, and HDX session proxy.

This section defines these scenarios in more detail, including architectural overviews of how each scenario is commonly designed. You find that the leading practice is to use Citrix Cloud services, and as such this document will focus on the Citrix Cloud brokered deployment models ("Greenfield" and "Hybrid").

Greenfield Deployment

The most common example of the green field deployment model is trial or proof of concept deployments of Citrix virtualization technology on the AWS cloud. Since you're essentially starting from scratch, the power of 'infrastructure as code' can be experienced since you're not trying to integrate with a bunch of existing 'stuff'. It is also a fantastic opportunity to 'play with' various cloud services and evaluate their suitability to your or your customers' needs.

A green field deployment is also the quickest and easiest Citrix virtualization system you can build. You can simply tear it down when the system is no longer needed, and you stop paying for the resources it consumed. All you need for this type of deployment is an active AWS account and either a trial or paid subscription to Citrix Cloud and Citrix DaaS. Armed with these two resources, you can use AWS' QuickStart CloudFormation templates to build a reference deployment. Citrix and AWS have collaborated on the Citrix DaaS on AWS quick start template, which helps you to either build an entire Citrix virtualization system from scratch, or create a Citrix Cloud "Resource Location" in an existing VPC with an existing Active Directory. When deploying the entire Citrix virtualization system from scratch, the resulting system on AWS is built closely matching the following reference architecture diagrams:

Diagram 3: Deployed system architecture detail using the Citrix DaaS on AWS QuickStart template and default parameters. Citrix Cloud Services not shown.

Diagram 4: Greenfield/Cloud Only deployment conceptual architecture with optional AWS services and Citrix Cloud Services.

It is worth noting that this deployment model (actually, all three deployment models) use AWS Availability Zones to provide a highly available design. See Availability Zones later in this document for more context.

As mentioned previously, this is a great place to start when learning about AWS and Citrix Cloud services. Many of the design patterns depicted in the preceding diagram are used for hybrid and even lift and shift deployment types, so learning these design patterns suits a Citrix on AWS architect well, regardless of the deployment model.

To summarize, the green field deployment model uses all cloud services, at least as a starting point:

We mentioned earlier that the green field deployment model is often used as a starting point for proof of concept and technology trial systems. If you start with this model and then drop in StoreFront or Citrix ADC/Gateway VPXs in, you're ostensibly creating our next type of deployment model: hybrid.

Hybrid Deployment

With the hybrid deployment model, customers may choose to install/configure/manage some of the Citrix virtualization system components themselves, but not the session brokering and administration subsystem. In a hybrid deployment model, this subsystem is provided as a cloud service called "Citrix DaaS", and it is delivered as a subscription from Citrix Cloud.

The hybrid deployment model is the most common deployment seen today, and is the model Citrix recommends for most customers. Here are some of the primary reasons why we take this position:

• Simplicity - With Citrix Cloud services, simplicity is a foundational design tenet. When multiple cloud services are used, they come pre-configured to work together, and when configuration is necessary, workflows and options are dramatically simplified.
• Infrastructure and licensing cost savings - customer managed Citrix virtualization services often require extra hardware and software to support them, and these have costs associated with them. One good example is Microsoft SQL Server: customer managed brokering and administration services require databases, and if you're going to build/manage your own, you must provide them. An alternative is using the AWS Relational Database Service (Amazon RDS) for SQL Server.
• Autoscaling - Citrix's managed brokering service (DaaS) includes the Citrix Autoscale feature, which provides built-in VDA capacity and cost management functionality. This feature can save customers a substantial amount of money on infrastructure when they're only paying for what they use. When running a Citrix virtualization workload on AWS, this often means the difference between paying for committed use discounts or paying for VM usage as you go. The cost savings can be dramatic for many use cases, and the Citrix Autoscale feature helps ensure you're only consuming what you need.
• Management savings - With cloud services, Citrix shoulders the responsibility for keeping the services highly available, performant, secure, and up to date. You still build and manage your VDAs regardless, but don't underestimate the value of delegating these responsibilities. Cloud services help free up IT resources, allowing them to focus on providing unique value to their businesses instead of these critical but tedious (and often time-consuming) tasks.
• "Free" upgrades and continuous innovation - with customer managed infrastructure, the onus is on the customer to upgrade and patch the components in their care. With Cloud services, most of those work efforts go away. The service providers (Citrix or AWS for example) tend to be constantly innovating, and they bring those innovations to the customers who consume the services, often without requiring any work on behalf of the customer.
• Access to more features, functionality, and services - modern service delivery platforms (such as Citrix Cloud and AWS EC2) give technology providers a powerful, cost-effective way to bring new features, capabilities, and services to market that would not otherwise be possible. Vendors such as Citrix are committed to meeting the customer wherever they are at in their digital transformation journey, but sometimes the only way to cost-effectively deliver new capabilities is to deliver them as a cloud service.
• Flexibility - with DaaS as the foundation of this deployment model, customers can mix and match customer managed or cloud service components of the Citrix virtualization system. This allows the system to meet various different use cases and support complex enterprise requirements for a Citrix virtualization system. We explore these choices in depth in a later section of this paper.

To summarize, the hybrid deployment model uses the following:

Given the options a customer can choose in the hybrid deployment model, and the flexibility provided by customer managed components, there isn't one, succinct architecture that fits all customers. There are, however, some common design patterns that can also be mixed/matched to suit the customers' needs. The foundational pattern, however, is the pattern for a Citrix Cloud "Resource Location" on AWS. It is also the pattern built by the Citrix DaaS on AWS QuickStart template, and it looks similar to the following architectural diagram:

Diagram 5: Conceptual Architecture, Citrix DaaS - Hybrid Deployment Model on AWS.

It is worth noting that this deployment model also uses AWS Availability Zones to provide a highly available design. See Availability Zones later in this document for more context.

It is also worth noting that the hybrid deployment model (a DaaS resource location on AWS) can be combined with a hybrid cloud model, connecting customer managed data centers/resources to AWS using AWS Direct Connect, AWS VPN, or other networking tools. With this model, the customers' existing Active Directory is often extended into AWS, and customers create more Citrix Cloud 'Resource Locations' which deliver apps, desktops, and resources from the customer managed data center. The resulting conceptual architecture looks something like the following diagram:

Diagram 6: Conceptual Architecture, Citrix DaaS: Hybrid Deployment/Hybrid Cloud Model.

Lift and Shift

Referring to our definition of the Citrix virtualization system components, when we're talking about a lift and shift deployment scenario, the key component is the session brokering and administration subsystem and associated infrastructure. If you're using self-managed brokering infrastructure (you're deploying Delivery Controllers instead of Cloud Connectors) then for the purposes of this paper you're lifting and shifting.

Lift and Shift - why

Despite Citrix guidance against this model, some customers still choose to go with this model and deploy/manage the Citrix virtualization system components themselves. Per CTX270373, the use of public clouds including AWS is only supported with LTSR product versions. For customers who do choose the lift and shift (self-managed) deployment model, we often find that non-technical reasons are behind it. Politics, time pressures, fear of the unknown, perceived skills deficits, loss of control, and license acquisition fall into this category. There are, however, a few technical reasons why this model is appealing. These include:

• System Isolation - some use cases, such as air-gapped systems with no Internet access, often make the lift and shift model appealing. Since cloud services require outbound Internet access to function, in a strictly air-gapped deployment, cloud services won't function. This mainly applies to the Cloud Connectors (primary component of managed session brokering services) as they need outbound Internet connectivity to communicate with and utilize the Citrix Cloud services. Some customers can consider utilizing a secure, outbound proxy for Cloud Connectors (while keeping all other infrastructure strictly air-gapped). This is often a suitable concession which allows the managed brokering services to be utilized, but even this may not be an option for some customers and use cases.
• Configuration flexibility - one person's 'complex' is another person's 'flexible', and flexibility has been a strong suite of customer managed Citrix virtualization infrastructure for more than two decades. Over the years the technology has gained a ton of features that support some very niche use cases and third-party integrations. The Citrix Cloud services focus on simplicity and pre-integration. In doing so, some of these niche features and integrations are not available. Therefore, some edge cases are still best served by a customer managed stack. That said, given the rapid pace of innovation coming to the Citrix Cloud services, these edge cases are becoming increasingly rare.
• Control - some organizations, cultures, and business models demand as much control as possible. With customer managed Citrix virtualization components, customers can completely own their destiny. This control comes at a cost (infrastructure, complexity, personnel, and such) but "control at all cost" is a thing for some customers.

To summarize, the lift and shift deployment model uses the following:

In its simplest form, a lift and shift deployment of Citrix virtualization technology onto AWS resembles a traditional customer managed deployment on-premises. It uses a CVAD 'site' deployed into an AWS region and uses basic AWS IaaS services such as EC2 virtual machines and VPC networking at a minimum. As mentioned previously, it requires the customer to build/configure/maintain all system components, plus supporting services such as SQL databases. The following diagram depicts this deployment model:

Diagram 1: Conceptual Architecture, CVAD: Lift and Shift Deployment Model on AWS.

It is worth noting that this deployment model also uses AWS Availability Zones to provide a highly available design. See Availability Zones later in this document for more context.

A lift and shift deployment model is often combined with a hybrid cloud infrastructure model, using AWS Direct Connect, AWS VPN, or similar networking technology to connect a customer managed data center and resources to AWS. Customers can optionally adopt some of AWS' more advanced cloud services (to provide a measure of simplification with the transition), and they may also choose to host some services (such as SQL databases, Citrix licensing, Citrix StoreFront, and Citrix ADC/Gateway) either on AWS, in a customer managed data center, or both depending upon their existing investments, use case requirements, and such. A conceptual architecture of this deployment model (using AWS RDS for SQL Server or on-premises SQL server) is shown in the following diagram. Only one active instance of Citrix Licensing is needed, but we've shown multiple to depict available options:

Diagram 8: Conceptual Architecture, CVAD: Lift and Shift Deployment Model with Hybrid Cloud infrastructure model and AWS managed cloud services.

Lift and Shift - why not

By now you've gathered that the Citrix leading practice/recommendation is to NOT do a full lift and shift. You can be wondering why, or where this is coming from. Referring to our breakdown of Citrix virtualization system components, the session brokering and administration subsystem is the most critical component you want to consider NOT lifting and shifting. We strongly recommend customers consider using Citrix's cloud services for session brokering and administration (deploy Cloud Connectors only, vs. deploying Delivery Controllers + SQL databases + Director servers + Citrix Licensing servers). Here are some of the primary reasons why we take this position (and they might sound familiar):

• Simplicity - While customer-managed session brokering services provide the ultimate in control and configuration flexibility, it comes at the cost of complexity and ongoing maintenance requirements. With Citrix Cloud services, simplicity is a foundational design tenet. When multiple cloud services are used, they come pre-configured to work together, and when configuration is necessary, workflows and options are dramatically simplified.
• Infrastructure and licensing cost savings - customer managed Citrix virtualization services often require extra hardware and software to support them, and these have costs associated with them. One good example is Microsoft SQL Server: customer managed brokering services require databases, and if you're going to build/manage your own, you must provide them.
• Speaking of infrastructure cost savings - this brings up a critical differentiator between the two session brokering options: Autoscaling. Citrix's managed brokering service (DaaS) includes the Citrix Autoscale feature, which provides built-in VDA capacity and cost management functionality. This feature can save customers a substantial amount of money on infrastructure when they're only paying for what they use. When running a Citrix virtualization workload on AWS, this often means the difference between paying for committed use discounts or paying for VM usage as you go. The cost savings can be dramatic for many use cases, and the Citrix Autoscale feature helps ensure you're only consuming what you need. Important note: This feature is only available to Citrix Cloud service (Citrix DaaS) customers - it is not available to customer managed brokering infrastructure (Citrix Virtual Apps and Desktops LTSR or CR releases). - Management savings - With cloud services, Citrix (and AWS in this case) shoulders the responsibility for keeping the services highly available, performant, secure, and up to date. You still build and manage your VDAs regardless, but don't underestimate the value of delegating these responsibilities. Cloud services help free up IT resources, allowing them to focus on providing unique value to their businesses instead of these critical but tedious and often time-consuming tasks.
• "Free" upgrades and continuous innovation - with customer managed infrastructure, the onus is on the customer to upgrade and patch the components in their care. With cloud services, most of those work efforts go away. The service providers (Citrix and AWS in this case) tend to be constantly innovating, and they bring these innovations to the customers who consume the services, often without requiring any work on behalf of the customer.
• Access to more features, functionality, and services - modern service delivery platforms (such as Citrix Cloud and Amazon EC2) give technology providers a powerful, cost-effective way to bring new features, capabilities, and services to market that wouldn't otherwise be possible. Vendors such as Citrix are committed to meeting the customer wherever they're at in their digital transformation journey, but sometimes the only way to cost-effectively deliver new capabilities is to deliver them as a cloud service.

Lift and Shift - more resources

Before Citrix Cloud services were born, customers were already successfully deploying Citrix virtualization technologies on AWS. In those days Citrix called the Virtual Apps and Desktops products XenApp and XenDesktop. Extensive work went into creating and publishing reference architectures and deployment guides for this deployment scenario. A good portion of the detail in these aging resources still applies to deployments who must go down this road today.

For customers who MUST go down this route, the following published resources provide you with useful background detail you can use to help you be successful. We recommend reviewing these materials before you continue on with this document, as we are highlighting important design decisions that have changed since these works were completed:

• Using AWS Directory Service and Amazon RDS with Citrix Virtual Apps and Desktops (blog)
• Deploying Citrix Virtual Apps and Desktop with AWS Directory Service and Amazon RDS Version 1.0 (deployment guide)

Design Decisions

This section explores key design decisions to consider as you're architecting your Citrix virtualization system on AWS. We walk down through each layer of the Citrix Architectural Design Framework, exploring key areas for you to consider.

About the Citrix Architectural Design Framework

Citrix's Virtual Apps and Desktops solution (the product family name collectively referring to Citrix's virtualization technologies) enables organizations to create, control and manage virtual machines, deliver applications and desktops, and implement granular security policies. The Citrix Virtual Apps and Desktops solution provides a unified framework for developing a complete digital workspace offering. This offering enables Citrix users to access applications and desktops independent of their device's operating system and interface.

The Citrix architectural design framework is based on a unified and standardized layer model. It provides a consistent and easily accessible framework for understanding the technical architecture for most of the common Virtual Apps and Desktops deployment scenarios. These layers are depicted in the following conceptual diagram:

Diagram 9: Conceptual Architecture, Citrix Virtual Apps and Desktops.

• User Layer - This layer defines user groups and locations of the Citrix environment.
• Access layer - This layer defines how users access the resources.
• Control layer - This layer defines the components that control the Citrix solution.
• Resource layer - This layer defines provisioning of Citrix workloads and how resources are assigned to the given users.
• Platform layer - This layer defines the physical elements where the hypervisor components and cloud service provider framework run to host the Citrix workloads.
• Operations Layer - This layer defines the tools that support the delivery of the core solutions.

User Layer Considerations

In the Citrix Architectural Design Framework, the User layer describes the user groups, their locations, specific requirements, and more. The user layer appropriately sets the overall direction for each user group's environment. This layer incorporates the assessment criteria for business priorities and user group requirements to define effective strategies for endpoints and Citrix Workspace App. These design decisions affect the flexibility and functionality for each user group.

When designing and deploying a Citrix virtualization system on any platform, the decisions and strategies adopted after careful assessment set the foundation for many other decisions that customers ought to consider as they work their way down through the other layers in the Citrix Architectural Design Framework. As such, this is a critical layer to understand thoroughly and get right.

Access Layer Considerations

In the Citrix Architectural Design Framework, the Access layer defines how users access AWS resources. The design of your access layer is critical to the functionality delivered by any Citrix virtualization system. It controls how users authenticate to the system. It also controls how users view and launch virtualized applications and desktops, plus what type of applications and content are available to them. Also, the Access layer controls how and when sessions are securely proxied or directly connected.

In the context of the Citrix virtualization system components we defined earlier, the access layer contains the following components and choices:

The following table contains critical decision points when determining which access layer component to deploy, but the choice is not binary. Citrix supports various different access methods that can be customized to suit your needs.

UI Service and Authentication Considerations

Consider the following when choosing how you want to provide UI services for your Citrix virtualization system on AWS:

HDX Session Proxy Considerations

Consider the following when choosing how you want to provide HDX session proxy functionality for your Citrix virtualization system on AWS:

Summary, Recommendations, and Leading Practices

Now that we've reviewed some of the attributes/features/capabilities that help drive your customer managed vs. cloud service decisions for the Access Layer subsystems, let's examine the top level decisions in the context of the deployment models we defined earlier.

Access Layer: Greenfield/Cloud Only Deployment

Since the green field or cloud only deployment model use cloud services across the board, the AWS specific implications on the design of your Citrix virtualization system are simple: there aren't any. It's not necessary build or configure anything on AWS since everything required for both UI and HDX proxy services is provided for you, configured and ready to go 'out of the box'.

The Access Layer of a Citrix deployment is a key requirement for delivering virtual apps and desktops to users. If an access point is unreachable or fails, users cannot access their resources. Network design and implementation can be complicated, but with Citrix Gateway Service and Citrix Workspace, redundancy, failover, maintenance, and global presence are all part of the package - with no networking knowledge required.Using the Citrix Gateway Service and Citrix Workspace can reduce your infrastructure footprint substantially. By moving the access layer to a cloud services model, users can securely access network resources from anywhere in the world. This approach requires the least deployment and maintenance efforts, so it is a great option if you want to get up-and-running quickly, have a limited IT staff, or if infrastructure is not your focus. With everything pre-configured, this deployment model is the least customizable, but for deploying a simple, secure, fully functional, globally accessible system, using Citrix Workspace and Gateway Service for your access layer is the way to go.

Access Layer: Hybrid Deployment

With the hybrid deployment model, you're going to be building/managing some of the Citrix virtualization system components, otherwise it is a green field or cloud only deployment by definition. With the hybrid model, you're possibly deploying Citrix ADC/Gateway VPXs on AWS or even on-premises, and depending upon your requirements, you might also be deploying Citrix StoreFront on AWS or on-premises. Customers who have made significant investments in their on-premises Gateway and identity solutions can benefit from the ability to use Citrix Gateway as the identity provider for Workspace.

This deployment model is common for security-focused deployments, deployments with current on-prem infrastructure (ADC or StoreFront), and for DR/failover sites for existing customer managed data centers. One of the key considerations for this model is keeping your users, resources, and access points as close together as possible. Choose AWS regions near the on-prem resource locations in which to deploy your Access Layer. Where possible, keep your ADCs and StoreFront servers as close as possible to each other. This is where things can get tricky. Consider the Citrix Virtual Apps and Desktops launch sequence when designing your hybrid deployment, noting especially that all traffic is routed through the Citrix ADC.

With Citrix ADC/Gateway and StoreFront as EC2-based instances in AWS, there is also much more potential for customization. In addition to the multiple StoreFront stores, multifactor authentication, and various industry-leading ADC features, hybrid deployments can also use native AWS services such as the Relational Database Service (RDS) and AWS Directory Services. Hybrid deployments lend well to a more gradual cloud transition and leave room for adjustments to the architecture along the way, as opposed to lift, and shift methods.

The hybrid approach does require a higher level of expertise and increased lead time to deploy than the greenfield/cloud only model, but can serve as a solid transition state between a traditional customer managed/on-prem deployment and cloud only state.

Access Layer: Lift and Shift Deployment

With the legacy lift and shift deployment model, you're deploying both Citrix ADC/Gateway VPXs and Citrix StoreFront on AWS, or potentially reusing existing on-premises deployments of these technologies for the same purpose. This type of deployment tends to have the least lead time for customers with existing on-prem Citrix virtualization environments, and is also the easiest transition from an Operations and Maintenance perspective. Staff with experience managing an on-prem environment has a shorter ramp-up time with the lift and shift deployment model, as the Citrix infrastructure remains largely unchanged. For the access layer specifically, this method is straightforward and allows for many customizations. The lift and shift is a great first step for existing deployments going into the cloud or for new or air-gapped AWS regions, but may be a hindrance to adopting a cloud-forward architecture in the future.

Citrix ADC/Gateway VPX on AWS

Deploying the Citrix ADC/Gateway on AWS is different than deploying it on-premises, though in the end you're managing them yourself. Fortunately deploying Citrix ADC/Gateway on AWS is thoroughly documented, so we recommend reviewing the following resources before you solidify your design and begin implementation:

• Citrix ADC VPX on AWS in Citrix Docs: Provides a comprehensive overview of Citrix ADC on AWS, including supported VPX models, AWS regions, EC2 instance types, and extra resource references.
• Citrix ADC and Amazon Web Services Validated Reference Design in Citrix Docs/Advanced Concepts - includes more details and deployment guidance.

While there are potential variants for a Citrix ADC/Gateway VPX architecture on AWS, the following diagram (from the Citrix ADC for Web Applications Quick Start Deployment Guide) depicts a multi-AZ Citrix HA pair deployment as deployed by the Quick Start template (with default subnets/CIDR blocks):

Diagram 10: Conceptual Architecture, Citrix ADC/Gateway VPX on AWS with HA across Availability Zones.

As discussed in Citrix ADC VPX on AWS on Citrix Docs, there are two primary deployment options available. They are:

• Standalone: Individual instances of Citrix ADC/Gateway can be deployed and managed as separate entities. This is commonly used for smaller scale or POC deployments where high availability is not a requirement.

• High Availability: This is the most commonly deployed model for production environments: pairs of Citrix ADC/Gateway VPX instances can be deployed using native Citrix HA mode on AWS. With older firmware versions, the pair is deployed in the same AWS Availability Zone. Starting with Citrix ADC 12.1 firmware, highly available pairs of VPX appliances can be deployed across Availability Zones (AZ). How high availability on AWS works explains the difference between deploying a pair of ADCs within the same AZ and across AZs. We dig into this option more deeply later in this section.

While Citrix ADC VPX generally supports single, dual, or multiple NIC deployment types, Citrix recommends using at least three subnets for each ADC when deployed on AWS, with a network interface in each subnet for optimum throughput and data separation. When deployed to support Citrix Virtual Apps and Desktops, the NSIP is typically attached to the "Private Citrix Infrastructure Subnet," the SNIP is attached to the "Private Citrix VDA Subnet," and the Citrix Gateway VIP to the "Public Subnet." The following simplified conceptual diagram depicts this configuration. It shows a single VPX instance in a single AZ - this design pattern would be duplicated (likely in a second AZ) for a High Availability configuration:

Diagram 11: Citrix ADC VPX instance interface mapping for CVAD/DaaS deployments.

ADC High Availability across Availability Zones

As mentioned earlier, this is the most common deployment model for Citrix virtualization systems. This model uses a pair of Citrix ADC VPXs deployed across Availability Zones by either using Citrix ADC's native HA (active/passive) feature or a combination of Citrix ADC's native Global Service Load Balancing (GSLB) and IPSet features. The latter option (which became feasible in early 2020) allows for an active/active configuration across AZs, and functions by allowing the ADC to act as an authoritative DNS source. This new option/architecture is expected to be popular for public cloud deployments, so we focus on that here.

The Domain Based Services for cloud load balancers allow for AutoDiscovery of dynamic cloud services. By deploying Citrix ADCs across multiple AZs in an active-active configuration, you can use cloud resources in different AZs to optimize High Availability/Disaster Recovery. Each AZ can contain cloud resources in the familiar Pod Infrastructure, to allow for easily managed updates, patching, and scalability for expansion. For detailed information about setting up GSLB between AWS AZs, see Citrix Documentation.

Diagram 12: Traffic flow before and after HA failover in multi-AZ HA deployment.

In the preceding diagram, we can see that each ADC has a different Gateway virtual IP (VIP). This is characteristic of an Independent Network Configuration (INC). When VPXs in an HA pair reside in different Availability Zones, the secondary ADC must have an INC, as they cannot share mapped IP addresses, virtual LANs, or network routes. The NSIP is different for each ADC in this configuration, while SNIPs and Load Balancing VIPs utilize a special Citrix ADC feature called IPset, or Multi-IP virtual servers, which can be used for clients in different subnets to connect to the same set of servers. With IPset, you can associate a private IP to each of the primary and secondary instances. A public IP can then be mapped to the primary ADC in the pair. In the case of failover, the public IP mapping changes dynamically to the new primary. For GSLB deployments in AWS, the service IP can be part of the IPset for both IPv4 and IPv6 traffic.

For more information on adding a remote node to an ADC to create an INC-based HA pair, see Citrix docs.

Citrix StoreFront on AWS

Deploying Citrix StoreFront on AWS is not much different than deploying it on-premises, and in the end, you're also managing all the components of StoreFront yourself too. See Plan your StoreFront deployment for general considerations which apply to all deployments including StoreFront on AWS. The main difference is that you typically deploy multiple StoreFront instances in a StoreFront server group across multiple AWS availability zones. It is important to note that the features enabled with this design are dependent upon latency between AZs. Per Plan your StoreFront Deployment/Scalability, StoreFront server group deployments are only supported where links between servers in a server group have latency of less than 40 ms (with subscriptions disabled) or less than 3 ms (with subscriptions enabled). Make sure you measure latencies between instances in all AZs you plan to host StoreFront and enable/disable subscriptions accordingly.

We already called this out in the UI Service and Authentication Considerations table earlier in this document, but it is worth calling out again: for Citrix DaaS environments with extensive resiliency requirements, Citrix strongly recommends a StoreFront implementation to fully benefit from the Local Host Cache feature (available in both CVAD and DaaS session brokering infrastructure types). For CVAD, this provides resiliency if there is a database outage. For DaaS, this architecture provides resiliency in case Cloud Connectors cannot reach Citrix Cloud. In either case, disconnected users will still be able to connect to new and existing sessions during an outage scenario. For more details, limitations, and implications of Local Host cache activation, see Local Host Cache (DaaS) and Local Host Cache (CVAD).

While we're on the topic of resilience, Citrix also recommends that your StoreFront implementation span multiple AZs (if the AWS region includes multiple AZs), but remember to take the ADC design into account. Citrix ADC is often used in front of StoreFront instances to provide load balancing and extra service resiliency.

By utilizing Citrix Zones, StoreFront redundancy can be built in by spreading satellite zones across two or more AZs in a VPC with a single site. Using Zones is a great way to have resources as close to the users as possible and highly available. Satellite Zones contain StoreFront servers, Delivery Controllers, and app/desktop resources, leaving the Primary Zone with the full infrastructure setup, including the license server and SQL. This allows for scalability of StoreFront web UI and Zone creation/destruction can be orchestrated. Keeping the Zones smaller will allow for optimal east-west scalability and reduce the impact in the case of an outage.

StoreFront on AWS is fully customizable, including Featured App Groups, splash page, coloring and logo, and apps and desktops can be arranged in the best way for your specific needs. StoreFront on AWS also requires knowledgeable administration and engineering to be kept up, but can provide a powerful web UI, especially when integrated with the Citrix ADC.

Resource Layer Considerations

The Resource Layer design focuses on personalization, applications, and image design. The Resource Layer is where users interact with desktops and applications. When deploying a Citrix virtualization system on AWS, the key things to keep in mind (aside from all the 'normal' stuff we won't cover here) are:

• CIFS storage and data replication - Regardless of the tooling you use for managing user personalization settings (the users' Windows profile and redirected folders) you've got to have Windows file shares to store them on. If you've got VDAs in multiple regions (and users can access apps/desktops in more than one) then you've also got to deal with data replication. Many applications also use Windows file shares, so CIFS storage and data replication are important for these too.
• Image design - Citrix App Layering and Citrix Provisioning Services (PVS) do not currently support Amazon EC2 - customers hosting a resource location in AWS use Machine Creation Services for VDA fleet creation, management, and updating.

CIFS Storage and Data Replication

Most Citrix virtualization systems on AWS require at least basic access to a Windows compatible file share to persist user settings, user data, and application data. When these shares are not available, the user experience and application functionality suffer, so it is important to ensure that whatever solution you choose to provide Windows compatible file shares is highly available and data is regularly backed up.

For multi-site deployments, reliable and performant data replication may also be necessary to meet availability, RPO, and RTO needs. This is especially true for environments where users may connect to desktops/apps in 2 or more regions, and application data/user settings must be available in the region where the apps/desktops run. The following section describes some solutions to consider for providing CIFS storage and data replication services on AWS.

While non-Windows solutions for providing Windows file shares exist, most of these solutions cannot deliver the indexing capabilities required for search functionality inside a Windows desktop or applications such as Microsoft Outlook running on Windows. As such, most customers turn to Windows-based file server solutions, at least for storing user profiles and persistent application data. Fortunately, both customer managed and cloud service options are available for use when Citrix virtualization systems are run on AWS.

Customer Managed: Windows File Servers on Amazon EC2

The first solution many customers consider for providing Windows compatible file services on AWS is building their own Windows file servers on EC2 to serve each resource location on AWS. Since Windows file servers are needed by various different types of applications and workloads, many IT shops may gravitate towards building and managing their own since this is something they know how to do. At the most basic level, the customer spins up one or more Windows EC2 instances, attach extra Amazon Elastic Block Store (EBS) volume, join the instance's to their Active Directory, and get busy configuring and setting up Windows File Services.

This option, as you might imagine, provides customers with the most control and flexibility. While this is very appealing to certain types of customers and certain verticals, it also comes at a cost: the responsibility to size, scale, build, manage, patch, secure, and maintain everything from the Windows OS up. Customers electing to go this route ought to also ensure these file servers are highly available. This is often accomplished using file servers in multiple Availability Zones, and using Windows DFS-N/DFS-R, though it's easy to end up in an unsupported configuration (per Microsoft) if you're not careful.

Note: Customers considering this option ought to review Microsoft's support statement regarding using DFS-R and DFS-N for roaming profile shares and folder redirection shares. One more point to consider since the Citrix virtualization system will be running on AWS: a new deployment or migration event may provide an excellent opportunity to evaluate using a cloud service for Windows file services instead of building your own. Fortunately, Amazon has some cloud service options worth considering. We touch on some of these now.

Cloud Service: Amazon FSx for Windows File Server

Amazon's FSx for Windows File Server is a cloud service which customers can consume on AWS. FSx for Windows File Server provides a fully managed, native Windows file system, and SSD-based storage with consistent submillisecond performance. Since FSx is built on Windows Server, it delivers a fully native, Windows compatible file system that provides storage and protection for Citrix virtualization systems on AWS. FSx for Windows File Server is also Citrix Ready Verified, meaning this AWS supported solution has been validated by Citrix to be compatible with Citrix Virtual Apps and Desktops. While it is not officially supported by Citrix, the service IS fundamentally native Microsoft Windows file server - it is just managed by AWS instead of the customer. For more information, see Amazon FSX for Windows File Server on Citrix Ready.

For IT teams, this is an excellent option that removes many of the more mundane or low-value tasks around deploying and managing storage. Most importantly, using FSx offloads security, data protection/backup, compliance, software updating/patching tasks, and the monitoring of storage infrastructure to make sure it meets required service levels. IT teams can treat the entire FSx file service as a single operational platform instead of managing a Windows operating system file server, storage, networking, and such. Also, FSx supports all the common management tools it already uses, such as Active Directory (AD) integration, Windows DFS Namespaces, DFS Replication, and others.

Each FSx managed file system you create essentially becomes a highly available and durable file server in a specific Availability Zone. For servicing a Citrix virtualization system, customers ought to ensure these "file systems" are highly available. This can be accomplished by provisioning FSx managed file systems in multiple availability zones, and using Windows DFS-N/DFS-R to create highly available Windows file shares, though it's easy to end up in an unsupported configuration (per Microsoft) if you're not careful.

Note: Since FSx is a Windows file server, customers considering this option ought to review Microsoft's support statement regarding using DFS-R and DFS-N for roaming profile shares and folder redirection shares.

More Cloud Service Options

Besides Amazon's first party managed Windows file service, AWS supports many more expansive and feature rich options, some of which integrate with traditional on-premises storage technologies. While these other options are outside the scope of this document, there are many options to choose from. A good place to start exploring options is on the AWS Marketplace. These types of solutions can be especially relevant for more complex, multi-region use cases where reliable, and resilient data replication is needed.

CIFS Storage and Data Replication: Summary and Conclusions

Customers can manage their own highly available DFS file share, benefit from this as an AWS service (FSx) to save on management effort, or use third party storage appliance solutions to extend on an on-premises environment. Citrix recommends that customers analyze the pros and cons of each to determine a solution that is right for them.

Image Design and Management

In a Citrix virtualization system on AWS, applications and desktops are delivered via EC2 instances called "VDAs" (named after Citrix's Virtual Delivery Agent software, which is installed into Windows or Linux instances containing the applications being delivered by the Citrix virtualization system). A group of identical VDAs are provisioned and maintained in "Machine Catalogs," a management construct defined and maintained through the session brokering and management subsystem (both DaaS and CVAD). The creation, sizing, and management of these instances is key, as many systems have large numbers of VDAs and the software stack in a VDA changes frequently as hotfixes, service packs, and software updates are applied. We discuss some of the higher-level considerations in this section.

VDA Provisioning and Image Management

On AWS EC2, Citrix virtualization systems use Citrix's Machine Creation Services (MCS) provisioning technology for VDA deployment and image management. MCS utilizes an IAM service account on EC2 to orchestrate the mastering process (turning a snapshot of a template VM's system disk into a generalized AMI), the cloning process (creating and managing a fleet of VDA instances based on the AMI created from the snapshot of the template VM), autoscaling Delivery Groups, updating deployed images, and more. We discuss MCS on AWS in much more detail in the Control Layer Considerations sections of this document.

Note: customers already using MCS for their on-premises environments can notice some differences between the options available to them when provisioning machines in AWS. MCS managed VDA instances on EC2 have two disks attached: the system disk (a read/write copy of the template image AMI created during the mastering process) and a 1GB personality disk. Depending on the machine catalog type and hosting connection options configured, the system disk (and sometimes the VM instance) will be deleted at shutdown and recreated at 'power on' (for pooled or shared catalogs) or they are retained (for persistent catalog types). See CTX234562 for more information.

Delivery and Persistence Models

Choosing the right delivery models is critical and has broad implications beyond just cost. Citrix virtualization technology supports three main delivery models, which can be mixed and matched and used in combination to support many different use cases. The three delivery models are:

• Hosted shared: The hosted shared model most commonly utilizes a Windows Server OS with the RDSH role installed, though Linux instances can provide the same functionality for compatible apps. With this model, a single VDA instance can support multiple simultaneous users, each running either a full desktop or connecting to one or more published applications. When using Windows Server OS/RDSH with the Desktop Experience and related components installed, desktops and apps look and feel like they're running on a Windows desktop OS. Since every user on a given instance shares OS instance, administrators typically pre-install and configure the mix of applications on hosted shared instances, and users do not have local administrator rights to the OS. Hosted shared instances can also run on shared infrastructure, and can be consumed using both on-demand and reserved instance pricing models. Administrators usually deploy a fleet of instances to support the hosted shared model, and both customer managed and cloud service types of Citrix brokering subsystems provide sophisticated load balancing capabilities to ensure every user experiences adequate performance. Hosted shared instances can also use GPU backed instance types on AWS to increase performance for graphically intensive workloads that can benefit from a GPU, though the GPU vendor can require extra licenses. Both Windows Server OS and RDS CAL licenses can be 'rented' under Microsoft's SPLA licensing model, though customers can avoid these additional costs by using Linux as the OS. This model is, hands down, the most cost effective to run on AWS.
• Server VDI: The "Server VDI" (Virtual Desktop Infrastructure) model also uses a Windows Server OS, and with the Desktop Experience and related components installed, it looks and feels to the user just like a Windows Desktop OS. The RDSH role is not installed with this model, so one instance supports one user at a time, and users are sometimes provided with elevated rights to the Server OS so they can install their own applications. Like hosted shared instances, server VDI instances can also run on shared infrastructure, can be consumed using both on-demand and reserved instance pricing models, can use GPU backed instance types, and Microsoft OS and RDS CALs can be 'rented' under Microsoft's SPLA licensing model. Given the tools available today, 99+% of Windows applications can be installed and ran on the Windows Server OS, and though sometimes software vendors don't explicitly support their applications on Windows Server, most Windows apps run as well on Windows Server as they do on a Windows Desktop OS. It's also worth noting that server VDI instances can also use GPU backed instance types on AWS to increase performance for graphically intensive workloads that can benefit from a GPU, though the GPU vendor may require another licenses. This is the second most cost-effective delivery model to run on AWS.
• Client VDI: The client VDI delivery model typically uses a Windows desktop OS such as Windows 10 or Windows 7, although a supported Linux OS version can be used as well. Client VDI is a 1:1 model, meaning each unique user requires their own OS instance. Customers who are new to Citrix virtualization technology often come into these types of projects asking for client VDI, even though more cost-effective models are available. Their vernacular can also have been influenced by other virtualization vendors whose technology stacks don't support hosted shared or server VDI deployment models. The client VDI model, while looking 'simpler' on the surface, gets much more complicated the deeper you get into it, though most of the complexity can be avoided by using Linux as the OS. Most of this complication is driven by Microsoft's licensing requirements for the Windows Desktop OS which, unlike Windows Server, is not available via Microsoft's SPLA licensing program. As such, customers must bring their own licensing for these products. Also - Windows desktop based client VDI instances cannot run on shared infrastructure. This means that client VDI instances must run in either AWS dedicated instances or on AWS dedicated hosts. This substantially increases the cost and complexity of managing the infrastructure required, reduces flexibility and cost control options, and gets expensive quickly. As you might expect, client VDI instances can also use GPU backed instance types on AWS to increase performance for graphically intensive workloads that can benefit from a GPU, though the GPU vendor can require more licenses. Client VDI is the most expensive delivery model to run on AWS. For both VDI models, another important consideration is the persistence model. VDI instances can be randomly assigned to users with no persistence (pooled) or users can have assigned machines that persist and are personalized (dedicated). Pooled instances can be easier to manage over time since all instances in a given pool are identical. Citrix's MCS can update the system disks attached to pooled instances with a few clicks, and capacity/cost management is more effective since an idle pool of instances can serve many users. Pooled instances are a bit less flexible than dedicated since end-user changes to pooled instances don't usually persist between reboots, though technologies such as Citrix App Layering's User Layer or Personalization Layer released in CVAD 1912 can be used to minimize the impact on the user experience. Dedicated instances can also be tougher to manage from a cost perspective too - since it is often tough to predict when a user will log on, the user must either wait while their instance is started, or administrators must keep them running during time windows where each user is expected to log on.

While we've mentioned it previously, we're going to mention it again here for clarity: various flavors of Linux can be used in a Citrix virtualization system, as long as one or more applications run on Linux. Citrix's virtualization technology supports both hosted shared and VDI delivery models, persistent and pooled models, and GPU backed instance types. The user and administrator experiences are different than with Windows based instances, but Linux based VDAs are often much less expensive to run since they don't require Microsoft licenses.

Finally, let's revisit the consideration of GPU acceleration. All three delivery models (for both Linux and Windows) can use NVIDIA accelerated GPU instances on AWS. G-series instances can be used for graphics accelerated use cases, but are not yet commercially viable for general purpose usage. Note that Citrix doesn't support the AWS Elastic GPU today, but since Elastic GPU only works for OpenGL, its impact on typical graphics workloads in the enterprise is minimal.

So - which delivery models do you use? It is worth noting that you can mix and match delivery models in the same system to meet the needs of different user groups or use cases. The most cost-effective delivery model from an infrastructure perspective is hosted-shared. The combination of server OS with multi-user concurrency is highly efficient, and the number of users per virtual machine can be sized accordingly depending on user type (for example - task worker vs. knowledge worker vs. power user) to ensure a great experience. For situations where users and apps require extra capabilities that aren't satisfied by hosted-shared, VDI is the way to go. Server VDI ought to be evaluated first: it is substantially more cost-effective to run than Windows 10 VDI for Windows workloads, and Server VDI can deliver a desktop that looks and feels similar to Windows 10. Also, Server VDI doesn't have the Microsoft EULA requirement to use dedicated instances/hosts - Client VDI (deploying Windows 10 or sometimes Windows 7) does. For Windows based workloads on AWS, Client VDI ought to be considered as a last resort, and deployed only when hosted shared and Server VDI delivery models are not possible.

To help with the decision making process, the following decision tree compares

. The tree doesn't explicitly differentiate between client VDI and server VDI models. When a use case suggests VDI is the appropriate delivery model for your workload, Server VDI ought to be considered wherever possible for running on AWS as it is substantially more cost effective and easier to manage.

AWS Instance Billing Model

Once you've decided which delivery model to use (hosted-shared, server VDI, or client VDI), the next step is to plan for an hourly on-demand billing model or a reserved billing model. Ideally, as many VDAs as possible are to be paid for by the hour with the on-demand billing model, and use the Citrix Autoscale feature to control costs. By using Citrix Autoscale (a feature exclusive to the DaaS cloud service brokering subsystem) VMs are powered on as needed with anticipation for peak hours. During off peak hours, however, VMs are shut down, so it's important to consolidate loads with the hosted-shared model and for all models ensure that users save their work and ideally log off gracefully from their sessions. Reserved instance capacity can be used for infrastructure components like the Cloud Connectors (which remain on 24/7) and a predetermined number of VDAs that will always remain on (for example, 10% of peak). Besides providing significant discount compared to On-Demand pricing, Reserve Instances also provide a capacity reservation when used in a specific Availability Zone.

VDA Instance Sizing and Cost Management

When running a fleet of VDAs on AWS, choosing the right instance type for your different workloads (VDAs) is a key decision, with substantial performance, manageability, and cost considerations. Choose too small of an instance and performance can suffer. Choose too large of an instance, and you're paying for resources you're not using. Choosing the right instance type ends up being a balancing act, and often requires fine-tuning for each specific workload.

Which AWS EC2 instance type to choose for your VDAs depends heavily upon the specific workload and delivery type. However, as a general guideline, "M" series instances are often most suitable for hosted-shared whereas "T" series instances are suitable for VDI. "M" series has balanced CPU and RAM designed for the mostly predictable resource consumption across multiple sessions on a host. "T" series are "burstable" in nature designed for the mostly unpredictable characteristics of VDI (for example - one minute a user is idling and the next they are running a macro calculation). For extra details on instance type selection and pricing, readers can refer to the Citrix on AWS cost estimation presentation (in sources section).

For more information regarding instance selection (especially as it applies to the hosted shared delivery model) sees Citrix Scalability in a Cloud World – 2018 Edition. This article, while slightly dated, discusses leading practices regarding instance selection based on performance, manageability, cost, reserved vs. on demand pricing models, and LoginVSI scalability testing. These concepts and considerations are still valid today, even though instance choices and pricing have likely changed since its initial publication.

Note: Some newer AWS instance types will not show up by default in the Machine Catalog creation wizard in Studio (either CVAD or DaaS). The UI is populated with instance types from a static XML file which resides on Delivery Controllers (CVAD) or Cloud Connectors (DaaS). This XML can be modified to include newer instance types, but this file is overwritten with default values during upgrades (both Citrix initiated Cloud Connector updates or customer-initiated Delivery Controller upgrades). See CTX139707 for more details on how to update the list of available AWS instance types. During this round of testing (a point in time reference) the M5.2Xlarge instance type (8vCPU, 32 GB RAM) turned out to be the winner in terms of $/user/hour (with an industry standard sample workload). Your numbers - given your specific workload characteristics and available AWS pricing - can vary, but the process and tooling can be used to approximate your monthly IaaS costing more accurately. Regardless of how you determine the instance types you start with, it is important to monitor usage over time and adjust as needed to keep the balance between resource availability, consumption, and cost. Customers ought to consider using services such as Citrix Analytics for Performance - the information such services provide can play a key role in keeping performance up and costs down.

Application Design

An extra consideration includes application design. As customers plan to migrate workloads to a cloud platform such as AWS, they must ensure that app performance is not impacted. A rule of thumb which has applied for over 20 years is that the data ought to reside as near as possible to the workload. This means more complex applications architectures ought to respect this rule. An example of this includes apps with a front-end and back-end (database). To avoid adding latency which will impact application performance, both the front-end and back-end are to be migrated. An alternative would be a hybrid approach using a mix of on-premises (for complex apps) and cloud hosted workloads (for simple applications). It is important to always consult with application vendors for compatibility. The linked Tech Zone decision matrix compares the different

, which include Hosted Shared Applications (single and multi-use) and Hosted Shared Desktops. The workload segmentation decision-making process these article outlines can be used as a guide for the workload design process.

One final word on application design, the Enterprise Layer Manager appliance does not currently run on AWS, and does not currently support exporting layered images into an immediately consumable disk format for use on AWS. If App Layering support on AWS is critical for your migration or deployment, email aws@citrix.com with information about your project. You’ll be added to the list to be an early adopter candidate for future releases, and your voice will be heard. For more information on Citrix App Layering, refer to the product documentation and App Layering reference architecture.

Control Layer Considerations

In the Citrix Architectural Design Framework, the Control layer defines the components that control the Citrix solution. This includes components like Active Directory (forest/domain, OU, and user group structure, group policies, and such), Microsoft SQL database usage, Citrix licensing, session brokering and administration, load management, and VDA provisioning/image management. As with previous sections of this document, here we focus on the considerations which are most important for Citrix virtualization systems on AWS, and provide links to existing documentation/guidance on others.

One of the most impactful decisions you are making for control layer components is the session brokering and administration choice. This decision is critical, with substantial implications on cost, complexity, availability, and ongoing maintenance efforts. We start by reviewing the deployment models we introduced earlier in this document, then dig more deeply into the AWS specific considerations.

Control Layer: Greenfield/Cloud Only Deployment

The green field or cloud only deployment model uses cloud services across the board. n). The AWS specific implications on the design of your Citrix virtualization system are minimal, but we walk you through them anyway. Since Citrix Cloud provides most of the infrastructure and administrative components as a service, you won't have to worry about SQL databases, Citrix License Servers, Citrix Director servers and more.

Control Layer: Hybrid Deployment

Remember that with the hybrid deployment model, you're going to be building/managing some of the Citrix virtualization system components, otherwise it is a green field or cloud only deployment by definition. The interesting thing here is that, in the context of the Control layer, they're almost identical.

Control Layer: Lift and Shift Deployment

With the legacy lift and shift deployment model, you're deploying all key control layer components (including Active Directory and all Citrix session brokering/management components) on AWS. If you have to go down the "lift and shift" path, it is both a blessing and a curse. It is a blessing in that most of these considerations have been thoroughly documented in various published works that are already available. It is a curse in that you'll have a lot more work to do both up front and over time to build, manage, secure, and maintain these components.

If you're a "lift and shift"er, you want to review and reference the following before you continue: collectively, they cover most of the design decisions you must consider to be successful deploying Citrix on AWS using the lift and shift deployment model:

• Using AWS Directory Service and Amazon RDS with Citrix Virtual Apps and Desktops (blog)
• Deploying Citrix Virtual Apps and Desktop with AWS Directory Service and Amazon RDS – Version 1.0 (deployment guide)

Active Directory Considerations

All deployment models for Citrix virtualization systems on AWS require Microsoft Active Directory. For a compelling user experience, functional Active Directory services must be available in every AWS region where you've got VDAs deployed. The structure and complexity of your Active Directory implementation must be carefully considered, but fortunately Citrix virtualization can flexibly integrate with various different AD designs and servicing models.

When deploying Active Directory on AWS, customers can build/maintain their own Active Directory Domain Controllers using Windows Server instances, use AWS Directory Service for Microsoft Active Directory, or a combination of the two. Active Directory trusts can also be used to connect two or more AD forests/domains depending upon the customer's needs.

For customers looking to minimize the administrative overhead required to build and maintain functional Active Directory services, the AWS Directory Service for Microsoft Active Directory (also known as AWS Managed Microsoft AD) is an option worth considering. This service provides you with a fully functional Active Directory forest/domain without the overhead of building and maintaining Windows Server VM instances. AWS Managed Microsoft AD is built on highly available, AWS-managed infrastructure. Each directory is deployed across multiple Availability Zones, and monitoring automatically detects and replaces domain controllers that fail. In addition, data replication and automated daily snapshots are configured for you. You do not have to install software, and AWS handles all patching and software updates. With AWS Managed Microsoft AD, you can use native Microsoft administrative tools, manage Windows machines and users with Microsoft Group Policy, join EC2 instances and AWS RDS for SQL Server instances to it, and even setup Active Directory trusts with existing AD instances to support various complex Enterprise scenarios.

Customers who choose to use the AWS Managed Microsoft AD service with Citrix virtualization technologies can expect these technologies to work with this AWS service, though there are a few important considerations to consider before doing so. For starters - you won't have Domain Administrator, Enterprise Administrator, or other 'super user' type access to the AD instance. You do, however, have full control of your own container at the root of the directory where you can create users, computers, groups, OU's, and group policies.

A few other things you CAN NOT do:

• Create AD objects in any of the default containers (such as /Computers): they're read-only. This brings up a common mistake some customers make when using Citrix's MCS provisioning technology: you must choose to create the machine accounts for your MCS managed VDAs in a container/OU that's writeable - if you don't choose such a location, MCS won't be able to create the machine accounts.
• Install and configure some AD integrated features such as Certificate Services. As such, this impacts customers who will be using Citrix's Federated Authentication Services ("FAS") technology (which requires AD integrated Certificate Services): these customers must build and manage their own Active Directory on AWS using EC2 Windows Server instances.
• Have local Server Administrator equivalence by default. In an 'out of the box' Active Directory installation, the Domain Administrators group is added to the local Server Administrators group by default. If you're using the AWS Managed Microsoft AD service, you must create your own server administrators' group, add your own users to it, create and apply a group policy to add your group to the built-in Server Administrators group on member servers/workstations.

While trust relationships, site/service configuration, replication, and other AD related topics will not be covered in this paper, Citrix has provided extensive documentation on these topics applicable to all three deployment models.

Note: AWS Directory Service for Microsoft Active Directory is a "Citrix Ready Verified" offering. While not officially supported by Citrix, the service IS fundamentally native Microsoft Active Directory - it is just managed by AWS instead of the customer. This AWS service does have some limitations imposed upon it to deliver it as a service at scale, and the currently known/most impactful limitations for a Citrix environment are listed here. For more information on Active Directory requirements for green field and hybrid deployments (environments using Citrix Cloud and the CVAD Service for session brokering and administration) see Citrix Cloud Connector Technical Details. Besides covering supported Active Directory functional levels, this article also covers deployments scenarios for Cloud Connectors in Active Directory.

For more information on Active Directory requirements for lift and shift deployments (environments using customer managed session brokering and administration via Citrix Virtual Apps and Desktops LTSR or CR versions) see CVAD System Requirements, Active Directory Functional Levels.

Session Brokering and Administration Considerations

As you've probably already gathered by now, the choice of how you provide session brokering and administration services is critical, and has broad reaching implications on overall cost, manageability, maintenance, and available capabilities for your Citrix virtualization system. As we've already discussed, Citrix recommends the use of the Citrix Cloud service (DaaS) for this critical component, but for certain requirements and scenarios, deploying a customer managed session brokering and administration subsystem (via CVAD LTSR or CR releases) can be necessary or recommended. The following table highlights some of these requirements and scenarios for your consideration:

Cloud Connectors, Delivery Controllers, and Resource Locations

Since both green field and hybrid models use cloud services (DaaS) for session brokering and administration, you deploy Cloud Connectors to create a resource location in each region where you plan to host VDAs. When you create a resource location in a region, you build a highly available configuration by deploying n+1 Cloud Connector instances and spreading the Cloud Connectors across Availability Zones in that region. Cloud Connectors are typically placed in separate private subnets from the VDAs to simplify security policy application, and the Cloud Connector instances must have outbound Internet access to facilitate connecting to Citrix Cloud. Placing them in a separate subnet from VDAs allows administrators to apply different routing policies to the two different resource types.

Diagram 13: Citrix DaaS Resource Location design pattern with separate subnets for VDAs and Cloud Connectors.

The same general concepts apply when we're talking about Delivery Controllers (CVAD), though we use the term zone vs. resource location in the customer managed brokering subsystem. Also note that Cloud Connector instances on EC2 are great candidates for reserved pricing since they are running anytime the system needs to be up. See this article for more information about sizing Cloud Connector instances.

Citrix DaaS Site Design Considerations

Resource Locations and Zones

Using Citrix zones (not to be confused with Availability Zones) can help users in remote regions connect to resources without necessarily forcing their connections to traverse large segments of the WAN. In a Citrix DaaS environment, each resource location is considered a zone. When you create a resource location and install a Cloud Connector, a zone is automatically created for you. Each zone can have a different set of resources, based on your unique needs and environment. For more information on zones, see the following link.

Machine Catalogs, Delivery Groups, and Resource Locations

Citrix administrators ought to ensure that VDAs are also spread across Availability Zones (AZ). An AWS Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region - a physical location around the world where AWS cluster data centers. A virtual private cloud (VPC) is a virtual network which spans Availability Zones in the Region. Subnets are a required subcomponent of a VPC, and virtual network interfaces are each attached to a single subnet. Each subnet must reside entirely within one Availability Zone and cannot span zones. By launching VDAs in separate Availability Zones, you can protect your applications from the failure of a single location. See What is an Amazon VPC? for more information. To ensure VDAs are spread between AZs you can create a Machine Catalog per AZ (using one Host Connection per AZ) which then can map to a single Delivery Group.

Provisioning in AWS: Machine Creation Services

Starting with the release of CVAD 1811, role-based authentication can be used when creating a host connection for MCS provisioning in AWS. An IAM role or IAM user account associated with a Delivery Controller or Cloud Connector on an EC2 instance can be used in the place of a user's secret key and API key, enabling increased security, delegated administrative rights, and PKI-based environments with temporary credentials and session tokens. To configure a host connection using role-based authentication, first create an IAM role with the permissions described in CTX140429. Associate this role with an EC2 instance with a CVAD 1811+ Delivery Controller or a Cloud Connector. On versions of CVAD earlier than 1811, admins must provide the API Key (Access Key) and Secret Key of an IAM user to create a host connection.

After creating the host connection, create a machine catalog as described here using an AMI created from the master VDA image in AWS. For more details about MCS in AWS, see the following articles: Citrix MCS on AWS Deep Dive 1 and How MCS works after pooled VMs are created in AWS.

Another item that ought to be considered when deploying VDAs in AWS using MCS is EBS Volume initialization (also known as pre-warming or hydration). For volumes that were restored from snapshots, the storage blocks must be pulled down from Amazon S3 and written to the volume before you can access them. This preliminary action takes time and can cause a significant increase in the latency of I/O operations the first time each block is accessed. Volume performance is achieved after all blocks have been downloaded and written to the volume. See Initializing Amazon EBS Volumes on Windows for AWS recommended steps to Initialize Amazon EBS Volumes on Windows instances and see Initializing Amazon EBS Volumes on Linux for Linux instances.

See Infrastructure (or Platform) Layer Considerations for details on VPC design as it relates to MCS.

Troubleshooting Machine Creation Services

This section lists some common issues and associated recommendations/resolution links.

• Some newer AWS instance types will not show up by default in the Machine Catalog creation wizard in Studio (either CVAD or DaaS). The UI is populated with instance types from a static XML file which resides on Delivery Controllers (CVAD) or Cloud Connectors (DaaS). This XML can be modified to include newer instance types, but this file is overwritten with default values during upgrades (both Citrix initiated Cloud Connector updates or customer-initiated Delivery Controller upgrades). See Updating AWS Instance Types for XenDesktop for more details on how to update the list of available AWS instance types.
• MCS Provisioning fails on AWS EC2 when using Dedicated instances
• Unable to create AWS hosting connection on Citrix Cloud DDC when proxy is configured on connector server
• When Creating Machines with MCS and AWS an Error "XDDS:2367399e" Occurs
• Configure the Volume Worker instance to use the Machine Catalog VPC and not the default VPC
• How to Ensure Region Compatibility When Using XenApp and XenDesktop MCS in AWS
• Troubleshooting tips from the field for Machine Creation Services in AWS

Infrastructure (or Platform) Layer Considerations

In the Citrix Architectural Design Framework, the Infrastructure (or Platform) layer defines the physical elements where the Citrix workloads run. In this document, that of course refers to AWS. AWS provides many cloud services (165+) and is both the oldest and largest of the Hyperscale Cloud providers in existence today. It was also the first public cloud supported by Citrix virtualization technology, and is a compelling option for new or existing Citrix customers looking to move existing or run new Citrix virtualization workloads in 'the Cloud'.

Infrastructure as Code and the AWS Object Model

To understand how Citrix virtualization technologies are integrated with and run on top of AWS, it is useful to start with a basic understanding of the object model behind some of their key/relevant services. This also allows us to describe the AWS platform in terms that are familiar to most IT professionals. To facilitate this understanding, we refer to the following diagram which represents the design pattern for a DaaS resource location on AWS:

Diagram 14: Deployed "Resource Location" architecture/design pattern for Citrix DaaS on AWS.

This design pattern is the foundation of most Citrix virtualization system architectures on AWS. It is also not just one massive pattern - it is built on various different, well maintained and documented design patterns for Enterprise IT on AWS. These patterns are represented, documented, and reproduced using AWS CloudFormation templates. AWS provides a library of Quick Start templates which can be run as-is, layered together ('nested') with other templates, and even duplicated and customized for your own specific needs. This highlights a couple of the other major advantages of public cloud infrastructure: infrastructure as code, and the 'pay as you go' nature of many cloud services. We dig more deeply into infrastructure as code in the Citrix virtualization world shortly, but we emphasize the point with a quick touchpoint that will likely resonate for the expected readers of this paper: for many enterprise IT architects, having access to such a vast library of services, design patterns, and technology tools at your fingertips is awesome. Combined with the ability to pay for resources as you consume them and simply remove them when you're done? This is a powerful way to learn about or evaluate new stuff, and it makes the ROI for at scale investments much easier to understand and communicate.

Back to the AWS object model for a moment: the top level object in diagram 14 is the AWS Region. You can think of AWS regions as clusters of well-connected but strategically separated data centers called Availability Zones. Each region will typically include 2 or more Availability Zones, which consist of one or more physical buildings with redundant power, networking, and connectivity. As of the time of this writing, AWS has 23 regions globally, which consist of 69 availability zones, but it is important to note that they're constantly investing in new regions and AZs. These numbers, while staggering to most of us, are likely already outdated by the time you read this. This highlights one of the other benefits of moving to public cloud infrastructure on AWS: you continue to benefit from the investments they're making (on a scale well beyond the reach of most IT organizations or even governments) over time. This continuous evolution/improvement, while daunting for change-averse IT organizations and business cultures, provides a broad reaching set of empowering benefits for Enterprise IT as it adapts to this 'new' model.

AWS region adoption choices are often based on proximity, services available, cost, compliance, or SLA. While choosing one or more right regions for your Citrix virtualization system is beyond the scope of this document, consider at least the following when making your choices:

• If you have one or more existing, customer-managed data centers you are connecting to AWS, consider one or more regions which provide the lowest latency network connectivity to your data centers and major offices.
• All regions can not have the AWS services or instance types you're looking for. AWS deploys new services or instance types initially to a few main regions, then expands to the rest over time. Also, newer regions cannot have older instance types - do your research before you build whenever possible.
• CVAD sites and DaaS resource locations are bound to a specific region. High availability for individual components of a site/resource location (such as cloud connectors, StoreFront servers, and ADC/Gateway VPX instances) is accomplished by placing resources in multiple availability zones in a given region.
• Don't go overboard spreading your infrastructure across regions: while it is easy to do on AWS, consider cost and complexity relative to the payoff you expect before you scale any system. You do end up paying for network traffic and storage traffic as well sometimes. The costs can be trivial for traffic while it is local to a region, but goes up when the traffic traverses regions or the Internet.

Stepping up a layer in Diagram 14 now, let's look at some of the networking constructs in this design pattern. The primary network construct on AWS is the VPC or "Virtual Private Cloud." VPCs are a regional construct (they span AZs) - you have at least one VPC in each region you deploy Citrix virtualization tech into. VPCs have a CIDR block of IP addresses defined, which must be unique if your network design routes traffic between multiple VPCs. VPCs are further broken down into subnets, and subnets are tied to an AZ (that is they do NOT span AZs in a region).

Subnets also have different attributes and objects associated with them, including routing policies and security policies. This is why the design patterns highlighted in this document (and other Citrix documentation) recommend putting VDAs in separate subnets from Cloud Connectors - so you can assign different routing and security policies to VDAs and Cloud Connectors.

Outbound Internet access from any subnet in a VPC (a regional construct) can be handled many different ways, but a common method is using NAT Gateways to provide Internet connectivity to private subnets. Public subnets are often served by Internet Gateways, which facilitate the routing of inbound connections to services you make accessible from the Internet.

Subnets are also commonly labeled as 'public' and 'private'. A public subnet is a subnet with Internet routable IP addresses assigned (in addition to the private IP addresses) and is associated with a route table that has a route to an Internet Gateway (IGW) for both inbound and outbound Internet traffic. A private subnet is a subnet with only private IP addresses assigned, and is associated with a route table that has a route for outbound internet access through a NAT Gateway or NAT Instances which reside in a public subnet. In a Citrix virtualization system, the Gateway virtual server (VIP) usually resides in a public subnet since it accepts inbound connections from client devices over the Internet and is used to securely proxy Citrix virtualization traffic into private subnets in a VPC.

There are many ways to build networks on AWS, with many innovative features and techniques available that you can't get elsewhere. We're not going to introduce you to them all here, but two tools/techniques worth looking into are VPC peering and transit gateways. These two constructs help introduce you to routing traffic between VPCs simply (VPC peering) or in a more Enterprise ready, hybrid cloud friendly model (transit gateways).

There's much more we can dig into here, and for the curious and motivated, there's a mountain of public domain knowledge available at your fingertips to learn more. For now, let's bring this back around to design patterns underneath all the diagrams you've seen in this paper.

One of the compelling attributes of the AWS platform is that it has been built on publicly consumable APIs. Why is this compelling? For one thing, this means that much any type of infrastructure component you can run on AWS can be reproducibly built from code. When combined with a powerful and comprehensive deployment service such as AWS CloudFormation, customers have a powerful framework for learning about, customizing, deploying, and managing IT systems. The concept of Infrastructure as Code can be new or perplexing for many traditional Enterprise focused technologists, but it can be transformational once adopted and practiced.

As we mentioned earlier, AWS provides a library of CloudFormation based Quick Start templates which can be run as-is, layered together ('nested') with other templates, and even duplicated and customized for your own specific needs. This library of templates is managed and maintained by AWS, in cooperation with technology partners such as Citrix, and these templates are often open-sourced (meaning they can be duplicated and modified as needed). As of the time of writing, the following Quick Start templates are available for Citrix technologies on AWS:

• Citrix ADC for Web Applications - deploys highly available Citrix ADC VPX instances on AWS. While the use case focus differs slightly, this design pattern is functional and relevant for Citrix Gateway deployments with CVAD/DaaS also.

Summary - Understanding Design Patterns for Citrix on AWS

Confused yet? If so, don't be alarmed: this can well be the start of your Citrix on AWS public cloud journey, and we've merely skimmed the surface of many deep topics here. Hopefully, however, we've successfully illustrated the following salient points:

• Infrastructure as Code is a powerful concept that can revolutionize the way complete systems are designed, built, and maintained.
• When deploying systems on AWS' public cloud, different components of any given solution can be represented by code, and built on-demand using AWS CloudFormation and other technologies.
• These components are represented by stack templates when using AWS CloudFormation, and templates can be copied and modified, as needed, to achieve the desired results.
• Templates can be nested, building complete systems (such as a fully functioning DaaS resource location on AWS) from the individual design patterns (templates).
• The Citrix DaaS on AWS Quick Start template is built upon three AWS managed/maintained foundation templates, which are well documented. Start with the following links to learn more about each:
• By using templates and performing trial builds, an Enterprise technologist can learn about, evaluate, and design systems that meet the specific needs of their organization or customer.

AWS Infrastructure Layer - extra Resources

The following resources can be used to help more about Citrix virtualization on AWS requirements and leading practices:

• Communication Ports Used by Citrix Technologies: a good global reference for the communication ports used by different components of the Citrix virtualization stack.

Operations Layer Considerations

This section defines the operational activities that administrators perform on a periodic basis. Many of these are not specific to AWS, and are detailed in existing published documentation. In the following tables, we've summarized some of the more important or AWS specific tasks. Readers can refer to the Monitor topic in Citrix product documentation for more information.

On-Demand Tasks

The following table outlines the tasks that are expected to be performed on-demand based on application requirements and troubleshooting efforts.

Daily Periodic Tasks

The following table outlines the tasks that ought to be performed daily.

Weekly Periodic Tasks

The following table outlines the tasks that are to be performed on a weekly basis.

Monthly Periodic Tasks

The following table outlines the tasks that are to be performed on a monthly basis.

Yearly Periodic Tasks

The following table outlines the tasks that are to be performed on a yearly basis.

Sources

Goal of this reference architecture is to assist you with planning your own implementation. To make this job easier, we would like to provide you with source diagrams that you can adapt in your own detailed designs and implementation guides: source diagrams.

This guide assists with the Architecture and deployment model of Citrix DaaS on Microsoft Azure.

The combination of Citrix Cloud and Microsoft Azure makes it possible to spin up new Citrix virtual resources with greater agility and elasticity, adjusting usage as requirements change. Virtual Machines on Azure support all the control and workload components required for a Citrix DaaS deployment. Citrix Cloud and Microsoft Azure have common control plane integrations that establish identity, governance, and security for global operations.

This document also provides guidance on prerequisites, architecture design considerations, and deployment guidance for customer environments. The document highlights the design decisions and deployment considerations across the following five key architectural principles:

• Operations - Operations includes a wide variety of topics such as image management, service monitoring, business continuity, support, and others. Various tools are available to assist with automation of operations including Azure PowerShell, Azure CLI, ARM Templates, and Azure API.

• Identity - One of the cornerstones of the entire picture of Azure is the identity of a person and their role-based access (RBAC). Azure identity is managed through Azure Active Directory (Azure AD) and Azure AD Domain Services. The customer must decide which way to go for its identity integration.

• Governance - The key to governance is establishing the policies, processes, and procedures associated with the planning, architecture, acquisition, deployment, and operational management of Azure resources.

• Security - Azure provides a wide array of configurable security options and the ability to control them so that customers can customize security to meet the unique requirements of their organization's deployments. This section helps to understand how Azure security capabilities can help you fulfill these requirements.

• Connectivity - Connecting Azure virtual networks with the customer's local/cloud network is referred to as hybrid networking. This section explains the options for network connectivity and network service routing.

Planning

The three most common scenarios for delivering Citrix Apps and Desktops through Azure are:

• Greenfield deployment with Citrix Cloud delivering resource locations in Azure. This scenario is delivered via the Citrix DaaS and used when customers prefer to go to a subscription model and outsource control plane infrastructure to Citrix.
• Extending an on-premises deployment into Azure. In this scenario, the customer has a current on-premises control layer and would like to add Azure as a Citrix resource location for new deployments or migration.
• Lift and shift. With this scenario, customers deploy their Citrix Management infrastructure into Azure and treat Azure as a site, using Citrix ADC and StoreFront to aggregate resources from multiple sites.

This document focuses on the Citrix Cloud deployment model. Customers can plan and adopt these services based on their organization needs:

Citrix DaaS

Citrix DaaS simplifies the delivery and management of Citrix technologies, helping customers to extend existing on-premises software deployments or move 100 percent to the cloud. Deliver secure access to Windows, Linux, and Web apps and Windows and Linux virtual desktops. Manage apps and desktops centrally across multiple resource locations while maintaining a great end-user experience.

Conceptual Reference Architecture

This conceptual architecture provides common guidelines for deployment of a Citrix Cloud resource location in Azure which will be discussed in the following sections.

Diagram-1: Citrix Cloud Conceptual Reference Architecture

Refer to the design guide on the scalability and economics of delivering Citrix DaaS on Microsoft Azure

Operations

In the operations subject area, this guide dives deeper into planning for the workspace environment requirements and hierarchy for foundational services. At the top layer, is found the subscription, resource group, and regional design considerations. Followed by common questions for VM storage, user profile storage, and Master Image management/provisioning. Also provided is guidance on Reserved instance optimization with Autoscale and planning for Business Continuity/Disaster Recovery.

Naming Conventions

The naming of resources in Microsoft Azure is important because:

• Most resources cannot be renamed after creation
• Specific resource types have different naming requirements
• Consistent naming conventions make resources easier to locate and can indicate the role of a resource

The key to success with naming conventions is establishing and following them across your applications and organizations.

When naming Azure subscriptions, verbose names make understanding the context and purpose of each subscription clear. Following a naming convention can improve clarity when working in an environment with many subscriptions.

A recommended pattern for naming subscriptions is:

When naming resources in Azure use common prefixes or suffixes to identify the type and context of the resource. While all the information about type, metadata, context, is available programmatically, applying common affixes simplifies visual identification. When incorporating affixes into your naming convention, it is important to clearly specify whether the affix is at the beginning of the name (prefix) or at the end (suffix).

A well-defined naming scheme identifies the system, role, environment, instance count, and location of an Azure resource. Naming can be enforced using an Azure Policy.

Subscriptions

Selecting a subscription model is a complex decision that involves understanding the growth of the customer's Azure footprint within and outside the Citrix deployment. Even if the Citrix deployment is small, the customer might still have a large amount of other resources that are reading/writing heavily against the Azure API, which can have a negative impact on the Citrix environment. The reverse is also true, where many Citrix resources can consume an inordinate number of the available API calls, reducing availability for other resources within the subscription.

Single Subscription workspace model

In a single subscription model, all core infrastructure and Citrix infrastructure are located in the same subscription. This is the configuration recommended for deployments that require up to 2,500 Citrix VDAs (can be session, pooled VDI, or persistent VDI). The limits are subject to change, check the following for most up to date VDA limits. Refer to the following blog for the latest start-shutdown scale numbers within a single subscription,

Diagram-2: Azure Single Subscription workspace model

Multi-Subscription workspace model

In this model, core infrastructure and Citrix infrastructure are in separate subscriptions to manage the scalability in large deployments. Often enterprise deployments with multi-region infrastructure designs are broken into multiple subscriptions to prevent reaching Azure subscription limits.

Diagram-3: Azure Multi-Subscription workspace model

The following questions provide guidance to help customer's understand the Azure subscription options and plan their resources.

Azure Regions

An Azure region is a set of data centers deployed within a latency-defined perimeter and connected through a dedicated regional low-latency network. Azure gives customers the flexibility to deploy applications where they need to. Azure is generally available in 59 regions around the world, with plans announced for 19 more regions as of the end of 2022.

A geography is a discrete market, typically containing two or more Azure regions, that preserve data residency and compliance boundaries. Geographies allow customers with specific data-residency and compliance needs to keep their data and applications close.

Availability Zones are physically separate locations within an Azure region. Each Availability Zone is made up of one or more data centers equipped with independent power, cooling and networking. Availability Zones allow customers to run mission-critical applications with high availability and low-latency replication. To ensure resiliency, there's a minimum of three separate zones in all enabled regions.

Consider these factors when choosing your region.

Availability Sets

An Availability Set is a logical grouping capability that can be used in Azure to ensure that the VM resources placed within an Availability Set are isolated from each other when they are deployed within an Azure data center. Azure ensures that the VMs placed within an Availability Set run across multiple physical servers, compute racks, storage units, and network switches. If a hardware or Azure software failure occurs, only a subset of your VMs is impacted, and the overall application stays up and remains available to customers. Availability Sets are an essential capability when customers want to build reliable cloud solutions.

Each component of a Citrix deployment is in its own Availability Set to maximize overall availability for Citrix. For example, Cloud Connectors use a separate Availability Set, another for Citrix Application Delivery Controllers (ADC), StoreFront, and so forth.

Once availability sets are optimized, the next step is to build resiliency around VM downtime within the availability sets. That minimizes/eliminates service downtime when VMs are restarted or redeployed by Microsoft. This can be expanded to planned maintenance events as well. There are two features that you can use which can increase the reliability of the overall service.

These two features do not protect against unplanned maintenance/crashes.

• Azure Planned Maintenance
• Azure Scheduled Events

Azure Planned Maintenance

Azure periodically does updates to improve the reliability, performance, and security of the host infrastructure in Azure. If maintenance requires a reboot, Microsoft sends a notice. Using Azure Planned Maintenance, it is possible to capture these notices and proactively take action on them on the customer's schedule, instead of on Microsoft's schedule.

Make use of the planned maintenance feature by sending email notifications to the service owner of each tier (for manual intervention) and build runbooks to automate the service protection.

Azure Scheduled Events

Azure Scheduled Events is an Azure Metadata Service that gives notices programmatically to applications to alert of immediate maintenance. It provides information about upcoming maintenance events (for example reboot) so the application administrator can prepare for and limit disruption. While it might sound like planned maintenance, it is not. The key difference is that these events are fired for planned maintenance and sometimes non-planned maintenance. For example, if Azure is doing host healing activities and needs to move VMs on a short notice.

These events are consumed programmatically, and will give the following advance notice:

• Freeze â�� 15 Minutes
• Reboot â�� 15 Minutes
• Redeploy â�� 10 Minutes

Disaster Recovery (DR)

Azure can provide a highly cost-effective DR solution for Citrix customers looking to gain immediate value from cloud adoption today. The deployment model topology determines the DR solution implementation.

Extending the Architecture

Under this topology, the management infrastructure remains on-premises, but workloads are deployed to Azure. If the on-premises data center is not reachable, existing connected users remain connected, but new connections will not be possible because the management infrastructure is unavailable.

To protect the management infrastructure, pre-configure Azure Site Recovery to recover the management infrastructure into Azure. This is a manual process and once recovered, your environment can be made operational. This option is not seamless and cannot recover components such as ADC VPX, however for organizations with more a more flexible recovery time objective (RTO) it can reduce the operational costs.

Hosting Architecture

When deploying this topology, the Citrix Management infrastructure is deployed into Azure and treated as a separate site. This provides functional isolation from on-premises deployment in the event of a site failure. Use Citrix ADC and StoreFront to aggregate resources and provide users a near instant failover between Production and Disaster Recovery resources.

The presence of the Citrix Infrastructure in Azure means that no manual processes need to be invoked and no systems need to be restored before users can access their core workspace.

Cloud Services Architecture

When using Citrix Cloud, Azure becomes just another resource location. This topology provides the simplest deployment as the management components are hosted by Citrix as a Service, and Disaster Recovery workloads can be achieved without deploying duplicate infrastructure to support it. The user experience during failover in the event of a disaster can be seamless.

The items in the following table help the customer with their DR planning:

Resource Groups

Resource Groups (RG) in Azure are a collection of assets in logical groups for easy or even automatic provisioning, monitoring, and access control, and for more effective management of their costs. The benefit of using RGs in Azure is grouping related resources that belong to an application together, as they share a unified lifecycle from creation to usage and finally, de-provisioning.

The key to having a successful design of resource groups is understanding the lifecycle of the resources that are included in them.

Resource Groups are tied to Machine Catalogs at creation time and cannot be added or changed later. To add extra Resource Groups to a Machine Catalog, the Machine Catalog must be removed and recreated.

Image Management

Image management is the process of creating, upgrading, and assigning an image that is consistently applied across development, test, and production environments. Consider the following when developing an image management process:

On-Demand Provisioning

The customer needs to determine if MCS be used to manage the Azure non-persistent machines or create their own Azure Resource Manager (ARM) templates. When a customer uses MCS to create machine catalogs, the Azure on-demand provisioning feature reduces storage costs, provides faster catalog creation and faster virtual machine (VM) power operations. With Azure on-demand provisioning, VMs are created only when Citrix DaaS initiates a power-on action, after the provisioning completes. A VM is visible in the Azure portal only when it is running, while in Citrix Studio, all VMs are visible, regardless of power status. Machines created via ARM templates or MCS can be power managed by Citrix using an Azure host connection in Citrix Studio.

Storage Account Containers

The customer needs to decide the organizational structure for the storing the source (or golden) images from which to create the virtual machines using Citrix Machine Creation Services (MCS). Citrix MCS images can be sourced from snapshots, managed or unmanaged disks and can reside on standard or premium storage. Unmanaged disks are accessed through general-purpose storage accounts and are stored as VHDs within Azure Blob storage containers. Containers are folders which can be used to separate Production, Test, and Development images.

Image Replication

The customer needs to determine the appropriate process for replicating images across regions and how Citrix App Layering technology might be used within the overall image management strategy. PowerShell scripts can be used with Azure Automation to schedule image replication. More information on Citrix App Layering can be found here, but keep in mind that Elastic Layering requires an SMB File share that does not reside on Azure Files. See the File Servers section for supported SMB share technologies that support Elastic Layering.

File Server Technologies

Azure offers several file server technologies that can be used to store Citrix user data, roaming profile information or function as targets for Citrix Layering shares. These options include the following:

• Standalone File Server
• File Servers using Storage Replica
• Scale Out File Server (SOFS) with Storage Spaces Direct (S2D)
• Distributed File System â�� Replication (DFS-R)
• Third-party storage appliances from Azure Marketplace (such as NetApp, and others)

The customer must select file server technologies that best meet their business requirements. The following table outlines some benefits and considerations for each of the different file serving technologies.

The recommended file server virtual machine types are generally DS1, DS2, DS3, DS4, or DS5, with the appropriate selection depending on customer use requirements. For best performance, ensure that premium disk support is selected. Extra guidance can be found on Microsoft Azure documentation.

Infrastructure Cost Management

Two technologies are available that can be used to reduce the costs of the Citrix environment in Azure, reserved instances and Citrix Autoscale.

Reserved Instances

Azure Reserved VM Instances (RIs) significantly reduce costs�up to 72 percent compared to pay-as-you-go prices�with one-year or three-year terms on Windows and Linux virtual machines (VMs). When customers combine the cost savings gained from Azure RIs with the added value of the Azure Hybrid Benefit, they can save up to 80 percent. The 80% is calculated based on a three-year Azure Reserved Instance commitment of a Windows Server when compared to the normal pay-as-you-go rate.

While Azure Reserved Instances require making upfront commitments on compute capacity, they also provide flexibility to exchange or cancel reserved instances at any time. A reservation only covers the virtual machine compute costs. It does not reduce any of the additional software, networking, or storage charges. This is good for the Citrix infrastructure and the minimum capacity needed for a use case (on and off hours).

Citrix Autoscale feature supports reserved instances as well to further reduce your costs - you can now use Autoscale for bursting in the cloud. In a delivery group you can tag machines that need to be autoscaled and exclude your reserved instances (or on-premises workloads) - you can find more info here: Restrict Autoscale to certain machines in a Delivery Group.

Citrix Autoscale

Autoscale is a feature exclusive to the Citrix DaaS that provides a consistent, high-performance solution to proactively power manage your machines. It aims to balance costs and user experience. Autoscale incorporates the deprecated Smart Scale technology into the Studio power management solution.

Diagram-4: Citrix Autoscale Flow

You can read more about Citrix Autoscale here.

Optimizing End-User Experience

Optimizing the end-user experience includes balancing the end user's perception of responsiveness with the business needs of staying within a budget. This section discusses the design concepts and decisions around providing an environment that is correctly sized for the business and the end user.

Defining the User Workspace

Review the following high-level questions to better understand existing use cases and the resources needed for their end users.

Azure VM Instance Types

Each Citrix component uses an associated virtual machine type in Azure. Each VM series available is mapped to a specific category of workloads (general purpose, compute-optimized, and so forth) with various sizes controlling the resources allocated to the VM (CPU, Memory, IOPS, network, and others).

Most Citrix deployments use the D-Series and F-Series instance types. The D-Series is commonly used for the Citrix infrastructure components and sometimes for the user workloads when they require extra memory beyond what is found in the F-Series instance types. F-Series instance types are the most common in the field for user workloads because of their faster processors which bring with them the perception of responsiveness.

Why D-Series or F-Series? From a Citrix perspective, most infrastructure components (Cloud Connectors, StoreFront, ADC, and so on) use CPU to run core processes. These VM types have a balanced CPU to Memory ratio, are hosted on uniform hardware (unlike the A-Series) for more consistent performance and support premium storage. Certainly, customers adjust their instance types to meet their needs and their budget.

The size and number of components within a customer's infrastructure will always depend on customer's requirements, scale, and workloads. However, with Azure we have the ability to scale dynamically and on-demand! For cost-conscious customers, starting smaller and scaling up is the best approach. Azure VMs require a reboot when changing size so plan these events within scheduled maintenance windows only and under established change control policies.

How about Scale-up or Scale-out?

Review the following high-level questions to better understand a customer's use case and the resources needed for their end users. This also helps them to plan their workload well in advance.

Scaling up is best when the cost per user per hour needs to be the lowest and a larger impact can be tolerated if the instances fail. Scaling out is preferred when the impact of a single instance failure needs to be minimized. The following table provides some example instance types for different Citrix components.

The latest instance type study was done to provide great insight in this area and we highly recommend the read. In all cases, customers evaluate the instance types with their workloads.

For graphic-intensive workloads, consider the NVv4-series virtual machines. They are powered by AMD EPYC 7002 processors and virtualized Radeon MI25 GPU. These virtual machines are optimized and designed for VDI and remote visualization. With partitioned GPUs, NVv4 offers the right size for workloads requiring smaller GPU resources at the most optimal price. Alternative the NVv3 series is optimized and designed for remote visualization, streaming, gaming, encoding, and VDI scenarios using frameworks such as OpenGL and DirectX. These VMs are backed by the NVIDIA Tesla M60 GPU. For further GPU options check the other offerings from Azure.

While scaling up is usually a preferred model to reduce the cost, Autoscale can benefit from smaller instances (15�20 sessions per host). Smaller instances host fewer user sessions than larger instances. Therefore, in the case of smaller instances, Autoscale puts machines into drain state much faster because it takes less time for the last user session to be logged off. As a result, Autoscale powers off smaller instances sooner, thereby reducing costs. You can read more about instance size considerations for Autoscale in the official documentation.

Storage

Just like any other computer, a virtual machine in Azure use disks as a place to store an operating system, applications, and data. All Azure virtual machines have at least two disks � a Windows operating system disk and a temporary disk. The operating system disk is created from an image, and both the operating system disk and the image are stored within Azure as virtual hard disks (VHDs). Virtual machines may also have extra disks attached as data disks, also stored as VHDs.

Azure Disks are designed to deliver enterprise-grade durability. Three performance tiers for storage exist that can be selected when creating disks: Premium SSD Disks, Standard SSD, and Standard HDD Storage, and the disks may be either managed or unmanaged. Managed disks are the default and are not subject to the storage account limitations like the unmanaged disks.

Managed Disks are recommended over the Unmanaged Disks by Microsoft. Consider Unmanaged Disks by exception only. Standard Storage (HDD and SSD) includes transaction costs (storage I/O) that must be considered but have lower costs per disk. Premium Storage has no transaction costs but have higher per disk costs and offers an improved user experience.

The disks offer no SLA unless an Availability Set is used. Availability Sets are not supported with Citrix MCS but should be included with Citrix Cloud Connector, ADC, and StoreFront.

Identity

The section focuses on Identity controls, workspace user planning, and the end-user experience. The primary design consideration is managing identities within both Azure and Citrix Cloud tenants.

Microsoft Azure Active Directory (Azure AD) is an identity and access management cloud solution that provides directory services, identity governance, and application access management. A single Azure AD directory is automatically associated with an Azure subscription when it is created.

Every Azure subscription has a trust relationship with an Azure AD directory to authenticate users, services, and devices. Multiple subscriptions can trust the same Azure AD directory, but a subscription will only trust a single Azure AD directory.

Microsoft's identity solutions span on-premises and cloud-based capabilities, creating a single user identity for authentication and authorization to all resources, regardless of location. This concept is known as Hybrid Identity. There are different design and configuration options for hybrid identity using Microsoft solutions, and in some cases, it might be difficult to determine which combination will best meet the needs of an organization.

Common Identity Design Considerations

Usually extending the customers Active Directory Site to Azure uses the use of Active directory replication to provide identity and authentication with the Citrix Workspace. A common step is to use AD Connect to replicate user to Azure Active Directory which provides you with the subscription-based activation required for Windows 10.

It is recommended to extend local Active Directory Domain Services to the Azure Virtual Network Subnet for full features and extensibility. Azure Role-Based Access Control (RBAC) helps provide fine-grained access management for Azure resources. Too many permissions can expose and account to attackers. Too few permissions mean that employees can't get their work done efficiently. Using RBAC, administrator can give employees the exact permissions they need.

Authentication

Domain Services (either AD DS or Azure AD DS) are required for core Citrix functionality. RBAC is an authorization system built on the Azure Resource Manager that provides fine-grained access management of resources in Azure. RBAC allows you to granularly control the level of access that users have. For example, you can limit a user to only manage virtual networks and another user to manage all resources in a resource group. Azure includes several built-in roles that you can use.

Azure AD Authentication is supported for Citrix Workspace, Citrix DaaS, and Citrix ADC/StoreFront authentication. For full SSON with Azure AD, Citrix Federated Authentication Service (FAS) or Azure AD DS (for core Domain Services) must be used.

Citrix FAS supports single sign-on (SSO) to DaaS in Citrix Workspace. Citrix FAS is typically adopted if you're using one of the following identity providers:

• Azure Active Directory
• Okta
• SAML 2.0
• Citrix Gateway

Active Directory and Azure Active Directory Outcomes

• Azure Active Directory Provisioned Tenant
• List of desired Organizational roles for Azure RBAC with mapping to Built-In or Custom Azure Roles
• List of desired Admin access levels (Account, Subscription, Resource Group and so on)
• Procedure to grant access/role to new users for Azure
• Procedure to assign JIT (just in time) elevation for users for specific tasks

Here is an example architecture of namespace layout and authentication flow.

Diagram-5: Architecture of namespace layout and authentication flow

Citrix Cloud Administration + Azure AD

By default, Citrix Cloud uses the Citrix Identity provider to manage the identity information for all users who access the Citrix Cloud. Customers can change this to use Azure Active Directory (AD) instead. By using Azure AD with Citrix Cloud, Customers can:

• Use their own Active Directory, so they can control auditing, password policies, and easily disable accounts when needed.
• Configure multifactor authentication for a higher level of security against the possibility of stolen sign-in credentials.
• Use a branded sign-in page, so your users know they're signing in at the right place.
• Use federation to an identity provider of your choice including ADFS, Okta, and Ping, among others.

Citrix Cloud includes an Azure AD app that allows Citrix Cloud to connect with Azure AD without the need for you to be logged in to an active Azure AD session. Citrix Cloud Administrator Login allows Azure AD identities to be used in the customers Citrix Cloud tenant.

• Determine if Citrix Cloud administrators use their Citrix Identity or Azure AD to access the Citrix Cloud the URL will follow the format https://citrix.cloud.com/go/{Customer Determined}
• Identify the Authentication URL for Azure AD authentication into Citrix Cloud

Governance

Azure Governance is a collection of concepts and services that are designed to enable management of your various Azure resources at scale. These services provide the ability to organize and structure your subscriptions in a logical way, to create, deploy, and reusable Azure native packages of resources. This subject is focused on establishing the policies, processes, and procedures associated with the planning, architecture, acquisition, deployment, operation, and management of Azure resources.

Citrix Cloud Administrator Login

Determine if Citrix Cloud administrators use their Citrix Identity, Active Directory Identity, or Azure AD to access Citrix Cloud. Azure AD integration enables multifactor authentication into Citrix Cloud for administrators. Identify the Authentication URL for Azure AD authentication into Citrix Cloud. URL follows the format https://citrix.cloud.com/go/{Customer Determined}.

RBAC permissions and delegation

Using Azure AD customers can implement their governance policies using Role-Based Access Control (RBAC) of Azure resources. One of the primary tools for the application of these permissions is the concept of a Resource Group. Think of a Resource Group as a bundle of Azure resources that share lifecycle and administrative ownership.

In the context of a Citrix environment organize these in a way that will allow for proper delegation between teams and promote the concept of least privilege. A good example is when a Citrix Cloud deployment uses a Citrix ADC VPX provisioned from the Azure Marketplace for external access. Although a core piece of Citrix infrastructure, the Citrix ADCs might have a separate update cycle, set of admins, and so on This would call for separating the Citrix ADCs from the other Citrix components into separate Resource Groups so the Azure RBAC permissions can be applied through the administrative zones of tenant, subscription, and resources.

MCS Service Principal