Nagaraj Harikar

Authors: Nagaraj Harikar, Dinesh Bansal In the realm of the internet infrastructure, DNSSEC (Domain Name System Security Extensions) plays a crucial role in safeguarding domain names and the associated data they point to. It employs cryptographic signatures to verify the authenticity and integrity of DNS records, preventing unauthorized modifications and protecting against DNS spoofing attacks. However, maintaining the effectiveness of DNSSEC requires regular key rollovers to ensure the continued validity of these signatures. Traditional key rollovers, often performed manually, can be a time-consuming and error-prone process. Automated DNSSEC signature rollover has emerged as a powerful and efficient solution to streamline this essential task. Understanding DNSSEC Key RolloverDNSSEC keys are employed to generate digital signatures that authenticate DNS records. These keys have a defined lifespan, and their timely renewal is essential for maintaining the integrity of DNSSEC protection. Key rollovers involve replacing the existing keys with new ones, ensuring that the cryptographic signatures remain valid and effective. Manual vs. Automated Key RolloverManual key rollovers, while effective, can be cumbersome and prone to human error. As shown in the steps below, the process involves generating new keys, updating the DNS zone, and propagating the changes across the DNS hierarchy. This manual intervention can be time-consuming and increases the risk of errors, potentially leading to disruptions in DNS resolution. Figure 1: DNSSEC Key rollover steps Steps involved in creating a new key: The first step involves creating a new cryptographic key on NetScaler. This key can be either a Zone Signing Key (ZSK) or a Key Signing Key (KSK) (create DNS key). In the second step, the newly created key is published. However, it cannot be used to sign any records (add DNS key).The published key is now active for use and is added to the zone to sign the zone (sign DNS zone).In the final step, the old key is deactivated and no longer used to sign any records (unsign DNS zone). Once the new signatures have been propagated and the old signatures are no longer needed, the old key is removed (remove DNS key). The entire process from step A to step D needs to be repeated in order to create a new ZSK or KSK. In the automated key rollover process, the steps from A to D are automated using the DNSSEC key rollover feature on NetScaler, which simplifies the key management and rollover tasks. For more information, refer to the Zone Maintenance documentation. Automatic Distribution of DNSSEC Keys in GSLB DeploymentsEarlier, if a global server load balancing (GSLB) domain was signed by a DNSSEC key that required a rollover, you had to create the keys on one of the GSLB site nodes and manually transfer these to other GSLB sites using scp or some other tool before they could be used. Now, this entire process can be automated by enabling the DNS zone transfer parameter and ensuring the AutomaticConfigSync option is enabled. For more information, refer to the Zone Maintenance for GSLB deployments. Benefits of Automated DNSSEC Signature RolloverAutomated DNSSEC signature rollover offers several compelling advantages: Reduced Operational Overhead: Automation eliminates the need for manual intervention, freeing up IT staff to focus on other critical tasks.Enhanced Security: NetScaler can perform rollovers more consistently and accurately, minimizing the risk of human error and any potential security vulnerabilities.Improved Efficiency: Automation streamlines the rollover process, reducing the time and resources required to maintain DNSSEC protection.Reduced Disruptions: NetScaler can perform rollovers without disrupting DNS resolution, ensuring consistent service availability.Implementing Automated DNSSEC Signature Rollover As mentioned above, there are two types of keys used by DNSSEC: Zone Signing Key (ZSK) and Key Signing Key (KSK). ZSK-type key is used to sign DNS resource records of various types such as A, AAAA, NS, SOA, etc. KSK-type key is used to sign DNSKEY records. Usually, the KSK-type key is created with a stronger algorithm and a bigger key size. Figure 2: Automatic DNSSEC key rollover with NetScaler In the following example, we use the ‘create DNS key’ command to generate a DNSSEC key (example.ksk) of type KSK in zone example.com with key size 1024 using algorithm RSASHA256. Then we publish this key in the zone ‘add DNS key’ command with auto-rollover enabled.The key has an expiry period of ten days and needs to roll over five days before the expiry determined by the notification period. Then use the ‘sign DNS zone’ command to use this key to sign the records under DNS Zone ‘example.com.’ All these steps will be performed automatically at the time of rollover of the successor key since auto-rollover is enabled on the key. This process with a rollover period R is shown in Figure 2 above. Figure 3: Example of configuring auto-rollover of DNSSEC key ConclusionThe Automated DNSSEC Signature Rollover feature will be critical for maintaining the effectiveness of DNSSEC protection. Streamlining the key rollover process, it reduces administrative burden, enhances security, and ensures the integrity of DNS records. As the demand for secure and reliable DNS services grows, automated DNSSEC signature rollover will play an increasingly important role in safeguarding the internet infrastructure. NetScaler also supports DNS over TLS, which encrypts DNS queries, enhancing privacy and security by safeguarding against potential eavesdropping and manipulation of domain name resolution, ensuring a safer online experience.

The limitations of static load balancing Traditional GSLB solutions rely on DNS-based load balancing to direct users to the closest available server. However, these solutions have a number of limitations. First, they lack the ability to accurately measure geolocation and network latency, which can result in suboptimal routing decisions. Second, they do not provide visibility into network congestion, micro outages or other factors that can impact performance. Third, they require manual configuration and maintenance, which can be time-consuming and error-prone. Need for real-time data and insights: In today's fast-paced and rapidly changing internet condition, there is a growing need for real-time data and insights that can help organizations to make better decisions. Internet state data from real users can help to provide this type of information, by enabling the collection and analysis of data in real-time. The benefits of modern GSLB Modern GSLB solutions, such as the one offered by NetScaler, leverage advanced technologies to overcome these limitations. For example, Intelligent Traffic Management (ITM) uses real-time network analytics to optimize traffic routing decisions (Figure1) based on actual network performance, rather than just geographical location. This ensures that users are always directed to the best-performing server, regardless of their location. Figure1: ITM optimized algorithm to improve your application performance In addition, ITM provides real-time visibility into network performance and congestion, allowing businesses to proactively address performance issues before they impact end-users. They also automate configuration and maintenance, reducing the risk of errors and freeing up IT resources for more strategic initiatives. Multi-CDN use-caseMulti-CDN global server load balancing (GSLB) with ITM (Citrix Intelligent Traffic Management) is a solution that enables organizations to provide a high-performance, highly available, and scalable user experience for their global customers. This solution leverages multiple content delivery networks (CDNs) to distribute user traffic across the best-performing CDN based on real-time data and performance metrics. With ITM, organizations can define GSLB policies that automatically route user traffic to the optimal CDN or the origin server based on factors such as user location, network conditions, and CDN availability. This approach helps to ensure that users are always directed to the optimal data source, reducing latency, improving the user experience and save cost. Here's how multi-CDN GSLB works: A user requests content from a website or application that is delivered by a CDN. The DNS query for the website or application is directed to the NetScaler ITM. The NetScaler ITM uses its geographic and network-aware load balancing algorithms to determine the most optimal CDN or Origin Datacenter for the user based on their location, network conditions, and other factors. The NetScaler ITM returns the IP address of the optimal CDN to the user's device. The user's device sends the request for the content to the CDN with the returned IP address. The content is delivered to the user's device from the CDN. As shown in Figure2, If CDN2's E2a edge becomes unavailable, the NetScaler ITM automatically redirects the user to the next optimal CDN1 E1b edge to ensure continuous delivery of the content. Figure2: ITM optimized multi-cdn high availability Figure3: ITM ensuring business continuity for its users during Azure global outage (Jan 25th 2023). Conclusion Traditional algorithms in GSLB solutions have been a mainstay of web traffic management for many years, but they are increasingly being supplanted by more modern and advanced solutions like NetScaler ITM. By leveraging real-time network analytics, providing visibility into network performance, and automating configuration and maintenance, modern GSLB solutions offer a more effective and efficient way to manage web traffic in today's complex digital environment. If you're application is deployed across multiple datacenters, it's time to consider upgrading to this modern solution. Contact NetScaler netscaler-itm@cloud.com today to learn more about how our Intelligent Traffic Management solution can help you optimize your web traffic and deliver a superior user experience.

CADS Service is a SaaS from Citrix, that radically simplifies application delivery and security and accelerates IT modernization by bringing intent-based configuration, automated self-healing and internet awareness to hybrid multi-cloud deployments (more info). CADS service will improve and simplify your secure app delivery and ensure compliance by deploying and configuring infrastructure in your public cloud data center in line with your business intent. CADS also provides Internet state awareness with billions of real users’ measurements a day, from every corner of the internet. It monitors internet traffic issues in real time and automatically steers your user traffic to an optimal site. Whether you host your applications and content on premises, in the cloud or in content delivery networks (CDNs), CADS service allows you to globally load balance all traffic, dynamically optimizing the user experience and lowering service costs. Real user monitoring (RUM) gives a direct understanding of how internet performance impacts customer satisfaction and engagement. CADS service gathers RUM data from clients as they access applications across clouds, data centers, and CDNs, and builds a holistic picture of internet health. Use-Case: Resilient delivery of applications for globally-distributed publicly deployed application workloads in hybrid data centres. Architecture: With the Citrix Managed multi-site application feature, you can configure Global Server Load Balancing (GSLB) to deliver applications from multiple cloud environments for high availability and reliability. GSLB enables fast site failover, disaster recovery and improved user experience. When an application is deployed across multiple sites, requests can be intelligently distributed across all of an organization’s data centers. Once an application is configured as multi-site, CADS services will monitor the health and availability of each site. The latency from users across the globe to each PoP is measured using real user measurements (RUM) in near real-time. For more details click here. Figure 1 shows a multi-site set up where both a Citrix-Managed deployment of CADS service and independent sites are configured into CADS service. This can improve the experience for each individual user as they access the application from across the globe. When an application is configured as a multi-site application on CADS Service, client requests are routed to the optimal data center for each individual user. This minimizes network latency and improves user experience by accelerating application response time. For more information click here. Figure 1: Architecture for CADS and ITM integration for global application delivery Pre-Requisites Before you deliver a multi-site application, you must complete the following preliminary steps:  Create a Citrix cloud account profile. Ensure your application environment is publicly addressable with an IP or FDQN. If you want Citrix Managed Service for your application delivery as shown in Figure 2, see Deliver an application. Steps to deploy a CADS service managed Application as your GSLB site Your Citrix manged application will be available in the CADS service user interface (UI) under Applications. The following example shows two Citrix Managed deployments with applications in the Virginia-Prod-Site and the EU-Prod-Env in the AWS North Virginia and Ireland regions respectively. Figure 2: Existing applications on your Citrix Managed Datacenter You need to configure your Multi-Site application by clicking on New Multi-Site Application. There are three steps to configure your Multi-Site Application delivery. Specify the name “GlobalApp”, you can choose the application FQDN type as “User Defined” for Route53 Hosted Zones or CADS service will generate an FQDN if the “Auto-allocated” option is selected (#.itm.appdeliverysecurity.com). You can also specify the DNS time to live as show in Figure 3. Figure 3: Create new multi-site application Specify the Site details. In this example, as shown in Figure 4, we will add the two managed Sites that are shown in Step 1 Figure 4: Create site1 from available Citrix Managed site Select Managed, specify the Site name “Site1”. Select the Application “Virginia-Prod-Site” and the Endpoint. The FQDN, the Location and Monitor details are auto-populated. You may, optionally, select Geo Fencing to ensure users from a particular region access a particular site (North American users will be sent to Virginia in this example) as shown in Figure 5. Figure 5: Geo fencing settings Add the Second Site, “Site2” for the Application “EU-Prod-Env” and endpoint and Add Site as shown in Figure 6 Figure 6: Add Site2 Now the Citrix Managed Sites are added as shown in Figure 7. Note: Here we have added Citrix-managed sites. If you would like to add sites which are user defined or self manged, refer to Appendix 4.a Figure 7: Site and location details added for Site1 and Site2 Select the GSLB Algorithm and Stickiness settings as shown in Figure 8. CADS Service supports three Algorithms – “Failover”, “Round Robin” and “Optimal RTT”. In this example deployment Optimal RTT is used. For detailed steps refer to this document. Figure 8 : GSLB method and stickiness configuration for the GSLB sites Once the deployment is successful as shown in Figure 9, Click “Manage Multi-Site Applications” to see the FQDN generated for the APP as shown in Figure 10. Figure 9: Deploying Multi-site application on CADS service Figure 10: Multi-site application details If you selected Route53 the autogenerated FQDN is automatically mapped to a friendly FQDN name, otherwise, you can Map your multi-site application FQDN to “0014.16ed2.itm.appdeliverysecurity.cloud.com” in your DNS provider (CNAME entry). Now you can test the application traffic routing in dnscheker.org, From the diagram it can be seen that all North America users are routed to North Virginia Site with IP 34.231.174.213 as shown in Figure 11. Figure 11: DNS checker results for the multi-site application Analytics for your multi-site application Figure 12: Site health information for selected duration The GlobalApp’s status is shown in Figure 12. There are legends which describe the site status. Namely, Health, Unhealthy (When all sites are down), Degraded (when some of the sites are down), Maintenance, Not Deployed. Figure 13 shows the max and average user request rate for your Multi-Site Application. Figure 13: Application user’s request rate Geo based access data is show in Figure 14. Which displays the total number of application requests that are received by CADS service, Top 5 locations from where the user traffic is originating. Figure 14: Geo map showing the top five locations of users accessing the multi-site application Conclusion CADS Service provides a simplified way to configure an internet-aware, intelligent global server load balancing solution for a multi-cloud environment. In this example Geo Fencing has been used to ensure application traffic from users accessing from a particular location are steered to a specific site. With this, the overall user experience will be improved with the use of the Optimal RTT algorithm offered by CADS service. Since the users are always routed to an optimal performing site, this implicitly provides disaster recovery across all the configured sites. Appendix Adding a non Citrix Managed Site You can Select “User-Defined” and specify the IP Address (v4 or v6) or FQDN names where the application is deployed publicly. Note: In order to configure Optimal RTT as the GSLB Algorithm, you need to enable “Configure Radar” in - Refer to section 4.2. Enable “Configure Radar” Option for the Site by specifying the path to the r20.gif on your server and location of the Site. If you do not have the radar tags configured, you can host it on an Apache server (Step 4.3) Steps to deploy our Radar Objects on a host with Apache2 Note: The instructions were tested against Ubuntu20, but the Radar Objects can be served by any modern operating system and webserver. Install required packages sudo apt-get install apache2 git Create the Apache2 directory and populate it sudo mkdir -p /var/www/radar-objects sudo git clone https://github.com/cedexis/testobjects /var/www/radar-objects/ Disable the default Apache2 VirtualHost sudo rm /etc/apache2/sites-enabled/000-default.conf Enable required Apache2 mods sudo a2enmod headers sudo a2enmod rewrite sudo a2enmod ssl sudo systemctl restart apache2 Add the Apache2 site configuration cat > /etc/apache2/sites-available/radar-objects.conf << 'EOF' ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined DocumentRoot /var/www/radar-objects Header add "Timing-Allow-Origin" "*" RewriteEngine on RedirectMatch 404 /\.git <VirtualHost *:80> RewriteRule ^/img/(.*)/(.*)$ /img/$2 [L] RewriteRule ^/sm/(.*)/(.*)$ /sm/$2 [L] </VirtualHost> SSLStaplingCache shmcb:${APACHE_RUN_DIR}/logs/stapling_cache(128000) SSLSessionCache shmcb:${APACHE_RUN_DIR}/logs/ssl_scache(512000) <VirtualHost *:443> SSLEngine On SSLCertificateFile /etc/ssl/public.crt SSLCertificateKeyFile /etc/ssl/private.key SSLCACertificateFile /etc/ssl/ca-certs.pem SSLProtocol -all +TLSv1.3 +TLSv1.2 SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS" SSLUseStapling On SSLSessionCacheTimeout 300 RewriteRule ^/img/(.*)/(.*)$ /img/$2 [L] RewriteRule ^/sm/(.*)/(.*)$ /sm/$2 [L] </VirtualHost> EOF Enable the new Apache site and restart Apache2 sudo ln -s /etc/apache2/sites-available/radar-objects.conf /etc/apache2/sites-enabled/radar-objects.conf sudo systemctl restart apache2 Verify that the Radar Objects are being served curl https://<FQDN>/img/r20.gif

Introduction Global server load balancing (GSLB) is regarded as an important internet infrastructure that supports business on the internet. The scalability and the availability of the Web can be provided by distributing the Web servers where the client requests must be balanced among these Web servers in order to improve the performance. Network latency has been identified as an important metric to improve the quality of service (QoS). Given the limitations to response times because of the geographic distance between servers and end-users, it becomes important to leverage advanced architectures and functionalities for GSLB to realize fast responses. In this article we will see how Citrix App Delivery and Security (CADS) service provide a SaaS based fully managed cloud load balancing solution. We will also discuss functionalities that contribute towards improving QoS. Need for advanced GSLB for applications GSLB is generally implemented to achieve: Disaster recovery Improved application performance Reliability Fulfil compliance requirements Efficient Connections Reduced latency Among this, disaster recovery or high availability is one of the primary reason that a lot of businesses deploy resources redundantly at multiple places. In most typical configurations, application data is served in one location (active) where most of its user accesses from and one or more standby (passive) locations that functions only when the active site fails. This mode of switching is also called as failover. The other configuration is active-active deployment, where multiple sites are active at the same time, which demands more advanced implementation of the GSLB. Some of which include constantly monitoring the health and latency of the sites from the actual end user’s location. The service needs to actively have this data and co-relate the end user experience to make better decisions in facilitating the users to access an optimal site depending on various factors like user’s ISP to public cloud server link performance, real user monitoring data of clients accessing various public cloud endpoints like web application, APIs, CDNs, factoring in usage data in the form of time of the day, server performance, etc. CADS service advanced GSLB capabilities CADS service leverages real-user performance monitoring (RUM) data functionality and perform data-driven DNS or API-based global load balancing. This platform is unique in that it employs end-user-based probes for collecting real-time information from clients, along with synthetic monitoring from multiple PoP locations spread globally as shown in the Figure 1. Here, based on the type of configuration specified on CADS service, users receive the optimal site information for their DNS queries. These sites can be on-prem, public or private cloud. Figure 1: CADS service with advanced GSLB capabilities CADS service offers a simplified workflow to quickly deploy GSLB service for globally distributed application sites. Following are the steps to onboard yourself and use CADS service for your advanced GLSB solutions. For detailed steps click here. In the following sections, let us see various use-cases that CADS service supports for Global server load balancing. Disaster recovery site for on-prem datacentre in cloud - Active-Passive Mode Use-case: Create a Disaster Recovery site in Public cloud for your existing On-Prem datacentre. Create a multi-site application with CADS service in GSLB Active-Passive mode. Note: Detailed procedure available Specify Main and Backup Site Add two sites (Existing On-Prem datacentre and new public cloud datacentre) for Active-Passive deployment as show in Figure 2. Specify GSLB parameters Select Fail-Over as the algorithm (Figure 3). By default a priority of 1 is assigned to the primary site (site1) and an increasing priority of 2 to the standby site (site2) as shown in Figure 4. Figure 3: Select desired GSLB configuration Figure 4: Option to change the priority of the site Note: Application data replication and sync between the primary and backup sites are not managed by CADS. Figure 5: CADS service Active-Passive Deployment scenario If site with priority 1 is down, client requests are directed to site with priority 2. If both of these sites are UP, traffic is directed to site1 since it has higher priority as shown in the Figure 5. Even distribution of users across datacentres with persistence. Use-case: Round-robin distribution of users across sites spread globally and ensure consecutive requests should go to the same datacentre. Create a multi-site application with CADS service in GSLB Active-Active mode, evenly distributing traffic across sites. Also configure stickiness. Note: Detailed procedure available . Following are steps specific to the use-case Specify the application site details Add two or more sites for Active-Active deployment Step3: Specify GSLB parameters Select Round-Robin as the algorithm (Figure 6), Enable Stickiness with a time to live value of 120 as show in Figure 7. This value controls the time duration in seconds within which subsequent client requests to the Multi-Site Application will be sent to the same site. Figure 6: Select GSLB configuration Figure 7: Enable site persistence with CADS stickiness settings Figure 8: CADS service Active-Active Deployment scenario with Round Robin algorithm with site stickiness As shown in the Figure 8, individual client DNS requests are served with the addresses of sites in a round robin way across all the datacentres. When the first site (On-Prem datacentre in this case) goes down, clients are redirected to the next datacentre in the round robin queue. Once the site is back up, client traffic continues to the current selected site until the client stickiness time of 120secs are not expired. Over time clients gets distributed across all the healthy datacentres. Note: You control the distribution of traffic across your datacenters using weights to each site. For example, assign a weight 90 to Site 1 and weight 10 to Site 2. Weights are proportional, i.e. 90 % of the traffic is received by Site 1 and 10% by Site 2. You can alter this to control the traffic proportions to your datacentres Optimal site selection with advanced GSLB capabilities Use-case: Route users to the optimally performing site irrespective of their location proximity. Create a multi-site application with CADS service in GSLB Active-Active mode with advance GSLB capabilities to optimally route users to the Site. Note: Detailed procedure available . Following are steps specific to the use-case Specify the application site details Add two or more sites for Active-Active deployment. Specify GSLB parameters Select Optimal RTT as the algorithm as show in Figure 9. This option is only available for managed site and user-defined sites with radar enabled (more details). Figure 9: Select GSLB configuration Figure 10: Penalty settings for sites with Optimal RTT GSLB configuration You can penalize a site by adding an additional latency (Figure 10). When you add penalty to a site, its additional latency is added to the one calculated by Real User Measurements. Figure 11: CADS service Active-Active Deployment scenario with Optimal RTT algorithm When a Site goes down, as shown in Figure 11, users are redirected to the next available optimal site. Once the Site recovers users are redistribute again to their respective optimal site which is dynamically determined. Figure 12: Optimal site selection As shown in Figure 12, even if user’s physical location is closer to Site2 user will be directed to Site1 if it has lower application latency (response time in ms) when compared with Site2. Geo based traffic steering Create a multi-site application with CADS service and configure Geo based site preference Specify the application site details Add two or more sites for deployment. While adding a site, configure the geo fencing for the site based on your preference. Figure 13: Configure geo fencing for site As shown in the Figure 13, all users accessing the application from North America region will be served from site located in San Francisco, CA, USA as per this configuration. This works for all the GSLB algorithms. Benefits to customers ITM PoPs spread across the globe monitoring internet service provider, public cloud provider, content delivery service provider performance and ensuring end user experience is improved and get optimal global server load balancing service. Call to Action Try out Citrix App Delivery and Security Service here.