Jump to content
Updated Privacy Statement
  • Nagaraj Harikar
    Authors: Nagaraj Harikar, Dinesh Bansal

    In the realm of the internet infrastructure, DNSSEC (Domain Name System Security Extensions) plays a crucial role in safeguarding domain names and the associated data they point to. It employs cryptographic signatures to verify the authenticity and integrity of DNS records, preventing unauthorized modifications and protecting against DNS spoofing attacks. However, maintaining the effectiveness of DNSSEC requires regular key rollovers to ensure the continued validity of these signatures.
    Traditional key rollovers, often performed manually, can be a time-consuming and error-prone process. Automated DNSSEC signature rollover has emerged as a powerful and efficient solution to streamline this essential task.
    Understanding DNSSEC Key Rollover
    DNSSEC keys are employed to generate digital signatures that authenticate DNS records. These keys have a defined lifespan, and their timely renewal is essential for maintaining the integrity of DNSSEC protection. Key rollovers involve replacing the existing keys with new ones, ensuring that the cryptographic signatures remain valid and effective.
    Manual vs. Automated Key Rollover
    Manual key rollovers, while effective, can be cumbersome and prone to human error. As shown in the steps below, the process involves generating new keys, updating the DNS zone, and propagating the changes across the DNS hierarchy. This manual intervention can be time-consuming and increases the risk of errors, potentially leading to disruptions in DNS resolution.

    Figure 1: DNSSEC Key rollover steps
    Steps involved in creating a new key:
     The first step involves creating a new cryptographic key on NetScaler. This key can be either a Zone Signing Key (ZSK) or a Key Signing Key (KSK) (create DNS key).  In the second step, the newly created key is published. However, it cannot be used to sign any records (add DNS key). The published key is now active for use and is added to the zone to sign the zone (sign DNS zone). In the final step, the old key is deactivated and no longer used to sign any records (unsign DNS zone). Once the new signatures have been propagated and the old signatures are no longer needed, the old key is removed (remove DNS key). The entire process from step A to step D needs to be repeated in order to create a new ZSK or KSK.
    In the automated key rollover process, the steps from A to D are automated using the DNSSEC key rollover feature on NetScaler, which simplifies the key management and rollover tasks. For more information, refer to the Zone Maintenance documentation.
    Automatic Distribution of DNSSEC Keys in GSLB Deployments
    Earlier, if a global server load balancing (GSLB) domain was signed by a DNSSEC key that required a rollover, you had to create the keys on one of the GSLB site nodes and manually transfer these to other GSLB sites using scp or some other tool before they could be used. Now, this entire process can be automated by enabling the DNS zone transfer parameter and ensuring the AutomaticConfigSync option is enabled. For more information, refer to the Zone Maintenance for GSLB deployments.
    Benefits of Automated DNSSEC Signature Rollover
    Automated DNSSEC signature rollover offers several compelling advantages:
    Reduced Operational Overhead: Automation eliminates the need for manual intervention, freeing up IT staff to focus on other critical tasks. Enhanced Security: NetScaler can perform rollovers more consistently and accurately, minimizing the risk of human error and any potential security vulnerabilities. Improved Efficiency: Automation streamlines the rollover process, reducing the time and resources required to maintain DNSSEC protection. Reduced Disruptions: NetScaler can perform rollovers without disrupting DNS resolution, ensuring consistent service availability. Implementing Automated DNSSEC Signature Rollover
     As mentioned above, there are two types of keys used by DNSSEC: Zone Signing Key (ZSK) and Key Signing Key (KSK). ZSK-type key is used to sign DNS resource records of various types such as A, AAAA, NS, SOA, etc. KSK-type key is used to sign DNSKEY records. Usually, the KSK-type key is created with a stronger algorithm and a bigger key size. 

    Figure 2: Automatic DNSSEC key rollover with NetScaler
    In the following example, we use the ‘create DNS key’ command to generate a DNSSEC key (example.ksk) of type KSK in zone example.com with key size 1024 using algorithm RSASHA256. Then we publish this key in the zone ‘add DNS key’ command with auto-rollover enabled.The key has an expiry period of ten days and needs to roll over five days before the expiry determined by the notification period. Then use the ‘sign DNS zone’ command to use this key to sign the records under DNS Zone ‘example.com.’ All these steps will be performed automatically at the time of rollover of the successor key since auto-rollover is enabled on the key. This process with a rollover period R is shown in Figure 2 above.
    Figure 3: Example of configuring auto-rollover of DNSSEC key
    The Automated DNSSEC Signature Rollover feature will be critical for maintaining the effectiveness of DNSSEC protection. Streamlining the key rollover process, it reduces administrative burden, enhances security, and ensures the integrity of DNS records. As the demand for secure and reliable DNS services grows, automated DNSSEC signature rollover will play an increasingly important role in safeguarding the internet infrastructure.
    NetScaler also supports DNS over TLS, which encrypts DNS queries, enhancing privacy and security by safeguarding against potential eavesdropping and manipulation of domain name resolution, ensuring a safer online experience.

    Ravi Shekhar
    NetScaler VPX 's storage allocation is pivotal and contingent upon your sizing estimations. By default, it offers a standard storage capacity of 20GB.

    If your data storage needs surpass this limit, attaching an additional disk becomes essential. This extra disk typically defaults to the /var/crash path, intended for storing heavy core-dumps and crash files.
    Yet, various folders within /var, such as nsinstall, nstrace, log etc., often contribute to space consumption, potentially impacting storage availability.
    In this article, we unveil an easy yet effective strategy to optimize storage by leveraging the additional disk for folders that might consume excessive space.
    In this article, we will give you a simple hack on how to utilize the additional disk for any folder that may consume more space.

    Key Considerations:
    Evaluate and estimate storage needs before attaching an extra disk.
    For NetScaler VPX deployments, we recommend using solid-state drive (SSD) technology.
    Step by Step Guide
    In this example, we bring you the detailed instructions on mapping the /var/log folder to the additional disk on a NetScaler VPX instance running on an ESXi hypervisor has been provided. 
    Step 1 - Shut down the NetScaler VPX virtual machine (if running) from the hypervisor management console

    Step 2 - Add a new virtual hard disk

    Step 3 - Power on the virtual machine

    Step 4 - The new virtual disk will be mounted at /var/crash after NetScaler VPX  boots up. 
    Please note that the mounted partition will be slightly smaller than the actual virtual disk size

    Step 5 - Create a new directory within /var/crash that will later replace the existing directory from your NetScaler VPX

    Step 6 - Use the new disk for storing all log files, you can create the log directory inside /var/crash

    Step 7 - Copy/move all files recursively from the old directory (/var/log/) to new directory (/var/crash/log/)

    Step 8 - Once the file operation has completed, delete the old directory (eg., /var/log/) and create a symlink at it's place pointing to the new directory (/var/crash/log/)

    Step 9 - Now the NetScaler ADC VPX will use the newly added disk for all files stored inside this directory

    In a similar way, multiple directories can be created inside /var/crash following the same method each mapped to a different directory path on the system (/root, /var/core, etc.)

    NetScaler Cyber Threat Intelligence
    CVE-2023-50164: Apache Struts - Files or Directories Accessible to External Parties - (v120 signature update published )
     NetScaler CTRI Team
    Last Updated: 12/13/2023
     A security vulnerability, identified as CVE-2023-50164, has been discovered in Apache Struts, a popular, open-source framework for building Java web applications.
     The vulnerability affects the file upload functionality of versions prior to Apache Struts 2.5.33 and Struts The problem stems from how the framework handles the HTTP parameters related to file uploading.
     An unauthenticated, remote attacker can manipulate file upload parameters to perform unauthorized path traversal. This could allow the attacker to upload malicious files on the server and potentially execute arbitrary code remotely.
     Please follow the guidelines as recommended by the vendor in their Security Bulletin
     NetScaler CTRI :
    NetScaler CTRI team is actively investigating this issue and will provide an update on the mitigation steps and a WAF Signature soon. 
    Update: Signature v120 published

    NetScaler Cyber Threat Intelligence
    NetScaler WAF Signatures Update v120
     NetScaler has released a new version of its integrated Web App Firewall signatures to help customers mitigate several CVEs with variable CVSS.
    The most critical is a security vulnerability, identified as CVE-2023-50164, which has been discovered in Apache Struts, a popular, open-source framework for building Java web applications. The vulnerability affects the file upload functionality of versions prior to Apache Struts 2.5.33 and Struts The problem stems from how the framework handles the HTTP parameters related to file uploading.
    An unauthenticated, remote attacker can manipulate file upload parameters to perform unauthorized path traversal. This could allow the attacker to upload malicious files on the server and potentially execute arbitrary code remotely.
    Please follow the guidelines as recommended by the vendor in their Security Bulletin
      Signatures included in v120:
    CVE ID
    WEB-STRUTS Apache Struts Prior to - Path Traversal Vulnerability (CVE-2023-50164)
    WEB-MISC ownCloud Prior to 10.13.1 - Access Control Bypass Vulnerability (CVE-2023-4105)
    WEB-MISC ownCloud Multiple Versions - Information Disclosure Vulnerability (CVE-2023-49103)
    WEB-MISC SysAid Server On-Premise Prior to 23.3.36 - Path Traversal Vulnerability (CVE-2023-47246)
    WEB-MISC Contec SolarView Compact 6.0 and Prior - OS Command Injection Vulnerability (CVE-2023-46509)
    WEB-MISC NETGEAR ProSAFE Network Management System Prior to - SQL Injection Vulnerability (CVE-2023-44450)
    WEB-MISC NETGEAR ProSAFE Network Management System Prior to - SQL Injection Vulnerability (CVE-2023-44449)
    CVE-2023-44351, CVE-2023-44353
    WEB-MISC Adobe ColdFusion - Deserialization of Untrusted Data Vulnerability (CVE-2023-44351, CVE-2023-44353)
    WEB-MISC CrushFTP Prior to 10.5.1 - Improper Control of Dynamically-Managed Code Resources Vulnerability (CVE-2023-43177)
    WEB-MISC SolarWinds Orion Prior to 2023.4.0 - Improper Input Validation Vulnerability Via TestAction (CVE-2023-40062)
    WEB-MISC SolarWinds Orion Prior to 2023.4.0 - Improper Input Validation Vulnerability Via /api/WriteToFile/ (CVE-2023-40062)
    WEB-MISC SolarWinds NCM Prior to 2023.4.1 - Directory Traversal Vulnerability Via SaveResultsToFile (CVE-2023-40055)
    WEB-MISC SolarWinds NCM Prior to 2023.4.1 - Directory Traversal Vulnerability Via txtConfigTemplate (CVE-2023-40054)
    WEB-MISC SolarWinds NCM Prior to 2023.4.1 - Directory Traversal Vulnerability Via txtPath (CVE-2023-40054)
    Zoho ADManager Plus Prior to 7203 - Directory traversal Vulnerability (CVE-2023-39912)
    WEB-MISC XWiki Multiple Versions - Arbitrary Code Injection Vulnerability (CVE-2023-35150)
    WEB-MISC Splunk Enterprise - Escalation of Privileges Vulnerability (CVE-2023-32707)
    WEB-MISC Moodle Prior to 4.1.3 - TinyMCE Loaders Stored Cross-Site Scripting Vulnerability Via loader (CVE-2023-30943)
    WEB-MISC Moodle Prior to 4.1.3 - TinyMCE Loaders Stored Cross-Site Scripting Vulnerability Via lang (CVE-2023-30943)
    WEB-MISC OpenEMR Prior to 7.0.1 - HTML Code Injection Vulnerability (CVE-2023-2943)
     NetScaler customers can quickly import the above signatures to help reduce risk and lower exposure associated with these vulnerabilities. Signatures are compatible with NetScaler (formerly Citrix ADC) software version 11.1, 12.0, 12.1, 13.0 and 13.1. NOTE: Software versions 11.1 and 12.0 are end of life, and you should consider upgrading for continued support. Learn more about the NetScaler software release lifecycle.
     If you are already using NetScaler Web App Firewall with the signature auto-update feature enabled, verify that your signature file version is 120 or later and then follow these steps.
    Search your signatures for <number> Select the results with ID  Choose “Enable Rules” and click OK  
    NetScaler WAF Best Practices
    NetScaler recommends that WAF users always download the latest signature version, enable signature auto-update, and subscribe to receive signature alert notifications. NetScaler will continue to monitor this dynamic situation and provide updates as new mitigations become available.
     Handling false positives
    If app availability is affected by false positives that result from the above mitigation policies, relaxations can be applied. NetScaler recommends the following modifications to the policy.
    Modifications to NetScaler Web App Firewall Policy:
    add policy patset exception_list
    # (Example: bind policy patset exception_list “/exception_url”) 
    Prepend the existing WAF policy with:
    # (Example :  set appfw policy my_WAF_policy q^HTTP.REQ.URL.CONTAINS_ANY(“exception_list”).NOT && <existing rule>^
    NOTE: Any endpoint covered by the exception_list may expose those assets to risks 
    Additional Information
    NetScaler Web App Firewall benefits from a single code base across all its form-factors (physical, virtual, bare-metal, and containers). This signature update applies to all form factors and deployment models of NetScaler Web App Firewall.
    Learn more about NetScaler Web app Firewall, read our alert articles and bot signature articles to learn more about NetScaler WAF signatures, and find out how you can receive signature alert notifications.
    Please join the NetScaler Community today and engage with your peers to learn more about how they are protecting their businesses with NetScaler WAF. 

    Juliano Reckziegel
    Previously, we explored customizing the Citrix Gateway authentication page via JavaScript, offering extensive flexibility, as detailed in a prior article. However, a recent scenario arose where a customer sought minor visual alterations to the login form. To accomplished this we used simple CSS instructions.
    The desired changes were straightforward:
    Relocate the form header to the top of the username input field and make it bold. Adjust the positioning of the "User name:" and "Password:" labels nearer to the input fields. The initial authentication page resembled the left part of the following image, while the right side view reflected the desired modifications:

    To implement this, we created a new RfWebUI theme and edited the initially empty style.css file situated at /var/netscaler/logon/themes/NAME_OF_YOUR_NEW_RfWebUI_THEME/ with the following content:
    @media (min-width: 610px) {
        #login_title {
            margin-left: 186px;
            font-weight: bold;
        .credentialform .plain {
            display: block;
            width: 175px;
            text-align: right;
            height: 44px;
            margin: 0;
    } The @media rule, utilized in media queries, facilitates applying distinct styles for varying media types or devices. This ensures that the browser implements these configurations solely when the screen width is at least 610px, preventing formatting issues on mobile devices.
    The first section targets the element identified by the ID 'login_title', representing the form title. You can verify this through browser developer tools:

    The subsequent section modifies elements containing the CSS classes 'credentialform' and 'plain', as depicted in the following image:

    Remember, NetScaler caches Gateway files within its internal optimization cache, irrespective of whether the IC feature is active or not. To purge cached objects from the NetScaler, execute the following command:
    flush contentGroup loginstaticobjects An important aspect to note is that these customizations endure NetScaler reboots and upgrades.
    This method empowers you to effect minor or significant updates in how this page displays. Consider this as a reference for potential modifications, allowing ample creativity in customization.

    Harihara Sudhan
    Contributed By: Pradeep Kumar M 
    The NetScaler VPX  product is a virtual application delivery controller that can be hosted on a wide variety of virtualization and cloud platforms. Several VPX users encounter disk size constraints for various reasons including the VPX instance upgrade, necessitating a minimum of 6GB free space in the /var partition. 

    This guide offers comprehensive steps to resolve this issue by expanding the disk space to meet higher capacity requirements.
    Note: This is not supported for VPXs on SDX.
    For SDX, Please refer : https://docs.netscaler.com/en-us/citrix-adc/current-release/deploying-vpx/deploy-vpx-faq.html#can-we-add-a-new-hard-drive-to-increase-space-on-netscaler-vpx-instance
    Default Behavior 
    Upon VM creation using the Netscaler build package/template, the default disk space allocated to the the VPX is 20GB. Notably, the /var partition size remains fixed at 14GB, as indicated below.

    Note: Screenshots in this article are taken from the Netscaler VPX running on ESXi Hypervisor.
    Note: Before changing the disk size, it is recommended to take the snapshot of the VM.
    Steps to Increase the Disk space on VPX
    Increase the disk size from Hypervisor or Cloud
    To expand the disk size, begin by shutting down the virtual machine (VM) and then proceed to increase the current disk capacity, elevating it from 20GB to either 30GB or 40GB.  (In the case of Azure, this involves scaling from an existing 32GB to 64GB)
    Jump into the boot prompt
    Power on the VM and boot into the boot prompt. i.e: OK prompt or the boot prompt. This can be achieved by pressing "Ctrl + C" during boot time.

    Enter single user mode
    Run the following command to log into single user mode.
    boot -s
    Note: To enable access in the single-user mode for Azure, it's necessary to configure the com console instead of the default console. This can be done by executing the following command at the ok prompt before initiating the 'boot -s' command.
    set console=“comconsole”   Setup the COM port and connect with Putty.
    Verify the disk space
    Check the newly allocated disk space using the following command.
    gpart show   In this instance, the disk space has been expanded to 30GB, resulting in 10GB of available free space as indicated in the output of the "gpart show" command for da0.

    Also, note the partition name. For this VM it is da0, but for other hypervisor or in cloud it may have different names (For example - AWS has nv0).
    Resize the disk partition
    The partitions can be resized by running below commands.
    Resize the da0 MBR partition to include the 10GB free space by running below command.
    gpart resize -i 1 da0   Incase a permission denied error is encountered while running the above command, run below mentioned command and rerun the previous resize command.
    sysctl kern.geom.debugflags=16    The partition looks like below after successfully running the commands.

    Currently, it is observable that the da0 partition size has expanded to 30GB, with noticeable available free space on the da0s1 partition.
    Merge the free space to the last partition
    Run below command to merge the free space to the last partition on da0s1. i.e da0s1e
    gpart resize -i 5 da0s1   The partition looks like below after successfully running the above command.

    Now, the 10GB free space has added to 5th partition under the da0s1 partition. i.e /var
    Run the below command to extend the filesystem to include newly allocated free space.
    growfs /dev/ada0s1e  
    Now the additionally allocated 10GB is successfully added to /var.
    Verify the increased disk space
    Reboot the instance and verify the new disk space by running the following command in the VPX shell prompt.
    df -h  

    Brian Huhn 2
    5 key NetScaler capabilities you may not be using
    Maximize the value you get from NetScaler with powerful capabilities for application delivery and security
     Imagine a typical day in the IT department: the pressure is always on to ensure consistent operations for your organization's applications and network. In a world where performance and user experience are paramount, having the right tools at your disposal is a must. This is where NetScaler comes in, offering a set of features and capabilities that go beyond the ordinary. From expediting troubleshooting to automatic upgrades to securing applications and APIs, NetScaler elevates your IT infrastructure. 
    “I can’t recommend [NetScaler’s analytics, insights, and security] enough to people that have NetScaler ADCs in their environment” - NetScaler customer in the finance industry
     With NetScaler’s comprehensive monitoring and analytics, you gain valuable insights into network performance, allowing for proactive adjustments and optimizations. Moreover, NetScaler’s security capabilities provide peace of mind in a world where data protection is paramount. 
    5 NetScaler capabilities to leverage
     1. Application dashboard for application usage, health, and performance: The application dashboard provides a holistic view of all your applications that are front-ended by NetScaler. From a single pane of glass, you can see the status of multiple applications at once. You can quickly gauge application usage based on the size of the application icon, get a precise indication of [what] with a comprehensive application score, and quickly spot anomalies with intuitive color coding. With NetScaler you can automatically discover applications from vServers (virtual servers each with their own IP address, certificate and policy set), thus allowing you to define custom applications based on multiple vServers. This flexibility means that you can monitor a single application spread across multiple data centers or clouds or any combination of hosting environments as a single entity. This is important as it allows IT flexibility to deploy where it is most appropriate and migrate components accordingly.2. Web insights: The NetScaler platform simplifies session troubleshooting, making it a breeze for even non-technical users. A case in point is the "network roundtrip" metric, commonly used to evaluate network performance. Prior to leveraging advantageous NetScaler features, networking teams often faced blame for extended response times. However, NetScaler's intuitive graphical interface allows you to dissect the entire journey, from the end-user, across the WAN, to the data center, and finally the application servers. By precisely pinpointing the source of issues, such as lagging times between the user and WAN, they can efficiently liaise with their managed service provider to resolve problems, fostering smoother operations. 
    Furthermore, if you use NetScaler to front-end Citrix Virtual Apps and Desktops (CVAD), the HDX insight feature provides end-to-end visibility into HDX traffic [HDX is a suite of proprietary technologies that deliver a high-definition experience to users of virtual desktops and virtual applications] flowing through NetScaler to CVAD. It equips you with a powerful toolkit, including real-time client and network latency metrics, historical reports, comprehensive performance data, and tools for troubleshooting performance issues. With HDX insight, you can even integrate these findings into your incident response processes, saving significant time for resolving problems.
    3. Gateway insight: The gateway usage function offers precise tracking of connected users. It also proves invaluable for monitoring bandwidth allocation and ensuring optimal functionality. With this granular knowledge, you can proactively size your capacity to maintain optimal application performance.
    NetScaler also empowers you with in-depth tracking capabilities, enabling meticulous monitoring of user activity. You can precisely pinpoint when a user accesses Virtual Desktop Infrastructure (VDI) and even determine the geographic regions of the user's session connections. This information can be critical not only for troubleshooting access failure issues, but also for security concerns, such as impossible travel. 
    4. SSL dashboard: With NetScaler you can define your SSL parameters across your enterprise and have these enforced on each individual NetScaler ADC. This is an invaluable feature and even allows for A+ SSL certification for all your applications with a single click. NetScaler monitors across applications and environments.
    The SSL dashboard provides a holistic view of everything SSL-related in one place. It shows the SSL servers you have running and the protocols that they are using as well as the key strengths utilized. Automated reports enable you to stay vigilant about unused or expiring certificates. Further, the dashboard streamlines your certificate-related tasks, automating installation, updates, deletion, linking, and downloads. With a unified console, you can configure automated policies to ensure compliance with your organization's IT policies.
    5. Automatic upgrades: Keeping your environment secure and using the latest features means upgrading your NetScaler. NetScaler makes this task easy and removes the perennial problems of human error. The automated upgrade process can handle tasks such as manually switching primary and secondary failover ADCs, ensuring their prompt return online, and meticulously verifying the successful completion of all upgrades and more.
    These time-saving measures provide substantial value to both you and your networking team, freeing you up to concentrate on the strategic objectives that drive your business forward.
     Take your business to the next level
    These examples illustrate just a glimpse of the powerful capabilities that NetScaler brings to the table. So, if you're an IT professional or someone managing network infrastructure, it's high time to unlock the full potential of the NetScaler platform. Dive in and explore its treasure trove of features. By doing so, not only can you streamline your tasks and troubleshoot with precision, but you'll also save valuable time and resources. From automatic upgrades that reduce downtime to comprehensive insights that boost productivity, NetScaler is your trusty ally in the world of IT. Start today, and you'll soon wonder how you managed without it. If you’re new to NetScaler, call 1-866-NetScaler to request a demo. If you’re already using NetScaler, reach out to your account manager to see what untapped capabilities you may be missing.

    Priyanka Yadav
    NetScaler ADM has “tasks” for you!
    Hey Admin, you have a task to act on immediately, that is – to explore the tasks feature on ADM!
    NetScaler ADM has been on a journey of continuous operational improvement, and its latest feature, ‘Tasks’, is set to redefine how administrators handle their responsibilities.

    What is ‘Tasks’?
    NetScaler ADM's Tasks feature is designed to provide administrators with a clear overview of critical issues and recommendations, ensuring the efficient management of their infrastructure. Here's a breakdown of what Tasks does.
    Centralized hub for all you ADM tasks: The Tasks feature in NetScaler ADM serves as a central hub for administrators to quickly identify critical issues, take immediate action, and receive proactive recommendations for optimizing the deployment. It ensures that administrators stay on top of potential security risks, application disruption, consistent configurations, compliance and continuously improve the efficiency and performance of their NetScaler infrastructure. Tasks are categorized into two sets:
    Actionable Tasks (Tasks): NetScaler ADM's Actionable Tasks ensures administrators can swiftly address pressing issues to uphold the health, security and compliance of their NetScaler instances. Actionable Tasks keep a vigilant eye on ongoing NetScaler issues, providing administrators with instant visibility into critical matters.
    Expired SSL Certificates: Provides information about expired SSL certificates. This Task provides you information about deleting unused certificates and updating expired certificates.
    Security Advisory: Immediate insights into NetScaler instances affected by any CVEs. Tasks gives information on the detected CVEs, impacted instances, and recommends remediation steps. Redirects to the Security Advisory page for taking the action.
    Expiring SSL Certificates: Monitors SSL certificates approaching expiration and helps you take actions.
    Config Drifts:  Provides information about configuration deviations in NetScaler instances. Tasks identifies instances with unsaved configurations and template deviations. Streamlines actions to save configurations and run correct commands for alignment.
    Recommendations: Optimizing network performance and efficiency requires strategic guidance. Enter Recommendations – NetScaler ADM's proactive advisor, guiding administrators on best practices, utilizing their entitlement effectively and getting a quick view of features they can use in the ADM to manage the deployment efficiently to achieve an optimal deployment state.
    Guide Me Option: Tasks are not just about helping you with visibility; they are designed to be your allies, making your job smoother and more efficient. The ‘Guide Me’ workflow for recommendations handholds you to perform any task. You do not know how to assign an analytics license, no worries, Guide Me will take you step by step into performing that action.
    Get notified of any new Tasks - You can configure and get notifications whenever NetScaler ADM identifies any open tasks that require your immediate action.This makes sure you are not missing out on anything. 
    Check out the capabilities of tasks here.
    In summary, NetScaler ADM's Tasks feature acts as a comprehensive solution for administrators, offering insights, actionable tasks, and proactive recommendations to ensure the optimal performance, security, and compliance of NetScaler instances and applications.
    Looks like you have a task to explore this new feature 😉

    Magnus Esse
    This document aims to look at a few different ways to write policy expressions to match on the hostname the client uses in the HTTP request and point out why the expression frequently used might not be the best option to use.
    Hostname Header
    I have created a simple website that can be accessed through a web browser by accessing the following address: http://testweb.domain.local:8080/page.html 
    To clearly show the differences in result for different policy expressions, I have elected to run the service on a non-standard port.
    If I open development tools in my browser and look at the headers, this is what I can see:

    Looking at the above will show us that the hostname header looks like this:
    Host: testweb.domain.local:8080    
    Policy expressions
    Let's look at some of the more common policy expressions I see being used to match against the hostname the user entered and how they will work. The main goal every time you write a policy expression is to match as exact as possible to not allow or block traffic unintentionally.
    Expression 1
    This expression will match all requests but doesn't help much if we want to make a logical decision based on the incoming request. The policy expression is useful in test environments if you want to make sure to get a match but it should be avoided in production environments unless the intention is to actually capture all traffic.
    Expression 2
    This expression will match all traffic which has a valid HTTP request formatting but doesn't help much if we want to make a logical decision based on the incoming request. The policy expression is useful if you want to allow all valid HTTP requests, but it should be avoided in production environments unless the intention is to capture all valid HTTP requests.
    Expression 3
    This expression will not match the request as you can see the port number has been appended in the end of the hostname, “:8080”. If you are using the standard port number 80 or 443 for your web service most browsers will not include the port number in the hostname header, but some clients might do it and therefore the result might differ based on which client is accessing the service. My recommendation is to avoid this expression unless you know it will for sure match your traffic correctly.
    Expression 4
    I have seen this used frequently when it has been discovered that EQ() wouldn't work. This expression will match our hostname in this case and it is a decent expression, however it will allow for something to be added in the end of the hostname and the request will be forwarded to the web server and depending on the web server it might or might not work. I would try to avoid using this expression when matching against hostnames if the intention is to match a single hostname.
    Expression 5
    This expression will match as we have used the SERVER statement together with EQ(). What the SERVER statement does is it will look at the hostname and filter out only the FQDN part of the hostname and then EQ() will match against this, the port number will not be evaluated. This is the policy expression to be used if you want to match against exactly the hostname for the web service.
    Case Sensitivity
    During all my years writing policy expressions I have never encountered any issues with case sensitivity when writing policies matching against the hostname header which makes me believe they are not case sensitive, All clients I have been able to test with when writing this has converted the hostname to lowercase before sending the request, and if I use uppercase in the NetScaler policy expression for the hostname it will still work. I have always written hostnames in lowercase as I see this as a good practice to do. Case sensitivity is important to have in mind when writing other policy expressions though, but can be ignored when looking at the hostname header.
    Always strive to match your policy expressions as exactly as possible.
    When matching based on the hostname header I suggest using HTTP.REQ.HOSTNAME.SERVER.EQ() if the intention is to match the exact FQDN that is used.

    Phani Bandaru
    Connection draining is an important feature of autoscaling support that handles client connection termination gracefully when a scale-in happens. 
    Google Cloud Platform(GCP) does not expose the required life cycle hooks and APIs for scale-in control to the third party load balancers like NetScaler to implement connection draining in the GCP environment.
    This article discusses a potential solution to achieve Connection draining with NetScaler on Google Cloud Platform.
    Backend autoscaling in NetScaler
    Backend auto scaling solution using NetScaler comprises a server farm. This server farm is a managed service offered by Google (cloud) and is called a managed instance group(MIG).
    Every MIG is associated with a scaling policy that the customer defines and is used by the cloud in determining scaling decisions to provide elasticity to the group. Typical parameters that determine the load are CPU utilization and throughput.  
    This MIG is front ended by the NetScaler which acts as a load balancer. A scale-in operation involves shutting down one or more servers from the farm when perceived load is below the threshold levels and scale-out involves adding one or more servers to the farm when load is above the threshold levels.
    NetScaler today periodically polls the backend server farm to get an update on whether there is a change in the server farm and makes changes to the local load balancing configuration based on the latest state of the servers it queries. 

    /applications/core/interface/js/spacer.png" data-src="/monthly_2024_01/image.thumb.jpg.789b20da2adf29ca00fdfc0c3b198e5d.jpg" data-ratio="50.4" width="1000" class="ipsImage ipsImage_thumbnailed" alt="image.jpg">
    What is connection draining?
    Connection draining is a process that ensures that existing and in-progress requests to a server are given time to complete when the policy engine decides to remove the server from the server farm. 
    This is typically accomplished by delaying the shutdown of the server by various means so that existing connections being served by the server are completed.
    The other aspect of connection draining is that new connections which get hashed to the server being taken away need to be hashed to a new server. 
    Problem with scale-in operation on GCP 
    When a server in the backend pool is being shutdown as part of a scale-in decision, NetScaler does not get to know until the server is actually shut down. This is because the scale in and scale out operations are completely managed and executed by Google. 
    As a result all the existing sessions that are currently being handled by that particular server can be terminated abruptly resulting in client server connections being reset. Some applications (like gaming, finance) are sensitive to such loss and require connection draining so that connection terminations are handled gracefully.
    Ideally if GCP provides life cycle hooks so that in the event of a scale in operation, NetScaler gets to know before the server is being shutdown and GCP waits for NetScaler to give a go ahead we can solve this problem by introducing a pre configured or pre determined wait time. 
    Unfortunately GCP does not provide any such hooks to achieve graceful termination. Though GCP internally implements connection draining for native load balancers.
    Delayed shutdown of the application server
    One way is to introduce some delay into the shutdown process of the applications server so that NetScaler gets some time to absorb the change and redirect new connections to a different server while server gets extra time to server the connections. We can introduce a delay in the shutdown of an instance by adding custom code in the modules that manage the startup and shutdown of the system, for example Systemd on Linux. Systemd units can be modified to add delays and alter shutdown/startup sequence of the systems. However, we need to make sure that the appropriate backend process (i.e. nginx, apache, etc.) keeps running during the period where the connections continue to be served. Otherwise connections will be reset and delaying shutdown does not serve the purpose.  Note: In the example below, the Apache HTTP server, controlled by the httpd service systemd unit file, is used as an example. Please adjust the solution accordingly if using a different service.
    There are two ways we can achieve this.Method 1
    The systemd Unit corresponding to the httpd service (typically apache.service) can be modified to add a ExecStop directive to add a delay up to 120 sec.
    ExecStop directive executes a command that follows it. If there is a sleep command, it would be executed prior to stopping the apache service. Please note that positioning of the directive is important here. The directives are executed in order.
    Please see the sample code below. 
    [unit]Description=The Apache HTTP ServerAfter=network.target remote-fs.target nss-lookup.targetDocumentation=https://httpd.apache.org/docs/2.4/[service]Type=forkingEnvironment=APACHE_STARTED_BY_SYSTEMD=trueExecStart=/usr/sbin/apachectl start+ExecStop=/bin/sleep 120ExecStop=/usr/sbin/apachectl stopExecReload=/usr/sbin/apachectl gracefulPrivateTmp=trueRestart=on-abort[install]WantedBy=multi-user.target 
    Method 2
    We can have a shutdown script defined as part of the metadata section of the GCP VM console. These shutdown scripts are plugged into the shutdown process of the server. However, GCP plugs in the shutdown script defined as part of the metadata after the http service in the shutdown sequence.
    As a result, http service gets shut down and the shutdown script will be executed. To mitigate this problem,  the shutdown sequence of the services/Units needs to be altered so that shutdown scripts are executed before the http service. 
    A sample shutdown script and code are as shown below. 

    /applications/core/interface/js/spacer.png" data-src="/monthly_2023_12/image.jpg.1045d2f0ea480165be32d1ed699bbff9.jpg" data-ratio="38.5" width="748" class="ipsImage ipsImage_thumbnailed" alt="image.jpg">
    [unit]Description=The Apache HTTP ServerAfter=network.target remote-fs.target nss-lookup.targetBefore=google-shutdown-scripts.serviceDocumentation=https://httpd.apache.org/docs/2.4/         [service] Type=forkingEnvironment=APACHE_STARTED_BY_SYSTEMD=trueExecStart=/usr/sbin/apachectl startExecStop=/usr/sbin/apachectl stopExecReload=/usr/sbin/apachectl gracefulPrivateTmp=trueRestart=on-abort    [install]WantedBy=multi-user.target
    Though small, both the above approaches require altering scripts/configuration of the image which is typically in the administrative domain of the customer. With both these approaches, shutdown could be extended up to 120 sec.
    Please note that true connection draining may still not be achieved within this time window, in case of a workload that has longer lived connections. 
    For full connection draining support, NetScaler needs to have full control on the shutdown timing of the backend servers. However the current approach can be used to prevent abrupt termination of short lived connections. 

  • Create New...