Raghav S N

Secure your DNS infrastructure with NetScaler ADC Submitted February 25, 2021 Author: Raghav SN DNS resolution is generally the first step in securing access to applications, and a DNS cache is a key asset that can be exposed to a variety of attacks. If a DNS system is compromised, attackers can hijack transactions and impact DNS resources and your network infrastructure. In this blog post, we’ll look at how NetScaler ADC can help you to protect your DNS infrastructure against some well-known DNS vulnerabilities. Cache Poisoning To reduce the chances of an attacker successfully spoofing the response, DNS clients are expected to randomize the DNS transaction ID or query ID and the ephemeral source port used to send the DNS request. This requires the attacker to guess correctly a 32-bit integer quickly. But if a DNS client sends multiple DNS requests to the nameserver on the same UDP 4 tuple, the attacker only has to guess a 16-bit query ID (because the UDP source ephemeral port is constant) to succeed in spoofing the response. Restrict the number of queries on the same UDP 4 tuple. Once the configured threshold is reached, NetScaler ADC starts dropping the requests coming to that UDP 4 tuple. You can tune this limit on your NetScaler ADC by using this command: set dns parameter -maxPipeline <positive-integer> Please note, setting the parameter to zero will imply there is no limit to the number of pipelined requests, so you should set the limit to an appropriate non-zero number. You can also enable DNSSEC for the zone hosted on the NetScaler ADC to mitigate cache poisoning. Amplification Attacks In a DNS amplification attack, the attacker uses small DNS requests to exhaust DNS server resources. This uses the DNS server’s compute resources, memory, and bandwidth and denies DNS services for legitimate application access. There are different ways to achieve DNS response amplification. One way to protect DNS infrastructure against amplification attacks is to whitelist or blacklist client IPs. If IP-based mitigation isn’t feasible, you can identify the attack pattern, configure the appropriate DNS policies using NetScaler ADC’s policy expressions, and force the clients to reinitiate the query over TCP. Additionally, you can configure DNS endpoints on an anycast IP to protect against DDoS attacks. Root Referral Amplification Attack DNS authoritative servers are recommended to return root NS records in the authority section as a response to queries requesting information about a domain name that they are not authoritative for. Root NS records’ size, including the glue records, will come close to 512 bytes. This can give at least five times the amplification for an average DNS query. An attacker can use this DNS specification by sending queries to an authoritative nameserver for a domain outside of the zone for which the nameserver is authoritative. You can mitigate this on NetScaler ADC with the command below. When you disable the configuration parameter, NetScaler ADC will not respond to queries for domains outside the zones for which the ADC is configured to be authoritative. set dns parameter -dnsRootReferral DISABLED ANY Query-Based Amplification Attack An ANY query type returns all records for a given domain name. That means the response size is huge despite the query size being comparable to other queries. Attackers use ANY queries to perform DDoS attacks on vulnerable network infrastructure. Use the command below in your NetScaler ADC to configure a rate-limiting DNS policy, with a rate-limiting identifier as the DNS query type, and restrict ANY queries to a threshold the network can handle. add stream selector dns_anyquery_amplification “dns.req.question.type.eq(ANY)”add ns limitIdentifierdns_ampl_attack_limitid -threshold 1000 -timeSlice 10000 -selectorNamedns_anyquery_amplificationadd dns policy dns_amp_attack_pol “sys.check_limit(\”dns_ampl_attack_limitid\”)”dns_default_act_Dropbind dns global dns_amp_attack_pol -priority 10add ns variable dns_tcp_requests -type ulong -scope transaction NXDOMAIN-Based Amplification Attack DNS bots can inject malicious queries for non-existent domain names. These domain names are often randomly generated. Crafting an NXDOMAIN response can be more intensive than serving a configured record for authoritative nameservers. NXDOMAIN responses in the case of DNSSEC-enabled zones can be even more resource-consuming. One common approach to mitigating NXDOMAIN-based attacks is to identify the domain pattern and drop such queries or force the client to come over TCP by sending a truncated response. Forcing the client to come over TCP is one way of ascertaining whether DNS clients are genuine. But malicious DNS bots have become more sophisticated with full TCP/IP stacks at their disposal and can respond by reinitiating the query over TCP. Ideally, you would identify the pattern in domain names and configure suitable policies to drop such queries. NetScaler ADC can function as an authoritative nameserver for a DNS zone, and you can mitigate NXDOMAIN-based amplification attacks using this configuration option: set dns parameter -NXDomainRateLimitThreshold <positive_integer> If your NetScaler ADC is configured as a DNS proxy, NXDOMAIN-leading queries will generally result in a cache miss. The back-end server will be swarmed with more queries than it can handle and will deny the service for genuine queries. When the NetScaler ADC receives the NXDOMAIN response, it is cached, denying cache resources for legitimate configured records. You can protect the DNS cache against NXDOMAIN attacks by allocating only a small portion of memory (MBs) for negative record caching and by setting the TTL of negative records to a low value (preferably a maximum 1 or 2 minutes) using this configuration set dns parameter -maxNegativeCacheSize 200 -maxnegcacheTTL 120 Please note, in the above sample configuration, maxNegativeCache size is given in megabytes, and maxnegcacheTTL is in seconds. DNS Tunneling Attackers can use a DNS tunnel for command and control, data exfiltration, or tunneling of any internet protocol (IP) traffic. A DNS tunnel enables the overloading of the DNS, denying the DNS resources needed to support legitimate application access. DNS tunnel detection techniques such as payload analysis and traffic analysis can help. With payload analysis, the DNS payload for one or more requests and responses is analyzed for tunnel indicators. With traffic analysis, the traffic volume and pattern are analyzed over time. Count, frequency, and other request attributes can also be considered. You can implement detection based on payload and traffic using NetScaler ADC’s rich DNS policy infrastructure. If NetScaler ADC is front-ending a farm of LDNS servers, you should apply mitigation for DNS tunneling so resources are used for only legitimate DNS use cases. The mitigation below is based on the assumption that the average length of domain names would be less than 200 bytes and DNS query payload size in legitimate use cases would be less than 256 bytes. add dns policy dns_tunnel_payload_1 “dns.req.question.domain.length.gt(200)”dns_default_act_Dropadd dns policy dns_tunnel_payload_2 “dns.length.gt(256)” dns_default_act_Dropbind dns global dns_tunnel_payload_1 20 -gotoPriorityExpression next -type REQ_DEFAULTbind dns global dns_tunnel_payload_2 30 -gotoPriorityExpression next -type REQ_DEFAULT TCP Slowloris Attack An attacker can send a DNS request in chunks, and the DNS stack would wait for the complete query to come for an idle timeout period of time. However, the attacker may not send the complete data or send chunks of data with a huge delay between each chunk. This could lead to denying service to legitimate queries (by consuming the memory on nameserver to maintain the TCP connection for idle period of time and maintaining multiple query chunks). You can mitigate this attack with NetScaler ADC by using the following configuration: set dns parameter -splitPktQueryProcessing DROP Learn More For most well-known attacks, mitigation is available in DNS parameters. For some, you can leverage NetScaler ADC’s rich policy infrastructure to inspect DNS requests and responses for anomalies and can take appropriate action to protect your DNS resources and, more generally, your network infrastructure. NetScaler ADC’s policy infrastructure is an extensible framework that IT admins can leverage to implement rules for newer vulnerabilities and immediately protect your DNS infrastructure in a security environment that’s constantly changing. To learn more about DNS policies, check out our documentation on responder policies for DNS and rewrite policies for DNS.

Submitted March 18, 2021 Author: Raghav SN One common global server load balancing (GSLB) issue we see often in the field is improper distribution of client traffic across data centers. This affects the end user experience by increasing latency while accessing an application, which can have an impact on the productivity of the end user. In most of these reported cases, it turns out to be an expected behavior due to one or more run time states. Rarely does it require a code fix. In this blog post, we’ll look at why improper distribution in GSLB happens and how you can avoid it. Review the State and Metastate of the GSLB Service The first step in analyzing improper distribution of GSLB traffic is to review the state of the GSLB services bound to the application-specific GSLB virtual server, as well as the meta-state. This will give you insights into the overall health of the application endpoint. You might run across scenarios where the state of a GSLB service shows as “Up,” even though it might not be in a state to serve the client traffic. This can happen for a number of reasons, including: The virtual server (load balancing or content switching or VPN virtual server) hosted on the Citrix ADC appliance represented by the GSLB service has reached its configured operating threshold (the soThreshold setting on the virtual server) or all the back-end servers bound to the virtual server have reached their configured operating threshold (maxClient or maxBandwidth setting on service/servicegroup entity in Citrix ADC). In case of a VPN deployment, the VPN virtual server has reached its maximum AAA user limit. All back-end servers bound to the virtual server are down for maintenance. If the virtual servers (LB or CS) are down and are configured to return a redirect URL. You can review the metastate of a GSLB service using the ‘show gslb service <name>’ command, as shown below: show gslb service gslb_svc1 gslb_svc1 (10.102.217.57: 80)- HTTP Server: 10.102.217.57 State: UP Monitoring Owner: 0 Last state change was at Tue Feb 6 13:58:25 2018 Time since last state change: 0 days, 00:00:09.230 Entity not participating in GSLB because: One or more services are in the transition out of service (TROFS) state. Idle timeout: Client: 180 sec Server: 360 sec Effective State: DOWN Max Conn: 0 Max Bandwidth: 0 kbits PublicIP: 10.102.217.57 PublicPort: 80 Type: LOCAL Site Name: siteA Review the Monitor Setting The next step is to review and correct the monitor settings. One of the main reasons for inconsistency between state and metastate is that the monitor bound to the GSLB service is of TCP type, which fails to monitor application health. The existence of TCP connectivity doesn’t indicate the absolute health of the application, so it’s important to bind application-aware monitors like HTTP monitors with appropriate request and response settings. For the scenarios listed above, HTTP monitors would fail, giving the admin a “503 Service Unavailable” error response, and the service would be marked as “Down”. GSLB Method and Persistence Setting If GSLB services are healthy, the next step is to review the GSLB method setting. If the configured GSLB method is round trip time (RTT), review the following conditions: If the GSLB appliance is getting the request from an LDNS IP for the first time, RTT values will not be available. The backup GSLB method would be applied to give out the IP in the response. Based on the backup GSLB method setting, the IP returned might not match the datacenter IP expected. You should: Configure static proximity as the backup method because its decision would be closer to that of RTT method If you always have to return an IP based on the calculated RTT, you can choose to drop all the requests from an LDNS IP until the RTT is calculated by enabling this configuration parameter : set gslb parameter -dropLdnsReq ENABLED If persistence is configured and the backup GSLB method is used to return the application endpoint for a client, the same IP would be returned even for subsequent requests from the same LDNS. You can interpret this as the GSLB wrongly routing the client to a suboptimal location. You can track GSLB decisions made using the backup GSLB method by using “Primary LB method Failures” counter reported under ‘stat gslb vserver <name>’ command (highlighted in below command output). If both the primary and backup GSLB method fail, GSLB decision will be made based on round robin method. This condition can be tracked using “Backup LB method failure counter, as highlighted in below command output. stat gslb vserver gv1 Gslb Vserver Summary Protocol State Health actSvcs inactSvc gv1 HTTP UP 86 6 1 VServer Stats: Rate (/s) Total Vserver hits 0 0 Primary LB Method Failures — 0 Backup LB Method Failures — 0 In case of the GSLB method being set to static proximity or to RTT, if there is more than one datacenter that is equidistant from the client, the datacenter IPs would be returned in response to clients in a round-robin fashion. The same IP might not be returned for subsequent requests, and a client might be routed to a datacenter that is different from the first time. An application might break in this case, unless you have site persistence configured. In the case of the static proximity GSLB method, you should review the accuracy of the IP database loaded. In some cases, a client IP address range might be missing or will be associated with a location that’s different from our assumption. These anomalies in IP database can alter the GSLB decision. If the client IP address range is missing, the backup GSLB method will be used to give out a DNS response. If the client IP address location is represented differently in the IP database, an optimal GSLB service would be chosen based on the location in the database. DNS Policy Setting DNS policies with actions of type “DNS view” and “preferred location” can influence your GSLB decisions. You should review your DNS policies to ensure you understand the GSLB distribution. Based on the DNS policy evaluation, if a DNS view is selected, the GSLB services bound with that view are the only ones considered for your GSLB decision. You need to make sure that DNS views are associated with all relevant GSLB services. Based on the DNS policy evaluation, if a list of preferred locations is selected, the configured GSLB method would be overridden, and your GSLB decision would be made based on the preference configured in the DNS policies. Conclusion Incorporating the above recommendations will help in the optimal use of your resources spread across multiple data centers and increase the productivity of your application user by helping you to deliver the best possible user experience. Learn more about NetScaler ADC’s GSLB solution.