Andrew Scott

Why? What have I been smoking? Where can I get some? What's with the service dog? Hold on…. It is not like that. Honest. The dog analogy, service dogs help you see. You have been selected for a training course in Citrix ADM, the goal is to provide you with enough information to be actually dangerous when talking to a customer or client. 30 days is a bit of an arbitrary number, but I am prepared to give you 2 minutes of material, can I get 2 minutes of your time? Today is all about Fleet management. What is ADM? Woof woof!Citrix has a management platform that works in conjunction with its NetScaler ADC, it offers many insights into what the NetScaler sees. This is a good thing as knowledge is power after all. Architecturally it looks like this. Looking at the detailed diagram above, I have one NetScaler, so the term ‘Fleet Management’ does look like a bit of a stretch. That is true, however, the idea is that once you have the tool to manage one appliance it scales up to manage many. The users have their connections handled by the NetScaler, which then ensures that they are directed to the best web server for their session/service etc. In simple terms. it's good to know what the status of your infrastructure is before your phone starts to ring. What does it look like when there is more than one NetScaler to look at? A view across your estate... You can have a table or the view above. There is then an option to drill into anything that jumps out and see more details on why it got flagged. For example, here is one: Show me.Here is a walk-through from Sanyukta. Summary.Fleet management is a really handy superpower, you want the user connections to work without issue and allow you to sell widgets by the boatload. ADM can tell when your NetScaler is having issues and then offers some pointers on what you can do to fix it. BTW, this capability is free. What’s not to like?

Hello! I have found that Internally Citrix has some nice reports to show builds and which is the latest and what might be included in the new release. Of course, it would be handy to have that delivered to my mailbox. You can get that! Subscribe to the NetScaler Times, click on the link.... https://andrew808.substack.com/p/netscaler-times-newsletter-for-week-444 A curated list of builds and useful links will be delivered to your mailbox every week. Some are links to the material here, but there is also material posted elsewhere. Let me know what would be handy and I would be happy to add it. Have a better one Andrew

IntroductionFrom time to time the NetScaler gets new features. Wow, such a hook! Ok, stay with me here. Things get added to the capability of the appliance and a lot of the time you have no context as to why that was added. Would it not be cool to have a write-up explaining what the problem was that needed fixing along with the use case? Of course, it would. The context would then offer some insight into how you could make use of that new option, this would then allow you to take another look at something that you might have been struggling with. In this case, I would like to credit Michal Grabczyk and Lakshmi Prasanna Guru for getting this added! What is TROFS?Do you know what TROFS is? I had to look it up, as it had been a while. TROFS stands for Transition to out of service, it is an option in the load balancing module to gracefully stop any new connections to a service that will be taken out of action for whatever reason. TROFS specifically allows an admin to have the monitor probe scan the web server for a specific, configurable response code. Why would you do that? The benefit of this is that the service status can be handed off to the web server team, they can set the TROFS code that then controls when a service is in the required state. They don’t need to raise a service ticket for whoever manages the NetScaler to do ‘something’. As they will know when patches or maintenance are needed on the web server itself. They just set the code, and the monitor sees it and stops sending new connections. It is like they have a remote control for the NetScaler! What is the problem with TROFS prior to 13.1.37.38?In this case, the customer was using this kind of setup. The MPX was providing a front end for a Web tier that runs in a CDN. The details of the CDN are not significant. The customer was seeing intermittent 500 error responses from the NetScaler “Internal Server Error 43531”. The NetScaler was set up as follows: SSL Vserver Autoscaling Service group using DNS.The CDN ensures that there are always 4 IP addresses returned from DNS, these addresses change pretty frequently (TTL set to 60s), and they all change at the same time. The main problem is the fact that all 4 services are marked as TROFS at the time when the new IPs are learned. These new addresses need to respond to a monitor before being used. This way there is a small period of time when all services are in TROFS and the error is returned.In this case, the CDN was not able to change the behaviour as the sets of IPs are returned from the closest POP and those IPs are rotated for many reasons. However, when the IP address changes, it does not mean that the content is immediately unavailable under the old IP. Using those IP numbers is not desired and the delivery of the content is not guaranteed, it really is suboptimal. There is no way to “stabilize”/control the ip numbers returned by the CDN. The solution in the new buildA new configurable option to delay putting old IPs in TROFS state for a period of monitor timeout, which will address the delay in new IP coming up was added to the firmware.*• add servicegroup sg1 tcp ? o -autoScale <autoScale> -memberPort <port> [-autoDisablegraceful ( YES | NO )] [-autoDisabledelay <secs>] [-autoTrofsGraceful ( YES | NO )] • set servicegroup sg1 -autotrofsGraceful yes* Internally every time a monitor is bound to the service group, the appliance stores the highest response time outs of the monitors in the servicegroup. Depending upon whether TROFS graceful is enabled/disabled, the corresponding movement from UP-TROFS will be delayed by the max resptimeout/done immediately. SummaryIn simple terms, this build adds some flexibility for the NetScaler to take account of changes in the infrastructure that it is fronting, in this case, that is hosted in a CDN. Always handy to have some flexibility! NetScaler rocks.

Introduction The following sections talk about a use case for load balancing something simple, UDP. When first looking at this problem, it was not obvious to me quite what the issue was. As I simply thought that setting up a load-balancing virtual server for UDP would be trivial and would do the job. The problem was that it did not behave as the customer wanted. Also, finding a simple fix was not obvious! Problem statement: What is Syslog and why is it important? This Syslog problem was raised by the head of IT security, he considered Syslog to be very important for him as it provided a way to track what activity his devices were seeing on the network. He stated that in his security role he considered Syslog as mission-critical. As part of the deployment, the customer used NetScaler to load balance the Syslog source devices onto the array of Syslog servers, which he could then query. The network looked like this diagram below. It should be noted that the diagram looks to show a two-arm deployment. It is only shown as it is to keep it simple. The appliance was deployed as a one-arm deployment. Device1, 2,3 and 4 on the left-hand side are all Syslog source systems, they all send their entries to a VIP defined on the NetScaler which then load balances the traffic onto two or more Syslog servers. The customer raised a question about how NetScaler actually load balances UDP-based Syslog traffic. His NetScaler setup was resulting in uneven load balancing on the Syslog servers in some cases. This was unacceptable to the customer.He had also seen packet fragmentation in other configurations which then resulted in Syslog traffic being damaged, so losing live data. Another unacceptable situation. This packet fragmentation happened when the MTU was exceeded.The simple statement the client raised was.We need to make it clear that we are performing UDP load balancing and not SYSLOG load balancing. We could be load-balancing SYSLOG, DNS, or NTP traffic, which all make use of UDP traffic. Naturally, this point about UDP is significant, as the traffic is ‘lightweight’ and lacks a lot of the options for traffic control that are present with TCP. Impact: As stated, Syslog is important in this use case for this client. Having a good way to evenly load balance the traffic will ensure that Syslog does not overload any one Syslog server What have they tried?Using session less load-balancing. This first option is to configure the NetScaler to use Session Less load balancing. This re-evaluates the load balancing decision individually for each UDP packet, which will ensure that each request gets load balanced based on the selected load balancing method, regardless of the Source IP/Source Port/Destination Port tuple. Reference https://docs.citrix.com/en-us/citrix-adc/current-release/load-balancing/load-balancing-manage-clienttraffic/sessionless-lb-vserver.htmlResult: It doesn’t work as expected, as fragmented packets resulted in lost data. There are also a number of issues with session less, one of which required that the appliance is deployed in two-arm mode. This alone was a significant piece of work if undertaken by the client due to the number of appliances that the customer had. Using persistence Option two was to look to use session persistence to ‘stick’ a network device to a particular Syslog server. The problem with this approach was that it resulted in this. The observation from the customer was that the NetScaler was not doing much actual load balancing.Why not just use Syslog over TCP? This was not an option for this client. What is the solution?All credit to my colleague Vemula who had spotted an option that could be used to help in this use case. When setting up the NetScaler configurations there is an ‘order’ option when the services are bound to the load-balancing virtual server. This was a feature that got added a while ago and so wasn’t something that I was familiar with. It uses the order feature in combination with load-balancing priority policies to distribute traffic evenly across backend Syslog servers. add service svc_exapps1 EXAPPS1 UDP 514 -gslb NONE -maxClient 0 -maxReq 0 -cip ENABLED client-ip -usip YES -useproxyport NO -sp OFF -cltTimeout 120 -svrTimeout 120 -CKA YES -TCPB NO -CMP NO add service svc_exapps2 EXAPPS2 UDP 514 -gslb NONE -maxClient 0 -maxReq 0 -cip ENABLED client-ip -usip YES -useproxyport NO -sp OFF -cltTimeout 120 -svrTimeout 120 -CKA YES -TCPB NO -CMP NO add service svc_exapps3 exapps3 UDP 514 -gslb NONE -maxClient 0 -maxReq 0 -cip ENABLED client-ip -usip YES -useproxyport NO -sp OFF -cltTimeout 120 -svrTimeout 120 -CKA YES -TCPB NO -CMP NO add service svc_exapps4 exapps4 UDP 514 -gslb NONE -maxClient 0 -maxReq 0 -cip ENABLED client-ip -usip YES -useproxyport NO -sp OFF -cltTimeout 120 -svrTimeout 120 -CKA YES -TCPB NO -CMP NO add lb vserver vip_logme_514_udp UDP 10.10.118.215 514 -persistenceType NONE -lbMethod ROUNDROBIN -backupLBMethod LEASTPACKETS -sessionless ENABLED -trofsPersistence DISABLED -cltTimeout 120 bind lb vserver vip_logme_514_udp svc_exapps1 -order 1 bind lb vserver vip_logme_514_udp svc_exapps2 -order 2 bind lb vserver vip_logme_514_udp svc_exapps3 -order 3 bind lb vserver vip_logme_514_udp svc_exapps4 -order 4 add lb action syslog_act1 -type SELECTIONORDER -value 1 2 3 4 add lb action syslog_act2 -type SELECTIONORDER -value 2 3 4 1 add lb action syslog_act3 -type SELECTIONORDER -value 3 4 1 2 add lb action syslog_act4 -type SELECTIONORDER -value 4 1 2 3 add lb policy syslog_pol1 -rule "client.ip.identification.mod(4).eq(0)" -action syslog_act1 add lb policy syslog_pol2 -rule "client.ip.identification.mod(4).eq(1)" -action syslog_act2 add lb policy syslog_pol3 -rule "client.ip.identification.mod(4).eq(2)" -action syslog_act3 add lb policy syslog_pol4 -rule "client.ip.identification.mod(4).eq(3)" -action syslog_act4 bind lb vserver vip_logme_514_udp -policyName syslog_pol1 -priority 10 -gotoPriorityExpression END -type REQUEST bind lb vserver vip_logme_514_udp -policyName syslog_pol2 -priority 11 -gotoPriorityExpression END -type REQUEST bind lb vserver vip_logme_514_udp -policyName syslog_pol3 -priority 12 -gotoPriorityExpression END -type REQUEST bind lb vserver vip_logme_514_udp -policyName syslog_pol4 -priority 13 -gotoPriorityExpression END -type REQUESTOutcomeThis resulted in no infrastructure changes to get this working, although the client was running an older version of firmware and did need to update to get the order option available to them. The details of the firmware change are that it will require a version later than 13.1.12 . The resulting setup provided a load-balancing setup that worked seamlessly with UDP traffic.