Jump to content
Welcome to our new Citrix community!

bandwidth exceeds licensed bandwidth


Natalya Shimko

Recommended Posts

Hi, There is a citrix netscaler vpx with a maximum bandwidth license of 200 megabits / second, used as a web application firewall. Recently, throughput on it reaches 210-215 mb / s, of course everything hangs. I unloaded it, that is, I started up all resources bypassing Сitrix, and the throughput still hangs at 209 and does not decrease. Later it decreased slightly, it was 50-60 mb/s. I saw nothing in the logs what could be the reason? What can load it like that?

Link to comment
Share on other sites

Hi nshimko,

everytime the throughput exceeds 200MBit/s, packets are dropped. This event is logged and you can check it with this command:

 

1. shell

2. cd /var/nslog

3. nsconmsg -K newnslog -g nic_err_rl_pkt_drops -d past -s disptime=1 | more

 

What do you exactly mean with "everything hangs"? When packets are dropped, TCP traffic needs to be retransmitted. If packet drops are not continuously logged and the Delta counter is not too high and decreases with every new logged packet drops and no new drops are logged again, the performance is slow but does not hang. It will only hang when packet drops are  continuously logged and the Delta counter does not decrease. What about your CPU and Memory consumption?

 

If you want to see the throughput of every Virtual Server at the moment, you can use this command. In this case it only shows Virtual Servers with a minimum of 1250000 Byte/s, which equals to 10Mbit/s:

 

1. shell

2. cd /var/nslog

3. nsconmsg -s ConLb=2 -d current -s ratecount=1250000 | grep -E "si_tot_ResponseBytes|si_tot_RequestBytes" | grep -v -E "svcgrp|cs_|internal|cons_si"

 

Next step could be to log all requests to this Virtual Server to see if there is one special client who is responsible for this throughput.

 

Best regards,

Jens

 

Link to comment
Share on other sites

command output nsconmsg -K newnslog -g nic_err_rl_pkt_drops -d past -s disptime=1 | more:

reltime:mili second between two records Mon Sep  9 17:18:15 2019
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
     27   73675      490802175       8065      965 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:18:15 2019
     28   64271      490810783       8608     1212 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:19:19 2019
     29   72411      490819040       8257     1162 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:20:32 2019
     30   78220      490827500       8460     1193 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:21:50 2019
     31   64043      490835480       7980     1119 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:22:54 2019
     32   85855      490843338       7858     1103 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:24:20 2019
     33   71474      490851875       8537     1205 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:25:31 2019
     34   71772      490860453       8578     1194 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:26:43 2019
     35   71224      490869040       8587     1210 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:27:54 2019
     36   71026      490877357       8317     1171 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:29:05 2019
     37   71155      490886016       8659     1210 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:30:17 2019
     38   78030      490894388       8372     1179 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:31:35 2019
     39   71718      490902288       7900     1114 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:32:46 2019
     40   71194      490911152       8864     1231 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:33:58 2019
     41   85911      490918977       7825     1086 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:35:23 2019
     42   79238      490928507       9530     1316 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:36:43 2019
     43  100513      490936284       7777     1097 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:38:23 2019
     44   71012      490944614       8330     1169 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:39:34 2019
     45   99552      490953517       8903     1248 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 17:41:14 2019
     46 1299741      490955754       2237      313 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 18:34:11 2019
     47    7095      490973289      17535     2471 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 18:34:18 2019
     48 2508958      490979761       6472      913 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 20:54:28 2019
     49 1913120      491149877     170116    23859 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 22:44:41 2019
     50    7086      491389981     240104    33884 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 22:44:48 2019
     51 1951583      491393151       3170      435 nic_err_rl_pkt_drops interface(0/1) Tue Sep 10 07:45:44 2019
     52  175907      491418713      25562     3195 nic_err_rl_pkt_drops interface(0/1) Tue Sep 10 07:48:40 2019
     53  790650      491423006       4293      601 nic_err_rl_pkt_drops interface(0/1) Tue Sep 10 08:26:28 2019

reltime:mili second between two records Tue Sep 10 08:59:44 2019
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
     54 1995990      491431215       8209     1135 nic_err_rl_pkt_drops interface(0/1) Tue Sep 10 08:59:44 2019
     55    7171      491448494      17279     2409 nic_err_rl_pkt_drops interface(0/1) Tue Sep 10 08:59:51 2019
 

Link to comment
Share on other sites

Your cpu and memory consumption looks good, but there are some very high package drops which could be the reason why you can`t access the ADC at this times.

 

Sorry for the command "nsconmsg -s ConLb=2 -d current -s ratecount=1250000 | grep -E "si_tot_ResponseBytes|si_tot_RequestBytes" | grep -v -E "svcgrp|cs_|internal|cons_si"". The second grep is specific to my naming conventions. Change "svcgrp" to the naming convention of your servicegroups and "cs_" for Content switches. The rest should be okay. Otherwise do only use this one:

 

nsconmsg -s ConLb=2 -d current -s ratecount=1250000 | grep -E "si_tot_ResponseBytes|si_tot_RequestBytes"

 

If you want to see historic data of Virtual Servers with more than 10Mbit/s at a specific time (like 50    7086      491389981     240104    33884 nic_err_rl_pkt_drops interface(0/1) Mon Sep  9 22:44:48 2019), first find out which newnslog stores your data.

 

1. cd /var/nslog

2. df -trl

3. Look for the newnslogs around your time (Mon Sep  9 22:44:48 2019)

4. nsconmsg -K newnslog.<Number>.tar.gz -d setime (Verify, that you found the right one which stores the data for this time)

5. nsconmsg -K newnslog.<Number> -s ConLb=2 -d past -s time=9Sep2019:22:44:00 -s ratecount=1250000 | grep -E "si_tot_ResponseBytes|si_tot_RequestBytes" |grep -v -E "svcgrp|cs_|internal|cons_si"

 

Sometimes commands do not work when you copy/paste. Then try to type it in again. If your logfile is not found, just check if it is unpacked and you don´t need the .tar.gz Extension.

 

Best regards,

Jens

Link to comment
Share on other sites

  • 4 weeks later...

Remember that the bandwith limit applies to ALL traffic into netscaler.... so once you exceed your licensed bandwidth, it's going to drop management traffic as much as any other traffic!

 

So why are you exceding the 200 meg? Busy day? Some sort of attack?

 

Whilst the firewall will stop any untoward traffic being passed to backend servers, it won't do much to stop an incoming attack, if the incoming data rate is huge (ie over the 200 meg limit!).

 

Clearly, if it was an attack, it's succeeding! There are several features on Netscaler which can be used to help mitigate / reduce the effect of many attacks.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...