Jump to content
Welcome to our new Citrix community!

Netscaler 100% CPU


Martijn Kools

Recommended Posts

Hi, since we've upgraded to latest version 12.0 53_22 the CPU is running at 100% 24/7.

 

Top shows NSPPE-00:

 

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1200 root         1  44   r0   970M   970M CPU1    1  70.1H 100.00% NSPPE-00
 

Is this a bug or is there a fix?

 

Link to comment
Share on other sites

2 minutes ago, Preetha Sampath1709156282 said:

Please paste the output of the following command from the CLI:

> stat ns | grep -i cpu

 

 

 

 

Ran it a few times, machine has 2 cores so seems like 1 is at or around 100%:

 

> stat ns | grep -i cpu
Packet CPU usage (%)                3.00
Management CPU usage (%)           46.70
> stat ns | grep -i cpu
Packet CPU usage (%)                2.50
Management CPU usage (%)           43.70
> stat ns | grep -i cpu
Packet CPU usage (%)                2.50
Management CPU usage (%)           43.70
 

Link to comment
Share on other sites

 

Hi,

 

Is this consistently high? What is the platform of the NetScaler?

 

NSPPE showing 100% is by design. The packet engines (NSPPE) perform their own kernel scheduling services internally and their utilization cannot be monitored with the top command.

Thus, this is not relevant.

 

Please paste the output of the top command.

Also note that the management CPU can go high intermittently because of the zgrep process which facilitates the  file compression on NetScaler

 

In addition, we need a profiler output  to understand which process is contributing to the cpu.

 

Link to comment
Share on other sites

1 minute ago, Preetha Sampath1709156282 said:

 

Hi,

 

Is this consistently high? What is the platform of the NetScaler?

 

NSPPE showing 100% is by design. The packet engines (NSPPE) perform their own kernel scheduling services internally and their utilization cannot be monitored with the top command.

Thus, this is not relevant.

 

Please paste the output of the top command.

Also note that the management CPU can go high intermittently because of the zgrep process which facilitates the  file compression on NetScaler

 

In addition, we need a profiler output  to understand which process is contributing to the cpu.

 

 

Yes it's consistently high, platform is ESX 5.5.

 

It looks like perl monitoring processes that are eating up the CPU. We do have load balancing and monitoring configured for several services.


And yeah I red abut the NSPPE, I even knew that already from a similar issue a long time ago :) 

 

Quote

last pid:  9990;  load averages:  1.45,  1.42,  1.44  up 2+23:32:18    12:19:48
71 processes:  4 running, 67 sleeping

Mem: 244M Active, 174M Inact, 2907M Wired, 199M Buf, 188M Free
Swap: 4200M Total, 4200M Free


  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1200 root         1  44   r0   970M   970M CPU1    1  71.5H 100.00% NSPPE-00
 7756 nsmonitor    1  64    0 43172K 24476K sbwait  0  10:00 22.85% perl
 8968 nsmonitor    1  54    0 43172K 24444K sbwait  0   7:00 15.28% perl
 1202 root         1  44    0 82708K 33712K kqread  0  22:26  0.00% nsnetsvc
 1309 root         1  44    0 36568K  7656K RUN     0   6:10  0.00% nsrised
 1444 root         1  44    0 54592K  4200K nanslp  0   4:27  0.00% nsprofmon
  988 root         1  44    0 31664K  5348K select  0   2:03  0.00% vmtoolsd
 1224 root         1  44    0 96260K 34104K RUN     0   1:39  0.00% nsaggregato
 1327 root         1  44    0 63324K 34024K nanslp  0   0:37  0.00% nscollect
 1253 root         1  44    0 43948K 19712K nanslp  0   0:35  0.00% nsgslbautos
   23 root         1  44    0 29668K 29756K kqread  0   0:31  0.00% pitboss
 1357 root         1  44    0 66808K 20816K kqread  0   0:27  0.00% nscopo
 1251 root         1  44    0 56308K 16292K kqread  0   0:27  0.00% nsconfigd
 1305 root         1  44    0 48136K  8116K kqread  0   0:25  0.00% snmpd
 1437 nsmonitor    1  44    0 29664K  3120K kqread  0   0:21  0.00% nsumond
 1225 root         1  44    0   101M  4012K kqread  0   0:15  0.00% nsclusterd
 1280 root         1  44    0 39316K 13396K select  0   0:15  0.00% php
  965 root         1  44    0   103M 27572K select  0   0:09  0.00% httpd
 

 

Link to comment
Share on other sites

Yes, looks like the perl script monitor is causing the CPU spike

Do you have a LDAP monitor with the secure option enabled? If so, can you please test with the following workaround?

 

1. Configure PLAINTEXT LDAP on port 389 (instead of secure LDAP on port 636)

                                        

2. But if secure LDAP (port 636) is required, the following workaround would help (by avoiding a blocking call from AAAD to LDAPS) :-

Step 1. Create SSL_TCP services for each secure(port 636) LDAP server.

Step 2. Create TCP LB Vserver for each such service.

Step 3. Pass the LB vserver's IP as -ServerIP to ldapaction with -secType PLAINTEXT.

 

  • Like 2
Link to comment
Share on other sites

52 minutes ago, Preetha Sampath1709156282 said:

Yes, looks like the perl script monitor is causing the CPU spike

Do you have a LDAP monitor with the secure option enabled? If so, can you please test with the following workaround?

 

1. Configure PLAINTEXT LDAP on port 389 (instead of secure LDAP on port 636)

                                        

2. But if secure LDAP (port 636) is required, the following workaround would help (by avoiding a blocking call from AAAD to LDAPS) :-

Step 1. Create SSL_TCP services for each secure(port 636) LDAP server.

Step 2. Create TCP LB Vserver for each such service.

Step 3. Pass the LB vserver's IP as -ServerIP to ldapaction with -secType PLAINTEXT.

 

 

Nice, removing secure from the monitors did the job! (We have 2 LDAP monitors).

 

Thanks for the quick support, much appreciated!

 

Link to comment
Share on other sites

34 minutes ago, Carl Stalhood1709151912 said:

Go to System > Settings. On the right, in the bottom of the second column, click Change VPX Configuration Settings. Change the CPU Yield drop-down to YES, and click OK.

 

That's for ESX 6.0 we run 5.5, however I had another client with a similar issue after upgrading who are on 6.0 and this solved the issue for them, thanks!

 

Link to comment
Share on other sites

 

 

The output of top doesn't mean that the CPU is at 100%. All that means is that the nskernel owns 100% of the CPU this will show for every PPE aka if you have 8 cores the first 8 lines of top will look the same as the output provided and increment NSPPE -01 ,NSPPE-02 etc. 

 

Output below from a system with 4 cores aka 1 management and 3 PPE:

 

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1796 root         1  44   r0  2895M  2896M CPU3    3 1585.7 100.00% NSPPE-02
 1794 root         1  44   r0  2895M  2896M CPU2    2 1585.7 100.00% NSPPE-01
 1795 root         1  44   r0  2896M  2896M CPU1    1 1585.7 100.00% NSPPE-00

 

Output from a 12 core system licensed for only 5 PPE and 1 management:

 

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1178 root         1  44   r0  2636M  2637M CPU2    2 984.8H 100.00% NSPPE-01
 1180 root         1  44   r0  2636M  2637M CPU7    7 984.8H 100.00% NSPPE-03
 1181 root         1  44   r0  2636M  2637M CPU8    8 984.8H 100.00% NSPPE-04
 1179 root         1  44   r0  2636M  2637M CPU6    6 984.8H 100.00% NSPPE-02
 1177 root         1  44   r0  2636M  2638M CPU1    1 984.8H 100.00% NSPPE-00
 

Link to comment
Share on other sites

  • 2 weeks later...

Yes, Netscaler Packet Processing cores always run at 100%. What CPU value you see, depends where you look: typically, the XS/ESX will show the "reported" CPU (ie the active CPU, as reported by Netscaler), although, in some versions, bugs make this go wrong. Internally, the PE cores are sitting in an "I'm waiting to do something" loop, and use 100% CPU to do this - just like DOS used to use 100% cpu, doing nothing!

 

Latest versions allow certain tweaks, including the "CPU Yield" setting, which will make Netscaler give up the CPU when idle.... this is fine for a dev unit, but not suggested for production use, where performance is essential

Link to comment
Share on other sites

10 minutes ago, Paul Blitz said:

Yes, Netscaler Packet Processing cores always run at 100%. What CPU value you see, depends where you look: typically, the XS/ESX will show the "reported" CPU (ie the active CPU, as reported by Netscaler), although, in some versions, bugs make this go wrong. Internally, the PE cores are sitting in an "I'm waiting to do something" loop, and use 100% CPU to do this - just like DOS used to use 100% cpu, doing nothing!

 

Latest versions allow certain tweaks, including the "CPU Yield" setting, which will make Netscaler give up the CPU when idle.... this is fine for a dev unit, but not suggested for production use, where performance is essential

 

Well, the client only has Netscaler standard and at most maybe 20 ICA only connections, it makes no sense to let it run at such high cpu load all the time. Besides older versions of Netscaler didn't do this. Either way it's annoying to see a CPU alert in VMware all the time, reporting 100% CPU whether it's true or not.

Link to comment
Share on other sites

This is actually the way Netscaler has ALWAYS worked.... the packet CPUs sit at 100%, albeit doing nothing for 99% of the time on a quiet day. However, (assuming the "hooks" are all working right) Netscaler will only report back to ESX / Xen that it's using the 1%.

 

The CPU yield is a new thing.

 

So, what DOES a CPU do (eg Windows) when it is "idle"? Ok, it slows down, but it basically just sits there "doing nothing" quite fast....

Link to comment
Share on other sites

  • 2 years later...
On 12/18/2017 at 6:55 AM, Carl Stalhood1709151912 said:

Go to System > Settings. On the right, in the bottom of the second column, click Change VPX Configuration Settings. Change the CPU Yield drop-down to YES, and click OK.

 

This definitely worked.  You do have to reboot for this to take effect.

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...