Jump to content
Welcome to our new Citrix community!

VPX instance crash and failover with "pid 1226 (NSPPE-01), uid 0, was killed: out of swap space"


Philip Gao

Recommended Posts

hi Guys

We encountered a netscaler instance crash issue.

SDX physical device: 14030

SDX firmware version: 11.1 56.120

 

VPX version: 12.1 53.12.nc

 

Event time: 03:12:45am 9/4/2019 PDT timezone

 

When the crash event happened, HA failover to the secondary, but we didn't get any dump file

 

Check detail logs in attachment

notice.log

Sep  4 03:12:45 <kern.crit> sjenglb02 kernel: swap_pager: out of swap space
Sep  4 03:12:45 <kern.crit> sjenglb02 kernel: swap_pager_getswapspace(16): failed
Sep  4 03:12:45 <kern.err> sjenglb02 kernel: pid 1226 (NSPPE-01), uid 0, was killed: out of swap space
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 proc NSPPE-01 (1226) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29b4)
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 NetScaler restart may be delayed if collecting core dump for NSPPE-01 (1226)
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 Pitboss declaring system failure: NSPPE-01 (1226) exited
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="arp -d -a "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="grep -v nsfsyncd /etc/crontab >/tmp/.crontab.tmp "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="cp /tmp/.crontab.tmp /etc/crontab "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="grep -v <88>iked\|nskrb\|websocketd\|nscrlrefresh\|ntpd /etc/monitrc >/etc/monitrc.bak "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="chmod 600 /etc/monitrc.bak "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="mv /etc/monitrc.bak /etc/monitrc "
Sep  4 03:12:46 <local6.notice> sjenglb02 sh[91324]: sh_command="killall -HUP monit "
Sep  4 03:12:52 <local0.alert> sjenglb02 restart[91323]:   Nsshutdown lock released !

 

 

ns.log

Sep  4 03:12:45 <local0.info> 10.1.254.124 09/04/2019:10:12:45 GMT sjenglb02 0-PPE-1 : default API CMD_EXECUTED 147744478 0 :  User gearusr - Remote_ip 10.8.88.121 - Command "show service" - Status "Success"
Sep  4 03:12:45 <local0.info> sjenglb02 [27]: proc NSPPE-01 (1226) EXITED with status 0x9
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 proc NSPPE-01 (1226) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29b4)
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 NetScaler restart may be delayed if collecting core dump for NSPPE-01 (1226)
Sep  4 03:12:45 <local0.info> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 Nsshutdown sysctl lock remains free ...
Sep  4 03:12:45 <local0.alert> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 Pitboss declaring system failure: NSPPE-01 (1226) exited
Sep  4 03:12:46 <local0.info> sjenglb02 [27]: pitboss Wed Sep  4 03:12:46 2019 Deleting watch on NSPPE-01 (1226) for ()
Sep  4 03:12:46 <local0.info> sjenglb02 restart[91323]:   Attempting intial cleanup of netscaler daemons
Sep  4 03:12:46 <local0.info> sjenglb02 nscopo: nsnet_tcpipconnect: connect() failed; returned -1 errno=61
Sep  4 03:12:47 <local0.info> sjenglb02 httpd: nsnet_tcpipconnect: connect() failed; returned -1 errno=61
Sep  4 03:12:52 <local0.info> sjenglb02 restart[91323]:   Cleanup completed
Sep  4 03:12:52 <local0.alert> sjenglb02 restart[91323]:   Nsshutdown lock released !

 

 

 

I opened a case to Citrix support, but they can't find the root cause, and they don't know why the core dump file was not generated.

They told me it's been escalated to their engineering team, but I didn't get feedback until now.

 

Do you guys have any suggestion?

Link to comment
Share on other sites

  • 2 years later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...