Jump to content

Filesystem filling up with mastools-xxx files


Marc Wilhelmi

Recommended Posts

Hi,

 

we have a Netscaler/ADC HA-Pair filling up the /var filesystem with files in /var/core/<number>/, all named mastools-<4-digit number> i.e. mastools-2787

every 2 minutes a new file is generated with 7.8M in size

there is no unusual network or cpu load

 

this behavior started 2 days ago and only affects one HA-pair, all others look fine.

 

Any ideas what causes this behavior and how to stop it?

Link to comment
Share on other sites

Update:

reboot does not fix the issue here

Killing the process mastools_monit.sh stops this

 

ps -ax | grep mastools_monit.sh
 1458  ??  I      0:00.02 /bin/sh mastools_monit.sh

 

today the process is running again on both nodes, but only on one node is just one new mastools-xxx file, the other node has no new files.

 


 


 

Link to comment
Share on other sites

We have the same issue, on a single ADC and then also on a HA pair.

 

From what I can see the mastools scripts updated on 8th of August, then the issue started.

The scripts updated again on 10th of August around 1700.

Then around 0700 on the 11th the issue stopped and no more files being generated...

 

I raised a support call on the 9th, when we saw the issue, but Citrix support as usual are being useless. They have said there is a known issue in 13.0.90.13 and the workaround is:

go to shell

go to /var/mastools/scripts/

type "mastoolsd stop

 

We are on 13.0 91.13, and supposedly the issue still exists in that version. I queried why the issue didn't start when we updated to 13.0 91.13, which was several weeks ago. They have no answer for that. I personally don't believe the issue we are seeing is related to the 'known issue' in 13.0.90.13 as:

1. It started well after the update to 13.0 91.13 was completed.

2. It started shortly after mastools scripts updated themselves

3. It stopped on it's own

 

I asked Citrix to confirm:

1. Why the scripts have updated themselves? We are not familiar with this process.

2. If Citrix know about an issue with the mastools scripts which started on the 8th and has now been resolved somehow with a further updated mastools.

 

As of yet they are not giving anymore info, they are being useless.

  • Like 1
Link to comment
Share on other sites

hi. I think I have seen an error that is related to this 13.0.91.13 - and perhaps related to this mastools. After upgrading to this version Netscaler seems to lose the registration to ADM in Citrix Cloud. The Netscaler is configured to use the build-in ADM serviceagent. I have tried to reregister the agent as specified in https://docs.netscaler.com/en-us/citrix-application-delivery-management-service/getting-started/initiate-built-in-agent - but the ADM fails to import the Netscalers with the following error: "Failed to get netscaler version for "ip_service" Error: Error in config/nshostname.  And the ADM was able to communicate before the upgrade.

Link to comment
Share on other sites

I agree with Chris Gundry.  Something else happened.  It wasn't the update.  We updated on July 18th because of the crit vulnerabilities and I see these files started showing up on August 8th!

 

It also appeared that it "mysteriously" stopped creating the files between August 10th at 23:00 and August 11 02:00 on all 4 of our netscalers.  I am just finding this out yesterday 8/21 when I couldn't login on them.

 

Thanks for posted those steps and dealing with Citrix support, they have been useless for years!  Your post helped me get things right again.

 

 

 

Link to comment
Share on other sites

Dear Members,

 

Kindly follow these steps to fix the issue.

 

1. Clear out old NSTRACE, CORE/CRASH and SupportBundle Files (to ensure enough space on device)
2. Login to NetScaler CLI and type "shell"
3. execute "/var/mastools/scripts/mastoolsd stop"
4. execute "rm /var/core/mastools*"
5. execute " /var/python/bin/python /var/mastools/scripts/mastools_upgrade.py" (upgrade the function manually)


Gaurav Hirwaniya

Senior Escalation Engineer  - NetScaler

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...