XEN 8.2 /var/log/secure filling up with to 3GB every few weeks crashing hosts

[David Nguyen1709155977]

We recently moved from VMWare to XEN and its been working ok so far, but I have been getting alerts regarding the Control domain getting full. I do see that the /var/log/secure file is growing a very fast pace with all stunnel traffic logs. 

 

At first i just ignored it, but then my host crashed due to it. I currently  have a script that purges the secure file every time i get the alerts which has been monthly for some XEN hosts. Although this seems likes it logging too much? has anyone every seen this grow to such a large size? This does not happen on all m y hosts at all sites which is strange. We are 8.2 and updated to E017.  Thanks

 

https://support.citrix.com/article/CTX202821

 

 

[Tobias Kreidl]

Did you install it or upgrade it to the new partition layout?

[David Nguyen1709155977]

Hi, this was a fresh build from ISO.  We did not muck with any partition layout. Thanks

[Tobias Kreidl]

It should default to the 42.5 GB and more than 2 partition setup.  Check the logs for excessive logging; it should never do this on its own, or something is amiss!

[Boby John1709155536]

Can you have the host patched to the latest and tell us what line is repeated the most in secure.log ? 

[Alan Lantz]

Agree with everyone, you have something wrong that is doing excessive logging.

 

--Alan--

 

[Tobias Kreidl]

Indeed. Need to review your logs for what is filling them up so fast. Rotating more often or truncating the logs isn't going to address the primary issue.

[David Nguyen1709155977]

We did open a case with Citrix, they had me try option "B"  that did work on some hosts. But not all hosts. We ended up doing opiton "A" and just limiting the logging.

 

A)

1) Edited file /etc/rsyslog.d/xenserver.conf and comment below line

authpriv.warning                                             -/var/log/secure

 

B)

1) Put the host into maintenance mode if HA is enabled.
2) Run command "> secure"
3) Run command "xe-toolstack-restart"
4) Run command "logrotate -f -v /etc/logrotate.conf"
5) Run command "logrotate -f -v /etc/logrotate-xenserver.conf"
6) Run command "systemctl restart rsyslog.service"
7) Repeat step 3

 

Now, i'm seeing my xensource.log getting filled up. 

[Tobias Kreidl]

Again, this simply addresses the symptoms, not the underlying cause.