Xenserver 7 file system on logs partition full

Nigel Page · May 30, 2018

Getting an error on our Xenserver 7 install. 'File System on Logs Partition Full'. I have read up on this and done all the suggestions of deleting old updates and old log files *.gz also I have changed some of the logging to only show warnings however it would seem that the Daemon.log is taking up all the space (3.8G) and I am not sure how to clear this? I have rebooted the hypervisor but this did not clear it down.

Need some help. Thanks

0 /var/log/audit.log
1.6M /var/log/audit.log.1
72K /var/log/blktap
8.0K /var/log/boot.log
16K /var/log/boot.log.1
0 /var/log/boot.log.1.gz
0 /var/log/btmp
0 /var/log/btmp.1
4.0K /var/log/cluster
0 /var/log/crit.log
0 /var/log/crit.log.1
0 /var/log/crit.log.1.gz
60K /var/log/cron
40K /var/log/cron.1
0 /var/log/cron.1.gz
3.8G /var/log/daemon.log
1.2M /var/log/daemon.log.1
0 /var/log/daemon.log.1.gz
40K /var/log/dmesg
44K /var/log/dmesg.old
4.0K /var/log/domainjoin-cli.log
824K /var/log/installer
4.0K /var/log/interface-rename.log
4.0K /var/log/interface-rename.log.1
0 /var/log/interface-rename.log.1.gz
208K /var/log/kern.log
440K /var/log/kern.log.1
0 /var/log/kern.log.1.gz
44K /var/log/lastlog
16K /var/log/lost+found
8.0K /var/log/maillog
0 /var/log/maillog.1
0 /var/log/maillog.1.gz
0 /var/log/messages
4.0K /var/log/ntpstats
4.0K /var/log/openvswitch
12K /var/log/ovs-ctl.log
12K /var/log/ovsdb-server.log
3.5M /var/log/ovs-vswitchd.log
4.0K /var/log/ovs-xapi-sync.log
8.0K /var/log/pbis-open-install.log
12K /var/log/restoreeswitchcfg.log
13M /var/log/sa
8.0K /var/log/samba
180K /var/log/secure
3.4M /var/log/secure.1
0 /var/log/secure.1.gz
0 /var/log/SMlog
1.4M /var/log/SMlog.1
0 /var/log/spooler
0 /var/log/spooler.1
0 /var/log/spooler.1.gz
0 /var/log/tallylog
60K /var/log/user.log
28K /var/log/user.log.1
0 /var/log/user.log.1.gz
32K /var/log/VMSSlog
76K /var/log/VMSSlog.1
0 /var/log/VMSSlog.1.gz
84K /var/log/wtmp
8.0K /var/log/xcp-rrdd-plugins.log
12K /var/log/xcp-rrdd-plugins.log.1
0 /var/log/xcp-rrdd-plugins.log.1.gz
368K /var/log/xen
272K /var/log/xensource.log
12M /var/log/xensource.log.1
0 /var/log/xenstored-access.log
696K /var/log/xenstored-access.log.1
0 /var/log/yum.log
4.0K /var/log/yum.log.1

NIELS HANSSEN · August 17, 2018

It's a somewhat old issue. We had the same issue.

Got it solved by doing this.

cd /var/log [enter]

ls -liahS [enter]

This will show the biggest filesize

rm <largefilename> -f [enter]

Removes the large file.

check again which files are large.
If there is still no drive space available after removing the large files. It might be that the file is still in use.
Check this with..

lsof +L1

Depending on the process holding the large files, you could try restarting the service. in my case the service rsyslog restart [enter]. did the trick.
Check again with df -h to see if you have space available again.

We had to do this a couple of times. I think once we restarted the poolmaster we got it sorted out.
Also check your root folder for a file called dead.letter it might contain information about the source of all evil.
I also wrote a monitor script that checks all partitions and the existence of a dead.letter, that gets reported to me every day.

Could have cleared it out with the same script. However we wanted to keep control on what's going on on the xen servers.

Alan Lantz · May 30, 2018

Thats odd. Can you remove it ? And if you do how long before you run out of space again ?

You will have to parse that log and see what is filling it up.

--Alan--

Nigel Page · May 30, 2018

Is it ok to remove this? and if so would it then require a restart of xenserver?

Alan Lantz · May 30, 2018

Sure, its just a log file. Sometimes the space will be freed up once deleted, other times a host restart is required to

get the space to show up as free again. But likely you have something wrong that is causing the excessive logging

that you will need to resolve.

--Alan--

Tobias Kreidl · May 31, 2018

Is this with the new partition layout? You might want to list and sort all your log entries by size to see which are the biggest ones. YOu may want to do the same and sort by date to see if some issue cropped up at a specific time or if this is ongoing. Any chance you might have a crash dump somewhere?

-=Tobias

Alan Lantz · August 17, 2018

dead.letter is just an aborted email, right ?

Glad to hear you had your space figured out, the new partitioning scheme really helps in this.

--Alan--

NIELS HANSSEN · August 21, 2018

Don't know for sure if that's an aborted mail.

It looks like to be a log file that sometimes shows crucial information to the cause of the error.

Like for example. When you local storage on the XenServer is full. It will give you a vague message that it can't start the VM.
In the dead.letter you will see a clear message saying that the disk is full.

Off course there are other ways to see if the disk is full. But that's besides the point.. ;)

Marc ONeill · August 21, 2018

Do NOT remove a file with a currently open file descriptor as this may hold on to the space until the host is rebooted. A safer option is to clear the file by echoing an empty string into it or redirect null into it.

echo " " > daemon.log

or

> daemon.log

Balaji Muthukrishnan · January 25, 2019

Have same issue here in my environment were /var/log partition gets filled up with tons of logs in quick time following XS management interface will go offline and un managed. Only option left is to poweroff and power on the server (blade) through iLO

list of logs which is growing in size are...

blktap

daemon.log

xensource.log

mine is a XenServer 7.0 (with all Hotfix uptodate). I am living with this issue for more than a year now. by manually clearing the logs from /var/log and rebooting the server, by this way the management interface would come up again...

we worked with support and finally came to a common agreement that the issue of log filling up /var/log partition is due to a bug in xentools version of XenServer 7.0 and the same was reportedly fixed in XenServer 7.1 + version.

Balaji Muthukrishnan · January 25, 2019

I have later upgraded one of the pool to XenServer 7.1 LTSR CU2 and found the issue was still appearing and XS goes offline due to /var/log partition gets filled-up.

I have tried limitting the excessive logging of XS by modifying the conf file. refer KB https://support.citrix.com/article/CTX214093. Even that didnt help, however this has extended the time log file growth.

~~~~
CTX214093
Excessive Logging Can Fill Up XenServer Host Partition When Installing XenServer Tools
~~~~

Later found that the logs were generated as the XenTools / windows management agent installed in the VDAs (for XenServer 7.0 version) were generating the logs causing the /var/log partition to full (blktap files)

Based on my understanding a schedule task created when we install XenTools named (Citrix Management Agent Auto-Updater), thought this will take care of updating the windows management agent / xentools when i upgrade from XS 7.0 XS 7.1 LTSR CU2 for the persistent VDAs.

But the Task scheduler isn't helping and not doing anything. The only option left was to update it manually or remotely using scripts and it helps.

(PsExec.exe \\testVDA cmd /c "c:\temp\guest-tools\setup.exe /quiet") or /passive

https://support.citrix.com/article/CTX222533 - Install XenServer tools silently
https://docs.citrix.com/en-us/xencenter/7-1/vms-installtools.html
https://support.citrix.com/article/CTX137027 - Unattended Installation of XenServer Tools

The only difference here is with persistent VM and non persistent VMs, for streamed VDAs we need to disable windows management agent auto update feature by disabling the schedule task or modify the registry entry as mentioned here...

https://support.citrix.com/article/CTX222494 - Disabling the Management Agent Updates

Note: When installing Windows Management Agent, it gives you option to Allow/Disallow I/O driver updates by the management agent. This option is not available when you update XenTools / Windows Management Agent through cmd line. So modify the registry entry as below after the upgrade is completed to set the windows management agent to Allow I/O driver updates by the management agent (Allow automatic I/O driver updates by the management agent)

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\XenTools\AutoUpdate
InstallDrivers REG_SZ YES

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\XenTools\AutoUpdate]
"InstallDrivers"="YES"
"Identify"="NO"

I am going to update the WIndows management Agent for the persistent VMs for a specific pool and going to observe the log size growth...

Thanks...

Joshua Holmes · April 22, 2020

So out of curiosity, which logs files can you remove and NOT have to reboot the host?

So for instance xensource.log gets rotated and now there's the main log plus xensource.log.1. If we delete the *.1 log, or whatever the oldest log is, do we have to restart the host still or any other services?

Alan Lantz · April 22, 2020

You shouldn't need to restart a host by remove log files. There are times when a host won't recognize the removal of file and not

give that free space back until a host is restarted. Thankfully with the new file system layout starting with 7.x being out of space

is not a common problem anymore.

--Alan--

Joshua Holmes · April 22, 2020

Thanks Alan. You're correct, I just went in and removed all the *gz and *.1 without an issue and then edit rsyslogs to not report debug level stuff and of course send to syslog server. Thinking we should be good now. Appreciate the reply

Xenserver 7 file system on logs partition full

Question

Link to comment

13 answers to this question

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Create an account or sign in to comment

Create an account

Sign in