Jump to content
Welcome to our new Citrix community!
  • 0

Xenserver 7 file system on logs partition full


Nigel Page

Question

Getting an error on our Xenserver 7 install. 'File System on Logs Partition Full'. I have read up on this and done all the suggestions of deleting old updates and old log files *.gz also I have changed some of the logging to only show warnings however it would seem that the Daemon.log is taking up all the space (3.8G) and I am not sure how to clear this? I have rebooted the hypervisor but this did not clear it down.

Need some help. Thanks

 

0       /var/log/audit.log
1.6M    /var/log/audit.log.1
72K     /var/log/blktap
8.0K    /var/log/boot.log
16K     /var/log/boot.log.1
0       /var/log/boot.log.1.gz
0       /var/log/btmp
0       /var/log/btmp.1
4.0K    /var/log/cluster
0       /var/log/crit.log
0       /var/log/crit.log.1
0       /var/log/crit.log.1.gz
60K     /var/log/cron
40K     /var/log/cron.1
0       /var/log/cron.1.gz
3.8G    /var/log/daemon.log
1.2M    /var/log/daemon.log.1
0       /var/log/daemon.log.1.gz
40K     /var/log/dmesg
44K     /var/log/dmesg.old
4.0K    /var/log/domainjoin-cli.log
824K    /var/log/installer
4.0K    /var/log/interface-rename.log
4.0K    /var/log/interface-rename.log.1
0       /var/log/interface-rename.log.1.gz
208K    /var/log/kern.log
440K    /var/log/kern.log.1
0       /var/log/kern.log.1.gz
44K     /var/log/lastlog
16K     /var/log/lost+found
8.0K    /var/log/maillog
0       /var/log/maillog.1
0       /var/log/maillog.1.gz
0       /var/log/messages
4.0K    /var/log/ntpstats
4.0K    /var/log/openvswitch
12K     /var/log/ovs-ctl.log
12K     /var/log/ovsdb-server.log
3.5M    /var/log/ovs-vswitchd.log
4.0K    /var/log/ovs-xapi-sync.log
8.0K    /var/log/pbis-open-install.log
12K     /var/log/restoreeswitchcfg.log
13M     /var/log/sa
8.0K    /var/log/samba
180K    /var/log/secure
3.4M    /var/log/secure.1
0       /var/log/secure.1.gz
0       /var/log/SMlog
1.4M    /var/log/SMlog.1
0       /var/log/spooler
0       /var/log/spooler.1
0       /var/log/spooler.1.gz
0       /var/log/tallylog
60K     /var/log/user.log
28K     /var/log/user.log.1
0       /var/log/user.log.1.gz
32K     /var/log/VMSSlog
76K     /var/log/VMSSlog.1
0       /var/log/VMSSlog.1.gz
84K     /var/log/wtmp
8.0K    /var/log/xcp-rrdd-plugins.log
12K     /var/log/xcp-rrdd-plugins.log.1
0       /var/log/xcp-rrdd-plugins.log.1.gz
368K    /var/log/xen
272K    /var/log/xensource.log
12M     /var/log/xensource.log.1
0       /var/log/xenstored-access.log
696K    /var/log/xenstored-access.log.1
0       /var/log/yum.log
4.0K    /var/log/yum.log.1
 

Link to comment

13 answers to this question

Recommended Posts

  • 2

It's a somewhat old issue. We had the same issue.

Got it solved by doing this.

cd /var/log [enter]

ls -liahS [enter]

This will show the biggest filesize 

rm <largefilename> -f [enter]

Removes the large file.

 

check again which files are large.
If there is still no drive space available after removing the large files. It might be that the file is still in use.
Check this with..

lsof +L1

 

Depending on the process holding the large files, you could try restarting the service. in my case the service rsyslog restart [enter]. did the trick.
Check again with df -h to see if you have space available again.

We had to do this a couple of times. I think once we restarted the poolmaster we got it sorted out.
Also check your root folder for a file called dead.letter it might contain information about the source of all evil.
I also wrote a monitor script that checks all partitions and the existence of a dead.letter, that gets reported to me every day.

Could have cleared it out with the same script. However we wanted to keep control on what's going on on the xen servers.


 

  • Like 2
Link to comment
  • 0

Sure, its just a log file. Sometimes the space will be freed up once deleted, other times a host restart is required to

get the space to show up as free again. But likely you have something wrong that is causing the excessive logging

that you will need to resolve.

 

--Alan--

 

Link to comment
  • 0

Is this with the new partition layout? You might want to list and sort all your log entries by size to see which are the biggest ones. YOu may want to do the same and sort by date to see if some issue cropped up at a specific time or if this is ongoing. Any chance you might have a crash dump somewhere?

 

-=Tobias

Link to comment
  • 0

Don't know for sure if that's an aborted mail.

 

It looks like to be a log file that sometimes shows crucial information to the cause of the error.

Like for example. When you local storage on the XenServer is full. It will give you a vague message that it can't start the VM.
In the dead.letter you will see a clear message saying that the disk is full.

Off course there are other ways to see if the disk is full. But that's besides the point.. ;)

 

Link to comment
  • 0

Have same issue here in my environment were /var/log partition gets filled up with tons of logs in quick time following XS management interface will go offline and un managed. Only option left is to poweroff and power on the server (blade) through iLO

 

list of logs which is growing in size are...

 

blktap

daemon.log

xensource.log

 

mine is a XenServer 7.0 (with all Hotfix uptodate). I am living with this issue for more than a year now. by manually clearing the logs from /var/log and rebooting the server, by this way the management interface would come up again...

 

we worked with support and finally came to a common agreement that the issue of log filling up /var/log partition is due to a bug in xentools version of XenServer 7.0 and the same was reportedly fixed in XenServer 7.1 + version.

 

 

 

Link to comment
  • 0

I have later upgraded one of the pool to XenServer 7.1 LTSR CU2 and found the issue was still appearing and XS goes offline due to /var/log partition gets filled-up.

 

I have tried limitting the excessive logging of XS by modifying the conf file. refer KB https://support.citrix.com/article/CTX214093. Even that didnt help, however this has extended the time log file growth.

~~~~
CTX214093
Excessive Logging Can Fill Up XenServer Host Partition When Installing XenServer Tools
~~~~

Later found that the logs were generated as the XenTools / windows management agent installed in the VDAs (for XenServer 7.0 version) were generating the logs causing the /var/log partition to full (blktap files)

 

Based on my understanding a schedule task created when we install XenTools named (Citrix Management Agent Auto-Updater), thought this will take care of updating the windows management agent / xentools when i upgrade from XS 7.0 XS 7.1 LTSR CU2 for the persistent VDAs.

But the Task scheduler isn't helping and not doing anything. The only option left was to update it manually or remotely using scripts and it helps.

 

(PsExec.exe \\testVDA cmd /c "c:\temp\guest-tools\setup.exe /quiet") or /passive

 

https://support.citrix.com/article/CTX222533 - Install XenServer tools silently
https://docs.citrix.com/en-us/xencenter/7-1/vms-installtools.html
https://support.citrix.com/article/CTX137027 - Unattended Installation of XenServer Tools


The only difference here is with persistent VM and non persistent VMs, for streamed VDAs we need to disable windows management agent auto update feature by disabling the schedule task or modify the registry entry as mentioned here...

https://support.citrix.com/article/CTX222494 - Disabling the Management Agent Updates

 

Note: When installing Windows Management Agent, it gives you option to Allow/Disallow I/O driver updates by the management agent. This option is not available when you update XenTools / Windows Management Agent through cmd line. So modify the registry entry as below after the upgrade is completed to set the windows management agent to Allow I/O driver updates by the management agent (Allow automatic I/O driver updates by the management agent)

 

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\XenTools\AutoUpdate
InstallDrivers REG_SZ YES

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\XenTools\AutoUpdate]
"InstallDrivers"="YES"
"Identify"="NO"

 

 

I am going to update the WIndows management Agent for the persistent VMs for a specific pool and going to observe the log size growth...

 

Thanks...

Link to comment
  • 0

So out of curiosity, which logs files can you remove and NOT have to reboot the host?

 

So for instance xensource.log gets rotated and now there's the main log plus xensource.log.1.  If we delete the *.1 log, or whatever the oldest log is, do we have to restart the host still or any other services?

Link to comment
  • 0

You shouldn't need to restart a host by remove log files.  There are times when a host won't recognize the removal of file and not 

give that free space back until a host is restarted. Thankfully with the new file system layout starting with 7.x being out of space

is not a common problem anymore.

 

--Alan--

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...