Having trouble rebooting Ubuntu 16.04 VMs

[Tom Kiblin1709158143]

We're having some serious issues with shutdowns and restarts locking up which happens across multiple hardware nodes, on local disk.  VMs are locking up about 10-20 seconds after rebooting from a hard reboot or hard shutdown.  If we do a command line or tools shutdown and Xen puts the VM into an off state it boots normally.  Occasionally,   Xen never puts the VM into an off state after reaching Shutdown Target, and we need to hard reboot it at that point.  Once we hit that we see the 10-20 second into boot lockup and we need to reboot it multiple times to get it to boot correctly.  Occassionally, it makes it through boot to login, but locks up.    We have the same stack running elsewhere and are not seeing an issue and there are no logs on the OS that indicate a problem.   Any tips to help debug this?

[Alan Lantz]

A lot will depend on your environment. What version of XenServer? Latest hotfixes installed? Does Ubuntu have

the latest xentools client installed ? Do you have any Windows VM's running that seem okay and its just Linux

VM's? If not configure a Windows VM for comparison testing.

 

I hate it when things just sorta kinda work.

 

--Alan--

 

[Tom Kiblin1709158143]

Hi Alan,

 

Appreciate the reply.  We have two XenServer pools where one is 7.6 and the other is 7.5.  The Ubuntu 16.04 VMs are running 7.2 and we have a handful of Centos 6.9 VMs that were migrated from VMware and are running 7.4.  Problem children are the Ubuntu VMs created from a full copy clone made in XenServer.   VMs seems to have more of a problem when they have OOMed at some point.  I can see the agent version the issue here, but here is a dumb question .. how do you actually update the agent version through XenCenter?  Where install the agent used to be is the virtualization state.

[Tobias Kreidl]

Are your NTP settings in place and is synchronization good? Check with "ntpstat -s" on each host.

 

-=Tobias

[Tom Kiblin1709158143]

Hi Tobias,

 

Was just reading your article on tips for setting up NTP and appreciate your reply.   I'm seeing a 32ms to 54ms delay.

[Tobias Kreidl]

Those are very solid values, so it's probably something else.

 

-=Tobias

[Alan Lantz]

For the XenCenter agent I think you can grab it from the ISO for the XenServer 7.6 installation and have the

latest ubuntu client. I haven't noticed XenServer Linux agents making any difference, It wouldn't hurt to

update it. I would more suspect access to the disks. Anything in /var/SMLog that looks suspicious ?

 

--Alan--

 

 

[Tom Kiblin1709158143]

Hi Alan,

 

Disks are local and it happens across host nodes, but it could be a compatibility issue.  Seems as though most issues with Ubuntu sits there at target shutdown are disk related.  Talked with Citrix about it.  Regretfully, we didn't have the client's permission to turn one off and show them, but the VMs effected all had very high memory usage (95%+).  They suggested that there may be an issue with the agent not being able to talk to Citrix due to memory issues, and Citrix believes the system is up when in fact it's down.  Not sure about that.  I would think that xen itself would listen for the ACPI calls for shutdown.

[Tobias Kreidl]

You can monitor system resource utilization readily with the utilities top and xentop.  Look for excessiveCPU usage and also dom0 running out of memory and starting to swap.

 

-=Tobias

[Boby John1709155536]

I would look at /var/log/daemon.log to see any obvious errors. Its worth looking at the in-guest logs as well.

Also a brand new Ubuntu 16.04 faces the same problem as well ? 

Also I assume that you are not using a Netboot Ubuntu ISO