XenServer appears disconnected in XenPool - But VMs running

[Keith Woodworth]

Hi folks, not sure where to post this but will try this in the CLI Forum.

 

Xenserver 7.2, 3 hosts in a pool.

Host 3 in the pool has become unresponsive/disconnected in XenCenter. Red dot with white line through it.

Currently there are 4 VM's running on that host. All hosts are up and working as normal, except one which I

have managed to put into a state where VDI is not available, but is still running and can SSH to that VM.

 

Not in HA

can ssh to disconnected host 3,

toolstack-restart works on that host 3

xapi is running and have restarted it on host 3

4 VM's running on host 3

xe host-list  on master and slaves all show the 3 Xenserver hosts in the list.

Pool.conf on host 3 shows correct: slave:10.11.32.230 for the master.

 

Xencenter shows ram and cpu usage for disconnected host.

I can list all the VM's running via CLI on host 3.

From /var/xapi the state.db file shows last update on Sept 21 @ 15:28.

All NTP times are in sync, DNS works.

 

I have not had a server in a pool do this before and did a command line shutdown of a test VM

so I could restart it on one of the other hosts.

 

I ran this CLI on Host 3:

xe vm-reset-powerstate uuid=a8b5fc05-07bc-c67e-339e-66e4110c7d32 force=true

 

I can now see this VM in Xencenter now but it is shutdown and is not longer in the CLI

list of VMs running on host 3.

 

But now I cannot start that VM as VDI is not available but it is still running as I can ssh

to that VM and do DNS queries against it. (test DNS server), so the powerstate command

did not shutdown the machine properly.

 

Can I issue a shutdown on the VM CLI? CentOS 7.latest, was fully patched last week.

 

So I have Host 3 disconnected in XenCenter, the VMs on the host are running and accessible

but one VM is now VDI not available.

 

How to reattach the VDI to the VM? Tons of articles, all slightly different and so far have not been able

to reconnect VDI to the VM.

 

I see this article: https://support.citrix.com/article/CTX221207

This link lists steps to reconnect a host into a pool, but it doesn't mention anything about any VMs

that might be running on a host.

 

All VM's in shared NFS mounted Netapp, no data on local SR of Host 3.

 

Some promising links in discussions.citrix.com are no longer working. Anyone have some

links to help reconnect host 3 and reattach the VDI to my VM?

 

Thank you,

Keith

 

 

[Keith Woodworth]

Hi.

So update to this as I don't like leaving these things hanging.

 

I have managed to move all the VM's that were still running on this host to other

xenserver hosts.

 

I followed instructions on this link:

 

https://www.serveroperations.net/citrix/start-a-xenserver-vm-that-throws-vdi-not-available-error-when-you-attempt-to-start

 

and was able to restart each VM on a new Xen host.

 

The Xen host is still not in the pool and the only option is to destroy the host from Xencenter.

 

Thing is, everything else on this Xen host seems ok, xsconsole from CLI works, I can see all network

interfaces, ssh into the Xen host, restart xentoolstack, restart xapi, but it is dropped from pool.

 

I'd like to try this:

 

https://support.citrix.com/article/CTX221207

 

But it is from an older version of Xen and not sure if it will work as the network stack seems to be

fully working.

 

Anything else I can try to get this host to rejoin the pool? One last, I have not actually restarted this

Xen host yet. Is this something that should be ok to do?

 

Thank you.

[Tobias Kreidl]

Make sure all your hosts are properly synchronized to NTP and the offsets re not too great. You can also try a "xe pool-sync-databasse" to see if that helps.

As long as the VMs are running, it should still be OK though you may run into problems if you try to Xenmotion a VM to a different host.

Finally, if you use any external databases, make sure they are also synchronized to the same NTP services.

-=Tobias

[Keith Woodworth]

Hi Tobias.

NTP uses same time servers and seem to be the same.

I use one local and 3 time.nrc.ca IP's.

 

According to ntpstat all three hosts are currently using a strat 3 time.nrc.ca server and time is correct on

all three to within 40-50 ms.

 

All VM's are set via ntp as well.

 

Thanks,

Keith

 

 

 

[Tobias Kreidl]

Try to do a forced power reset on any VMs in question and see if you can then successfully restart them.

Make sure first they have a VDI properly associated with them.

 

-=Tobias

[Keith Woodworth]

I did try on a test VM I have running on that host.

 

First I did:

 

xe vm-list resident-on=9b86ecc6-7a97-4c71-b507-9785cd72a1c2 (UUID being Host 3)

 

To get the list of VM UUID's.

 

Then did on Host 3:

xe vm-reset-powerstate uuid=a8b5fc05-07bc-c67e-339e-66e4110c7d32 force=true

Should this have been done from the master?

 

It shutdown the VM and now shows up as a powered off VM, but in reality the VM is still

running but it has disassociated the VDI and get a VDI is not available when trying to start

the VM in Xencenter on another host.

 

Is there a different xe cli to do a forced reset associate the VDI back to the VM?

Thank you.

Keith

[Alan Lantz]

If its just issues with the VM starting and the vdi is indeed okay I've had it occur where restarting the 

host that the VM last ran on cleared up whatever lock's were going on with the vdi. 

 

--Alan--

 

 

[Tobias Kreidl]

Can you migrate the VM to another server host or restart it on a different host? If not, there's something probably locked on that particular host and it needs to be restarted.

 

-=Tobias

[Alan Lantz]

I don't know of anything that would prevent that from working. Making it standalone again and 

rejoining the pool sounds like good advice.

 

--Alan--

 

[Tobias Kreidl]

Yes, a forced eject and rejoining the pool might be one good option. You could also try a "xe pool-database-sync" (I think that's the right command).

[Keith Woodworth]

Hello.

Thank you for answering my questions.

 

Final update.

 

I found a topic thread very similar to what I was seeing:

 

https://discussions.citrix.com/topic/389164-xenserver-host-disabled-after-pool-master-migration/page/2/

 

Main diff, I didn't do anything with pool-master before this happened.

 

One of the people in that thread just restarted their server. That seemed my easiest route rather than changing and

renaming files on the server.

 

So that's what I did, just hit the console and did a reboot. Took a while (forgot how long it can take to restart

one of these) but it eventually came back up and rejoined the pool automatically.

 

Though I'm still without network stats on the console directly, but xsconsole shows all the network and

management info.

 

Thank you.

Keith