how to delete a VM running in memory but not listed by vm-list command

[Yiping Zhang]

Hi, List:

 

I am in a very weird situation after live storage migrating a VM to a different pool. Soon after the migration,  we noticed that the migrated VM is showing symptoms of having conflict IP address on the network.  The VM is listed as running on the destination pool.  After we shut it down, its IP is still pingable.

 

Our network engineer identified the MAC address associated with the offending IP and traced it to the original hypervisor where the VM was running before the migration.  However,  neither XenCenter nor XE cli tool shows this VM as running on this hypervisor or even in this pool.  From output of xenstore-ls command  we found local domain 925 has matching name-label, IP address, and MAC address of the VM.  It seems that  we have a ghost VM running in memory!

 

For now,  I have deleted the vif925.0 from the bridge it is associated with using  following command:

 

brctl delif xapi5 vif925.0

 

This has disconnected the ghost VM from the network, thus making it not pingable and not interfering with the actual VM instance.  However,  the ghost VM seems still running memory.   How can I permanently destroy this ghost VM short of putting the hypervisor into mainteance mode and reboot it?

 

My pools are running XenServer release 6.5.0-90233c (xenenterprise), and are part of CloudStack set up.

 

Attached file is the relevant output from command "xenstore-ls -f"  about local domain 925.

 

Thanks,

 

Yiping

domain_925.log

[Olivier Lambert]

`xl destroy <domID>` will kill it on Xen side.

[Yiping Zhang]

A little more info:

 

Actually,  I have encountered this situation a few times before, always after live migration (with or without storage migrations).  But in early cases, I could still ssh into the ghost VM,  and shut it down from guestOS.  Today, we can only ping the ghost VM, but not ssh into it.  Therefore,  I need a way to destroy it from host.

 

Yiping

[Tobias Kreidl]

You should check the power-state of the VM and halt it if need be. If it's not showing up as not halted, you won't be able to delete it. There is an xe command to force the power state to halted: xe vm-reset-powerstate uuid=<UUID of VM> force=true

[Yiping Zhang]

Hi, Tobias:

 

As I mentioned earlier, this ghost VM does not show up in command output of "xe vm-list" at all, so there is no way to find out its UUID.

 

There are a few uuid values which are lists in output of "xenstore-ls -f" for this domain,  such as :

 

/local/domain/925/vm = "/vm/527b03fc-c03a-b800-3d47-d7b2b6819621"

/local/domain/925/unique-domain-id = "ab945092-f177-776e-e53e-734ba6fea1f9"

/xapi/925/private/vif/0/vif-uuid = "4b5ca1f8-b371-7c75-01b0-728785d158bf"

 

Using above uuid values in command  such as "xe vm-list uuid=" or "xe vif-list uuid="  gave no output at all. 

 

The only evidence that this VM or its components still reside in the host memory are:

* the VM shows up in xentop and it is still taking up some cpu cycles!
* vif925.0 is UP as shown by command "ifconfig vif925.0"

 

I really don't want to put this host into maintenance mode to reboot it because that would force all running VM to live migrate to other hosts in the pool, and we may end up with more of such very situations which I am trying to avoid/fix to begin with!

 

Thanks

[Yiping Zhang]

Wow,  that did it! 

 

Thanks a lot.

[Olivier Lambert]

You are welcome :) We start to know very well the platform since we forked it ;)