Jump to content
Welcome to our new Citrix community!
  • 0

how to delete a VM running in memory but not listed by vm-list command

Yiping Zhang


Hi, List:


I am in a very weird situation after live storage migrating a VM to a different pool. Soon after the migration,  we noticed that the migrated VM is showing symptoms of having conflict IP address on the network.  The VM is listed as running on the destination pool.  After we shut it down, its IP is still pingable.


Our network engineer identified the MAC address associated with the offending IP and traced it to the original hypervisor where the VM was running before the migration.  However,  neither XenCenter nor XE cli tool shows this VM as running on this hypervisor or even in this pool.  From output of xenstore-ls command  we found local domain 925 has matching name-label, IP address, and MAC address of the VM.  It seems that  we have a ghost VM running in memory!


For now,  I have deleted the vif925.0 from the bridge it is associated with using  following command:


brctl delif xapi5 vif925.0


This has disconnected the ghost VM from the network, thus making it not pingable and not interfering with the actual VM instance.  However,  the ghost VM seems still running memory.   How can I permanently destroy this ghost VM short of putting the hypervisor into mainteance mode and reboot it?


My pools are running XenServer release 6.5.0-90233c (xenenterprise), and are part of CloudStack set up.


Attached file is the relevant output from command "xenstore-ls -f"  about local domain 925.






Link to comment

6 answers to this question

Recommended Posts

  • 0

A little more info:


Actually,  I have encountered this situation a few times before, always after live migration (with or without storage migrations).  But in early cases, I could still ssh into the ghost VM,  and shut it down from guestOS.  Today, we can only ping the ghost VM, but not ssh into it.  Therefore,  I need a way to destroy it from host.



Link to comment
  • 0

Hi, Tobias:


As I mentioned earlier, this ghost VM does not show up in command output of "xe vm-list" at all, so there is no way to find out its UUID.


There are a few uuid values which are lists in output of "xenstore-ls -f" for this domain,  such as :


/local/domain/925/vm = "/vm/527b03fc-c03a-b800-3d47-d7b2b6819621"

/local/domain/925/unique-domain-id = "ab945092-f177-776e-e53e-734ba6fea1f9"

/xapi/925/private/vif/0/vif-uuid = "4b5ca1f8-b371-7c75-01b0-728785d158bf"


Using above uuid values in command  such as "xe vm-list uuid=" or "xe vif-list uuid="  gave no output at all. 


The only evidence that this VM or its components still reside in the host memory are:

* the VM shows up in xentop and it is still taking up some cpu cycles!
* vif925.0 is UP as shown by command "ifconfig vif925.0"


I really don't want to put this host into maintenance mode to reboot it because that would force all running VM to live migrate to other hosts in the pool, and we may end up with more of such very situations which I am trying to avoid/fix to begin with!



Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...