Jump to content
Welcome to our new Citrix community!
  • 0

Migration VHD stuck on Xen-Center


Trang Wardhani

Question

Hi,


I'am sorry if this post was duplicate but i need urgent help.

Currently i'am using Xen-server ver. 7.1 with a pool consists of 3 host with HA enabled.

I tried to live migration VM's VHD from Shared storage (iSCSI) to Local Storage on a host-1 inside a Pool using Xen-Center.
But then connection got problem and the progress transfer was stuck for a few hours.
I realized that when i want to move another VM's to Host-1 the progress also become stuck.

 

I tried to collect data using xe task-list via console in every host (host-1,host-2,host-3), and there are some tasks pending duplicated.

 

Could anyone give me an idea how to solve this issue? maybe clearing the task or restart toolstack? any help would be appreciate

 

Thank You,

 

Below is the xe task-list

 

 

[root@svr-4 ~]# xe task-list
uuid ( RO)                : 01fe7393-3351-a7ed-126d-580cd02c4bdf
          name-label ( RO): Async.VDI.pool_migrate
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : b8cb243a-95bc-48db-a392-da691c50d068
          name-label ( RO): Connection to VM console
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

uuid ( RO)                : a3351c78-3c45-e2b1-6b7a-4518e1641c68
          name-label ( RO): Async.VM.migrate_send
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.025


uuid ( RO)                : 69072093-1f9d-fde0-3e7c-7b3c6b8050d8
          name-label ( RO): Async.VM.start
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.467

=================================================================

[root@svr-2 ~]# xe task-list 
uuid ( RO)                : 81e23198-d3f4-5c60-5c7b-bae97eb0ca40
          name-label ( RO): Connection to VM console
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : 01fe7393-3351-a7ed-126d-580cd02c4bdf
          name-label ( RO): Async.VDI.pool_migrate
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

uuid ( RO)                : a3351c78-3c45-e2b1-6b7a-4518e1641c68
          name-label ( RO): Async.VM.migrate_send
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.025


uuid ( RO)                : 69072093-1f9d-fde0-3e7c-7b3c6b8050d8
          name-label ( RO): Async.VM.start
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.467

=================================================================

[root@-svr-3 ~]# xe task-list 
uuid ( RO)                : 96b4929e-a24d-0e71-6de6-4f3d7ae676e4
          name-label ( RO): Connection to VM console
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : 01fe7393-3351-a7ed-126d-580cd02c4bdf
          name-label ( RO): Async.VDI.pool_migrate
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

uuid ( RO)                : a3351c78-3c45-e2b1-6b7a-4518e1641c68
          name-label ( RO): Async.VM.migrate_send
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.025


uuid ( RO)                : 69072093-1f9d-fde0-3e7c-7b3c6b8050d8
          name-label ( RO): Async.VM.start
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.467
 

Pool.PNG

progress bar.PNG

Link to comment

3 answers to this question

Recommended Posts

  • 0

Depending on the size of these it may take a while. Also, I find you never see more than 4 active migrations, even if you queue up a lot of them, so that the system doesn't get overwhelmed. Run "top" and make sure dom0 isn't saturated for memory and/or CPU cycles. Storage migrations take up a lot of resources.  Worst case, kill off the pending tasks and try them one-by-one, instead.

 

-=Tobias

  • Like 1
Link to comment
  • 0

Storage migrations are dangerous ! If your hosts are interrupted in anyway regardless of the reason they are suppose to fail in such a way as to prevent data loss. However, depending on your hotfix level and your luck you can lose data. You can keep an eye on performance monitor and see if you data really is transferring. As Tobias states, it takes a long time. I think the last time I tried and measured it was like 40GB/Hour. If you have something crazy like a 1TB disk you are trying to migrate it could take 24 hours which if going over the task will fail. This is a good reason to keep your disks as small as possible. But back to your dilemma, if you feel like your host isn't transferring data and you have a backup a restart of the host may be required to get it back to operating properly again. Trying to cancel tasks usually doesn't work well in XenServer unfortunately.

 

--Alan--

 

Link to comment
  • 0

I have a quite a similar issue, on xenserver 7.2.

 

I tried to move the 2 vdisks (25G each) for the same VM (system and the data partitions of the same linux box), from one Synology NAS to another (NFS mounts).

 

While it took 1.5 hours to move the first 25G (system) disk, the migration of the second one seems to be stuck at 100%.

 

The VM is still in running state, but it is not responding at all. Cannot ping it, and cannot access the console via xen-center. Have options to gracefully or forcefully restart or shut down the box, but not sure if this is a good idea?

 

The hypervisor says that the progress is 100% but the task in the xen-center is still hanging for more than 20 hours.

 

[root@hypervisor01 log]# xe task-list
uuid ( RO)                : a466bcf4-b7b8-c1ca-a033-d330d7a9b079
          name-label ( RO): Async.VDI.pool_migrate
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 1.000

 

This is what I see when I check the vdi-list:

 

#NEW_NAS (23.94GB)
uuid ( RO)                : ab5474a3-5e45-4641-ab46-c6aae57f0f66
          name-label ( RW): data_disk
    name-description ( RW): data storage
             sr-uuid ( RO): 46ef093e-8a23-b764-aa17-ed4bdcd8cd2f
        virtual-size ( RO): 26843545600
            sharable ( RO): false
           read-only ( RO): false

#OLD_NAS (146.3MB)
uuid ( RO)                : 0b461460-9714-4987-b5ad-0e40e26a80fd
          name-label ( RW): data_disk
    name-description ( RW): data storage
             sr-uuid ( RO): 6caa0c9d-f2ba-be49-0369-858293a0c135
        virtual-size ( RO): 26843545600
            sharable ( RO): false
           read-only ( RO): false

 

#NEW_NAS (19.32G)
uuid ( RO)                : 0f3b9732-9b36-4776-9ab3-4d8f6a273163
          name-label ( RW): sys_disk
    name-description ( RW): System disk
             sr-uuid ( RO): 46ef093e-8a23-b764-aa17-ed4bdcd8cd2f
        virtual-size ( RO): 26843545600
            sharable ( RO): false
           read-only ( RO): false

 

So it seems that the task almost finished, but there are still about 150MBs left on the old NAS, for some reason.

 

There is also what it looks a full copy of a data_disk on the OLD_NAS, in the same folder as the previously mentioned one:

 

ee9e0599-c43a-4357-b71a-d7c3197fd652.vhd (24.95GB).

 

uuid ( RO)                : ee9e0599-c43a-4357-b71a-d7c3197fd652
          name-label ( RW): base copy
    name-description ( RW):
             sr-uuid ( RO): 6caa0c9d-f2ba-be49-0369-858293a0c135
        virtual-size ( RO): 26843545600
            sharable ( RO): false
           read-only ( RO): true

 

But this one is not associated with my VM in any way, it seems?

 

And when I check the list of disks associated with this VM on the hypervisor, I see this:

 

[root@eqx-hypervisor01 log]# xe vm-disk-list vm=prd-vm-01
Disk 0 VBD:
uuid ( RO)             : 7c85119b-ba33-39fa-0562-bbcae684dccb
    vm-name-label ( RO): prd-vm-01
       userdevice ( RW): 1


Disk 0 VDI:
uuid ( RO)             : 0b461460-9714-4987-b5ad-0e40e26a80fd
       name-label ( RW): data_disk
    sr-name-label ( RO): VM02 @ OLD_NAS
     virtual-size ( RO): 26843545600


Disk 1 VBD:
uuid ( RO)             : 1131dc7b-9eb7-49a0-c7ad-f3955824b382
    vm-name-label ( RO): prd-vm-01
       userdevice ( RW): 0


Disk 1 VDI:
uuid ( RO)             : 0f3b9732-9b36-4776-9ab3-4d8f6a273163
       name-label ( RW): sys_disk
    sr-name-label ( RO): VM01 @ NEW_NAS
     virtual-size ( RO): 26843545600

 

So not really sure who to trust?

 

I read articles on different forums that say that cancelling the tasks that are moving storage is not a good idea and it might result in data loss.

Also, I read articles that say that restarting xen-toolstack is also a bad idea when there are tasks that are moving storage running.

 

Any ideas where to go from here?

 

I read on a few places on the forum that the task might timeout after 24 hours, but any ideas what should I do if it doesn't timeout?

 

Any help is appreciated.

 

Thank you in advance.

 

Best regards,

Adnan

 


 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...