XenCenter can't enable HA because HA enable is in progress

[Cory Farinella]

I have a Xen cluster of 4 nodes running Xenserver 7.2. One of the nodes failed and HA worked and restarted the VM's on other nodes. I then removed the failed node ran some xe commands and got it in a state that Xencenter could manage it. I then disabled HA and brought the 4th node back into the pool. I then tried to start HA which ran for several hours. then the 4th node was failed and Xencenter could not connect to it again. I am able to ssh to the failed nodes management address and ran this command: xe host-is-in-emergency-mode which returned true. so I then tried to stop or cancel the HA enable task from Xencenter. I then was going to configure HA with just the 3 nodes and not worry about the 4th node at this time. The problem I am having is that I can't start HA it gives the following error in Xencenter: "The operation could not be performed because HA enable is in progress" but I do not see this process in progress anywhere. How do I get the Pool into a state that I can turn on HA again with out it thinking there is still a running task to start it? If I try and disable or enable it from the command line I get the same error "The operation could not be performed because HA enable is in progress"

[Tobias Kreidl]

I am guessing shutting off HA isn't working from XenCenter, either.

Is there a VM that was on the failed host that might need to be taken out of the HA list that's causing problmes, maybe?

Or, how about forcing the pool master to a different host to see if that clears things up?

[Tobias Kreidl]

You'll need to probably do an emergency HA disabling.  I'd run on that failed host if you can:

xe host-emergency-ha-disable force=true
followed by
xe-toolstack-restart

 

and on the pool master run:

xe pool-ha-disable

 

See if things clear up OK, then re-enable HA on your pool, assuming three hosts can handle the load OK until you can get your fourth ode back in or you have enough capacity still for another host to fail.

 

-=Tobias

[Cory Farinella]

OK, first off thanks for responding so quickly. Now here is what happens when I do the instructions you've laid out:

On the failed node(xen-mgmt-00):

[root@xen-mgmt-00 ~]# xe host-is-in-emergency-mode  
true
[root@xen-mgmt-00 ~]# xe-toolstack-restart
Executing xe-toolstack-restart
done.
[root@xen-mgmt-00 ~]#

and then on the pool master(xen-mgmt-02):

[root@xen-mgmt-02 ~]# xe pool-ha-disable
The operation could not be performed because HA enable is in progress
[root@xen-mgmt-02 ~]#

This has been going on for at least 10+ hours that "HA enable is in progress" so what do I do  next? I tried

xe pool-sync-database

and that doesn't help either. Any help would be welcomed.

 

Thanks,

 

Cory

 

[Tobias Kreidl]

Run "xe task-list" and see if the HA enabled task is still active (which it seems to be). If so, get its UUID and use "xe task-cancel uuid=(UUID-of-task) force=true" to try to stop it. If the host is in emergency mode, you will need to fix that in which case there are a couple of options; see here and follow the outlined procedures, depending on which is your pool master:

https://vdinotes.wordpress.com/2013/01/13/xenserver-get-out-of-emergency-mode/

 

-=Tobias

[Cory Farinella]

Hi, I tried that and it seems to not have any tasks in the list but it still won't let me turn on or off HA

root@xen-mgmt-02 ~]# xe task-list
[root@xen-mgmt-02 ~]# xe pool-ha-enable
The operation could not be performed because HA enable is in progress
[root@xen-mgmt-02 ~]# xe pool-ha-disable
The operation could not be performed because HA enable is in progress
[root@xen-mgmt-02 ~]# xe host-emergency-ha-disable --force
The operation could not be performed because HA is not enabled on the Pool
[root@xen-mgmt-02 ~]#

 

[Tobias Kreidl]

Did you try a reboot? I know, not an elegant option, but there's a communication link hung up and it needs to be somehow cleared.  Is the "bad" host still in emergency mode?

If there's still this inter-communication confusion there, another option might be to forcibly kick it out of the pool and re-join it (as long as you first save any data on any local SRs on that host, which will be destroyed if the host is ejected from the pool!).

 

-=Tobias

[Cory Farinella]

so it seems like the pool has already ejected the bad host but I am still having the problems stated above I can't seem to start or stop HA on the Pool.
 

[root@xen-mgmt-02 ~]# xe host-list params=uuid,name-label,host-metrics-live
uuid ( RO)                 : c8d0b657-b8b2-4114-9155-6b1f04e3accf
           name-label ( RW): xen-mgmt-01
    host-metrics-live ( RO): true


uuid ( RO)                 : 749ddcd1-7cc6-412f-ba19-040d0ca247cd
           name-label ( RW): xen-mgmt-02
    host-metrics-live ( RO): true


uuid ( RO)                 : c9d24331-f349-4486-a3a4-e4137e71cfb6
           name-label ( RW): xen-mgmt-03
    host-metrics-live ( RO): true


[root@xen-mgmt-02 ~]#

 

[Cory Farinella]

Hello Tobias,

 

So your suggestion of electing a new master worked. I ran the following

xe pool-designate-new-master host-uuid=c8d0b657-b8b2-4114-9155-6b1f04e3accf

then I was able to reconnect to the pool in Xencenter and turn on HA via the UI and it took a min. or two and then it was enabled.

[root@xen-mgmt-02 ~]# xe pool-param-get param-name=ha-enabled uuid=9eecaae2-7f49-86b5-1e8e-c0a25319ae80
true
[root@xen-mgmt-02 ~]#

Thanks for your help with this. I really appreciate it.

 

Cory

 

[Tobias Kreidl]

Awesome! Please mark as resolved so others may benefit, as well. So glad it worked out for you, Cory!

 

-=Tobias