Xenserver 7.2 lost NICs, management console

[Sushrut Pavanaskar1709159899]

We have a small setup on a ACER ATC-710 desktop that is virtualized with Xenserver 7.2 and has 2-3 VMs. In doing some hardware changes and reverts (added and removed a graphics card), we have lost our NICs in Xenserver. The server boots fine but has no NIC configured or seen. I tried many xe pif-* commands, emergency network reset, etc. but the pif-scan sees/finds nothing, ifconfig -a only shows xenbr0 and lo but no eth0. The machine also has a wifi card that we have never used in Xen but that is not seen either. At this point, I am about to give up (reinstall) but I will like to keep VMs. If nothing else, how can I backup VMs to some removable drive or even local drive and use in some other Xenserver machine? I am not a system-admin or anything so please forgive my ignorance in stating the current state or doing things stupidly :)

[Tobias Kreidl]

Try this fix procedure and if it doesn't work, I have a slightly more involved alternative as a backup. Note sure why this keeps happening, but seems like a bad bug that has persisted across at least a couple of XS 7.X releases.

 

Option 1:
1) If HA is enabled on the pool, turn it off.

2) Check to see if the affected host is in emergency mode or not using:

xe host-is-in-emergency-mode

3) If the result is "true", then run the pool recover slaves command

4) xe pool-recover-slaves

Option 2:
1) It may be necessary to do an "xe host-emergency-management-reconfigure interface=ethX" (where X is the physical interface, so for example, eth4) to revert to a single primary network interface. A reboot may be needed at this point to be able to proceed.

2) Identify if there is a new pool master and note its UUID and IP address! Turn off HA on the pool: xe pool-ha-disable

3) Turn off HA on the individual host:  xe host-emergency-ha-disable --force

4) Make sure the host knows who the real current master is: xe pool-emergency-reset-master master-address=10.15.9.X (pick the right IP address for the current pool master!). A reboot may be necessary after this point.

5) Scan for network interfaces: xe pif-scan host-uuid=(UUID-of-current-host)

6) Verify that the PIFs (physical interfaces) and various other network definitions show up: xe pif-list

7) Restart the toolstack: xe-toolstack-restart

8) Reboot and verify that all proper network definitions show up.

9) Not that some host-specific configurations may have disappeared, in particular, storage-related (iSCSI and/or NFS) settings.

These will need to be re-created for each affected host.

Note: It may be possible to make use of "xe host-restore ..." to restore from a backup the host control domain. but we have not investigated that procedure at this point.

-=Tobias

 

[Sushrut Pavanaskar1709159899]

Hi Tobias, thanks for the reply. 

 

I tried both the procedures you mentioned. The host is not in emergency mode and when I tried to do the next thing, with eth0, (xe host-emergency-management-reconfigure interface=eth0) it said that the "pool is not in emergency mode" and did nothing. The eth0 is not even seen by XS, I guess. Also ifconfig -a shows only "lo", "ovs-systesm" sometimes and sometimes shows additionally "tap3.0", "vif3.0", and "xenbr0" but never eth0 or something. ALso tried the emergency management reconfigure with xenbr0 but same pool not in emergency thing. HA is not enabled on this pool. 

What is the other procedure you have?

 

Thanks again!

[Tobias Kreidl]

They are both there (1 and 2). That's all I've got!  Sometimes a reboot will just "work" as long as it's not the pool master.

Worst case, you may have to re-configure everything by hand (and I've had to do that before, as well). Sorry, but really, this should not

be happening. Maybe if you report it to https://bugs.xenserver.org/ someone will take another look; I sent in a report a long time ago and

I don't think anything became of it.

 

-=Tobias

[Alan Lantz]

An emergency network reset with eth0 is usually enough to get networking going again.

 

--Alan--

 

[Tobias Kreidl]

Yes, but then you generally have to redo all your network customizations. The fact that servers lose their network configurations seemingly randomly is really scary and this has happened to me several times, with o other option than to reconfigure everything from scratch, including VLANs, bonds, iSCSI connections MTU settings, etc. It's a lot of work to have to do and it's easy to make a mistake. This just simply should not happen. The fact that is does means there's pretty much got to be a bug somewhere.

 

-=Tobias

[Richard Davies1709155471]

I have been having this issue since 7.0 and as late as yesterday with 7.6. I even tried xcp-ng. Same issue. I have determined that it is caused by a bios issue. If I use uefi machines I don’t have the problem but all of my dell c6100 machines do this. But not when using xen 6.5 or less. I know it is bios because it affects both onboard and addin NIC cards.