Jump to content
Welcome to our new Citrix community!
  • 0

xenserver 7.1: Memory_interface.Internal_error("Sys_error(\"Broken pipe\")")


Rainer Krienke

Question

Hello,

 

I run a Xenserver Pool that since yesterday shows a strange problem. Whenever a machine is rebooted on a dom0 with this problem it will not start on this dom0  which has this problem. It is also not possible to migrate a VM on a affected dom0 to another dom0 in the pool.  It also seems there is no difference what OS is running on the VM, it happens with Linux (Ubuntu, SuSE) as well as with Windows VMs.  It is however still possible to shutdown a  VM and start it on another dom0 in the pool if this dom0 doesn't also suffer from this problem.

 

What helps in this situation is to restart the dom0 Toolstack or of course as well to reboot the affected dom0.

 

The error message I see in XenCenter is the following:

"Failed","Starting VM 'myVMname'
Internal error: xenopsd internal error: Memory_interface.Internal_error("Sys_error(\"Broken pipe\")")
Time: 00:00:02","dom0Name","Jul 26, 2019 10:39 AM"

 

In /var/log/Xensource.log I see basically the same messages. See attached extract of xensource.log (myDom0Name is the name of the server where I tried to start the VM on).

 

The really strange thing is that I did not change anything on these dom0-hosts since quite a while. So there were no new patches installed on the dom0s in the last days. The VMs however change every day by OS-Updates and new VMs and others beeing deleted.

 

Any idea what the reason for this problem might be or how to find out more?

 

Thanks

Rainer

 

dom0-xensource.log

Link to comment

7 answers to this question

Recommended Posts

  • 0
On 26.7.2019 at 4:14 PM, Tobias Kreidl said:

Maybe you are out of disk space on that one XenServer host? You didn't mention what version you're running. Or it may be at least related to something like this: https://support.citrix.com/article/CTX137243

 

-=Tobias

 

I think the problem report you cited is different since it is storage related which is expressed by the error message. In my error message I do not see a hint towards a storage problem. I also checked the dom0 disk space  but there is no problem (16GB i n / are free, 2GB in /var/log are free).

The XenServer version I use is in the subject or are you talking about any other version?

 

One thing I changed last Friday  was resolv.conf on each dom0. Since we had a change in our nameservers where the old one (still in resolv.conf for citrix-servers) although still existent has now more load and additionally a new faster nameserver was added. So I replaced the old by the new faster one last Friday on all dom0s. Over the weekend nothing bad happened. Yesterday (Monday) one of the dom0s, the pool master  had the problem again but not all of dom0s like last Friday morning.

 

Another observation is that on my pool master the xapi daemon used up to 60% cpu according to top with an average of about 10% in the long run. This seems quite a lot to me which led me to the idea that my problem might be somehow load related (perhaps load on xapi), but this is not more then a guess.

 

Rainer

Link to comment
  • 0

Hello Boby,

I think that a single VM cannot be the problem since as a said I had this problem on all eight dom0s with about 60VMs. And over the weekend with non of the dom0s (after a Toolstack Restart on Friday) with the same number of VMs running.

Anyway: Is there a simple way to check if VM metadata is corrupt?

 

Thanks

Rainer

Link to comment
  • 0

You can dump the VM metadata by running the poorly documented command option:

 

xe vm-export metadata=true uuid=UUID-OF-VM) filename=/full_path/OUTPUT_FILE.XVA

 

You can then examine its content with:

 

tar –xf /full_path/OUTPUT_FILE.XVA

 

which should give you a file named something like ova.xml .

 

To be able to examine the output in a nicer format, instead run this:

 

tar -xOf /full_path/OUTPUT_FILE.XVA |  xmllint –format - >/tmp/output_VM.xml

 

Then, you can examine that XML file, which will be in a much nicer format.

 

This was all documented in an article I write for the xenserver.,org blog which got deleted along with most of the blogs, and has -- sadly -- still not yet been re-published on the https://blogs.citrix.com site.

 

-=Tobias

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...