Can only SHELL into Xenserver cannot stop or force stop VM.

[Jim Van Etten]

I have been having some issues with my Xenserver losing network connection and now all of a sudden I cannot connect with XenCenter and one of the VM's is showing as running but no matter what steps I follow I cannot shut it down. Even from the console it shows <No Operations Available>. 

 

This really sucks. I have been running Xenserver for 6 years now and really depend on it for testing and some server functions.

 

I am attaching my xscontainer.log and xensource.log which are fresh for the latest bootup. 

 

Dell In.

PowerEdge 2950

XenServer 7.6.2

Logical CPUs 8

Populated CPU Sockets 2

2 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz

Total memory               32768 MB

 

 

xscontainer.log xensource.log

[Alan Lantz]

Since its just a standalone server you can rule out most poolwide issues. How is timesync between

management workstation and the XenServer? Also, are you using the new drive layout (or I guess

a better question is % disk space free).  Is you management interface bonded? If so you might want

to go back to a single interface to see if the issue is at that level. Getting your money's worth out

of that old Dell 2950, aren't you !  I got rid of my lasts few a couple of years ago. 

 

--Alan--

 

[Tobias Kreidl]

Did you run a forced VM power reset? https://support.citrix.com/article/CTX220777

[Jim Van Etten]

18 hours ago, Alan Lantz said:

Since its just a standalone server you can rule out most poolwide issues. How is timesync between

management workstation and the XenServer? Also, are you using the new drive layout (or I guess

a better question is % disk space free).  Is you management interface bonded? If so you might want

to go back to a single interface to see if the issue is at that level. Getting your money's worth out

of that old Dell 2950, aren't you !  I got rid of my lasts few a couple of years ago. 

 

--Alan--

 

timesync is fine

 

freespace is

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        288M     0  288M   0% /dev
tmpfs           298M   12K  298M   1% /dev/shm
tmpfs           298M  776K  297M   1% /run
tmpfs           298M     0  298M   0% /sys/fs/cgroup
/dev/sda1       3.4G  2.4G  909M  73% /
/dev/loop0       45M   45M     0 100% /var/xen/xc-install
xenstore        298M     0  298M   0% /var/lib/xenstored
tmpfs            60M     0   60M   0% /run/user/0
 

[Jim Van Etten]

18 hours ago, Tobias Kreidl said:

Did you run a forced VM power reset? https://support.citrix.com/article/CTX220777

 

Tried all those commands. Just hangs on the destroy_domain

 

list_domains shows:

id |                                 uuid |  state
 0 | 00000000-0000-0000-0000-000000000000 |     R 

 

 

 

[Tobias Kreidl]

Hmmm, that would indicate there are no VMs running and just dom0. Try rebooting the server?

[Jim Van Etten]

21 minutes ago, Tobias Kreidl said:

Hmmm, that would indicate there are no VMs running and just dom0. Try rebooting the server?

 

I have multiple times hoping it would straighten out. :-)

[Tobias Kreidl]

Did you look at stats from ethtool and or ifconfig? Maybe your cables are bad or there's a problem on the network switch port?

[Boby John1709155536]

Hope you have the latest driver as well? Ref : https://support.citrix.com/article/CTX257603 

when the XenCenter isnt connected - I assume that SSH works fine ? 

[Jim Van Etten]

2 minutes ago, Boby John1709155536 said:

Hope you have the latest driver as well? Ref : https://support.citrix.com/article/CTX257603 

when the XenCenter isnt connected - I assume that SSH works fine ? 

yes ssh works no problem.

[Jim Van Etten]

20 hours ago, Tobias Kreidl said:

Did you look at stats from ethtool and or ifconfig? Maybe your cables are bad or there's a problem on the network switch port?

All network connections seem fine. I can ssh to the server.

[Boby John1709155536]

Might not be very easy to troubleshoot over forums.

1. Are you able to update the driver?
2. try running xe vm-reset-powerstat --multiple --force 
3. Also try running xl list and using the ID number , try xl shutdown <ID> and it still doesnt work try xl destroy <ID>
4. Also look for link down or link up messages in kern.log

[Jim Van Etten]

3 hours ago, Boby John1709155536 said:

Might not be very easy to troubleshoot over forums.

1. Are you able to update the driver?
2. try running xe vm-reset-powerstat --multiple --force 
3. Also try running xl list and using the ID number , try xl shutdown <ID> and it still doesnt work try xl destroy <ID>
4. Also look for link down or link up messages in kern.log

 

xe vm-reset-powerstate --multiple --force hangs and does nothing.

xl list also hangs.

 

I am getting a lot of these errors in my kern.log which I have attached.

 

Sep 16 09:18:09 xenserver kernel: [  832.640684] oxenstored[4312]: segfault at 405640 ip 0000000000472850 sp 00007fff69299f38 error 7 in oxenstored[400000+b6000]
Sep 16 09:18:09 xenserver kernel: [  832.741835] oxenstored[4328]: segfault at 405640 ip 0000000000472850 sp 00007fff0ca86ba8 error 7 in oxenstored[400000+b6000]
Sep 16 09:18:09 xenserver kernel: [  832.812049] oxenstored[4342]: segfault at 405640 ip 0000000000472850 sp 00007fff3f7a3158 error 7 in oxenstored[400000+b6000]
Sep 16 09:18:09 xenserver kernel: [  832.935278] oxenstored[4357]: segfault at 405640 ip 0000000000472850 sp 00007ffc4721e2c8 error 7 in oxenstored[400000+b6000]
Sep 16 09:18:10 xenserver kernel: [  833.071610] oxenstored[4372]: segfault at 405640 ip 0000000000472850 sp 00007ffc55f3d5e8 error 7 in oxenstored[400000+b6000]
Sep 16 09:20:09 xenserver kernel: [  952.923658] oxenstored[4474]: segfault at 405640 ip 0000000000472850 sp 00007ffc6f5a0a98 error 7 in oxenstored[400000+b6000]
Sep 16 09:20:09 xenserver kernel: [  953.014401] oxenstored[4489]: segfault at 405640 ip 0000000000472850 sp 00007ffff3fd91f8 error 7 in oxenstored[400000+b6000]
Sep 16 09:20:10 xenserver kernel: [  953.109483] oxenstored[4503]: segfault at 405640 ip 0000000000472850 sp 00007ffefafb20c8 error 7 in oxenstored[400000+b6000]
Sep 16 09:20:10 xenserver kernel: [  953.181232] oxenstored[4517]: segfault at 405640 ip 0000000000472850 sp 00007fff3d93c228 error 7 in oxenstored[400000+b6000]
Sep 16 09:20:10 xenserver kernel: [  953.297553] oxenstored[4529]: segfault at 405640 ip 0000000000472850 sp 00007ffd0e4ea198 error 7 in oxenstored[400000+b6000]
 

kern.log

[Tobias Kreidl]

You may have a corrupted OS given all the errors you are reporting from various command. When you reboot, do you see any major errors show up or in .var/log/messages (or after running dmesg)?

 

-=Tobias

[Jim Van Etten]

4 minutes ago, Tobias Kreidl said:

You may have a corrupted OS given all the errors you are reporting from various command. When you reboot, do you see any major errors show up or in .var/log/messages (or after running dmesg)?

 

-=Tobias

Did a grep on dmesg for error:

 

[    0.028149] Could not initialize VPMU for cpu 0, error -95
[   15.008962] oxenstored[1428]: segfault at 405640 ip 0000000000472850 sp 00007fff20084e88 error 7 in oxenstored[400000+b6000]
[   17.254826] oxenstored[1755]: segfault at 405640 ip 0000000000472850 sp 00007fff1ecc71a8 error 7 in oxenstored[400000+b6000]
[   17.349994] oxenstored[1769]: segfault at 405640 ip 0000000000472850 sp 00007ffd3905c998 error 7 in oxenstored[400000+b6000]
[   17.425183] oxenstored[1787]: segfault at 405640 ip 0000000000472850 sp 00007ffe95cfa358 error 7 in oxenstored[400000+b6000]
[   17.545562] oxenstored[1801]: segfault at 405640 ip 0000000000472850 sp 00007ffce085e6f8 error 7 in oxenstored[400000+b6000]
[  110.903689] oxenstored[2602]: segfault at 405640 ip 0000000000472850 sp 00007ffd073acb48 error 7 in oxenstored[400000+b6000]
[  111.000232] oxenstored[2620]: segfault at 405640 ip 0000000000472850 sp 00007ffff8fdef88 error 7 in oxenstored[400000+b6000]
[  111.096511] oxenstored[2633]: segfault at 405640 ip 0000000000472850 sp 00007ffdcc910638 error 7 in oxenstored[400000+b6000]
[  111.219505] oxenstored[2648]: segfault at 405640 ip 0000000000472850 sp 00007fffda9fafe8 error 7 in oxenstored[400000+b6000]
[  112.390948] oxenstored[2698]: segfault at 405640 ip 0000000000472850 sp 00007ffc3da8dad8 error 7 in oxenstored[400000+b6000]
[  231.166469] oxenstored[3196]: segfault at 405640 ip 0000000000472850 sp 00007fffbf89df18 error 7 in oxenstored[400000+b6000]
[  231.258235] oxenstored[3211]: segfault at 405640 ip 0000000000472850 sp 00007ffe380c6548 error 7 in oxenstored[400000+b6000]
[  231.350246] oxenstored[3224]: segfault at 405640 ip 0000000000472850 sp 00007fffebf3cfb8 error 7 in oxenstored[400000+b6000]
[  231.445421] oxenstored[3239]: segfault at 405640 ip 0000000000472850 sp 00007ffee654fa98 error 7 in oxenstored[400000+b6000]
[  231.571685] oxenstored[3253]: segfault at 405640 ip 0000000000472850 sp 00007ffc889ef838 error 7 in oxenstored[400000+b6000]
[  351.555552] oxenstored[3726]: segfault at 405640 ip 0000000000472850 sp 00007fffbcc30158 error 7 in oxenstored[400000+b6000]
[  351.629850] oxenstored[3746]: segfault at 405640 ip 0000000000472850 sp 00007ffd39cda888 error 7 in oxenstored[400000+b6000]
[  351.716279] oxenstored[3760]: segfault at 405640 ip 0000000000472850 sp 00007ffefa158168 error 7 in oxenstored[400000+b6000]
[  351.824470] oxenstored[3775]: segfault at 405640 ip 0000000000472850 sp 00007ffc36de5d68 error 7 in oxenstored[400000+b6000]
[  351.957611] oxenstored[3791]: segfault at 405640 ip 0000000000472850 sp 00007ffcebdf1168 error 7 in oxenstored[400000+b6000]
[  471.894390] oxenstored[3847]: segfault at 405640 ip 0000000000472850 sp 00007ffd1e8d0088 error 7 in oxenstored[400000+b6000]
[  471.961513] oxenstored[3861]: segfault at 405640 ip 0000000000472850 sp 00007ffdfc14f538 error 7 in oxenstored[400000+b6000]
[  472.042554] oxenstored[3876]: segfault at 405640 ip 0000000000472850 sp 00007ffd4973c498 error 7 in oxenstored[400000+b6000]
[  472.133248] oxenstored[3892]: segfault at 405640 ip 0000000000472850 sp 00007fff94758008 error 7 in oxenstored[400000+b6000]
[  472.280896] oxenstored[3904]: segfault at 405640 ip 0000000000472850 sp 00007ffcc7586948 error 7 in oxenstored[400000+b6000]
[  592.147387] oxenstored[3996]: segfault at 405640 ip 0000000000472850 sp 00007ffe2acc60d8 error 7 in oxenstored[400000+b6000]
[  592.232713] oxenstored[4011]: segfault at 405640 ip 0000000000472850 sp 00007fff357c47a8 error 7 in oxenstored[400000+b6000]
[  592.331519] oxenstored[4025]: segfault at 405640 ip 0000000000472850 sp 00007ffc8ac9b388 error 7 in oxenstored[400000+b6000]
[  592.463196] oxenstored[4042]: segfault at 405640 ip 0000000000472850 sp 00007ffd640352b8 error 7 in oxenstored[400000+b6000]
[  592.623453] oxenstored[4055]: segfault at 405640 ip 0000000000472850 sp 00007ffef3ec4148 error 7 in oxenstored[400000+b6000]
[  712.399821] oxenstored[4167]: segfault at 405640 ip 0000000000472850 sp 00007fff9b51b468 error 7 in oxenstored[400000+b6000]
[  712.489472] oxenstored[4182]: segfault at 405640 ip 0000000000472850 sp 00007ffcd9a54a78 error 7 in oxenstored[400000+b6000]
[  712.586039] oxenstored[4193]: segfault at 405640 ip 0000000000472850 sp 00007ffc139ebfe8 error 7 in oxenstored[400000+b6000]
[  712.698988] oxenstored[4204]: segfault at 405640 ip 0000000000472850 sp 00007ffcfb2fa8b8 error 7 in oxenstored[400000+b6000]
[  712.839866] oxenstored[4220]: segfault at 405640 ip 0000000000472850 sp 00007fff9450ad38 error 7 in oxenstored[400000+b6000]
[  832.640684] oxenstored[4312]: segfault at 405640 ip 0000000000472850 sp 00007fff69299f38 error 7 in oxenstored[400000+b6000]
[  832.741835] oxenstored[4328]: segfault at 405640 ip 0000000000472850 sp 00007fff0ca86ba8 error 7 in oxenstored[400000+b6000]
[  832.812049] oxenstored[4342]: segfault at 405640 ip 0000000000472850 sp 00007fff3f7a3158 error 7 in oxenstored[400000+b6000]
[  832.935278] oxenstored[4357]: segfault at 405640 ip 0000000000472850 sp 00007ffc4721e2c8 error 7 in oxenstored[400000+b6000]
[  833.071610] oxenstored[4372]: segfault at 405640 ip 0000000000472850 sp 00007ffc55f3d5e8 error 7 in oxenstored[400000+b6000]
[  952.923658] oxenstored[4474]: segfault at 405640 ip 0000000000472850 sp 00007ffc6f5a0a98 error 7 in oxenstored[400000+b6000]
[  953.014401] oxenstored[4489]: segfault at 405640 ip 0000000000472850 sp 00007ffff3fd91f8 error 7 in oxenstored[400000+b6000]
[  953.109483] oxenstored[4503]: segfault at 405640 ip 0000000000472850 sp 00007ffefafb20c8 error 7 in oxenstored[400000+b6000]
[  953.181232] oxenstored[4517]: segfault at 405640 ip 0000000000472850 sp 00007fff3d93c228 error 7 in oxenstored[400000+b6000]
[  953.297553] oxenstored[4529]: segfault at 405640 ip 0000000000472850 sp 00007ffd0e4ea198 error 7 in oxenstored[400000+b6000]
[ 1073.169963] oxenstored[4585]: segfault at 405640 ip 0000000000472850 sp 00007ffc178a8198 error 7 in oxenstored[400000+b6000]
[ 1073.269044] oxenstored[4597]: segfault at 405640 ip 0000000000472850 sp 00007fff7ac20b58 error 7 in oxenstored[400000+b6000]
[ 1073.372634] oxenstored[4612]: segfault at 405640 ip 0000000000472850 sp 00007ffdd9d06408 error 7 in oxenstored[400000+b6000]
[ 1073.478994] oxenstored[4625]: segfault at 405640 ip 0000000000472850 sp 00007fff23ff92e8 error 7 in oxenstored[400000+b6000]
[ 1073.613318] oxenstored[4638]: segfault at 405640 ip 0000000000472850 sp 00007fff43d1bd28 error 7 in oxenstored[400000+b6000]
[ 1164.144590] oxenstored[4679]: segfault at 405640 ip 0000000000472850 sp 00007ffccc0758b8 error 7 in oxenstored[400000+b6000]
[ 1164.851369] oxenstored[4693]: segfault at 405640 ip 0000000000472850 sp 00007ffd55f08c68 error 7 in oxenstored[400000+b6000]
[ 1164.954172] oxenstored[4707]: segfault at 405640 ip 0000000000472850 sp 00007ffd0473e7d8 error 7 in oxenstored[400000+b6000]
[ 1165.729746] oxenstored[4722]: segfault at 405640 ip 0000000000472850 sp 00007ffdfd1c4ac8 error 7 in oxenstored[400000+b6000]
[ 1165.862431] oxenstored[4736]: segfault at 405640 ip 0000000000472850 sp 00007ffda8e4f958 error 7 in oxenstored[400000+b6000]
[ 1284.420707] oxenstored[4799]: segfault at 405640 ip 0000000000472850 sp 00007ffdb4192478 error 7 in oxenstored[400000+b6000]
[ 1284.491564] oxenstored[4813]: segfault at 405640 ip 0000000000472850 sp 00007ffd2c64ab68 error 7 in oxenstored[400000+b6000]
[ 1284.570161] oxenstored[4827]: segfault at 405640 ip 0000000000472850 sp 00007ffe4d70d5d8 error 7 in oxenstored[400000+b6000]
[ 1284.690198] oxenstored[4844]: segfault at 405640 ip 0000000000472850 sp 00007ffc7fdfe2e8 error 7 in oxenstored[400000+b6000]
[ 1284.817528] oxenstored[4856]: segfault at 405640 ip 0000000000472850 sp 00007ffeafbdaa38 error 7 in oxenstored[400000+b6000]
[ 1404.673523] oxenstored[4913]: segfault at 405640 ip 0000000000472850 sp 00007fff5d7d5e08 error 7 in oxenstored[400000+b6000]
[ 1404.746354] oxenstored[4927]: segfault at 405640 ip 0000000000472850 sp 00007fff44268df8 error 7 in oxenstored[400000+b6000]
[ 1404.835323] oxenstored[4942]: segfault at 405640 ip 0000000000472850 sp 00007ffe13989118 error 7 in oxenstored[400000+b6000]
[ 1404.934164] oxenstored[4955]: segfault at 405640 ip 0000000000472850 sp 00007ffddd4faab8 error 7 in oxenstored[400000+b6000]
[ 1405.042943] oxenstored[4971]: segfault at 405640 ip 0000000000472850 sp 00007ffc4c5593a8 error 7 in oxenstored[400000+b6000]
[ 1524.915115] oxenstored[5029]: segfault at 405640 ip 0000000000472850 sp 00007ffe97b938b8 error 7 in oxenstored[400000+b6000]
[ 1525.010183] oxenstored[5043]: segfault at 405640 ip 0000000000472850 sp 00007ffe409cb948 error 7 in oxenstored[400000+b6000]
[ 1525.101340] oxenstored[5058]: segfault at 405640 ip 0000000000472850 sp 00007ffd60d97fc8 error 7 in oxenstored[400000+b6000]
[ 1525.235181] oxenstored[5075]: segfault at 405640 ip 0000000000472850 sp 00007ffd0608b2e8 error 7 in oxenstored[400000+b6000]
[ 1525.367804] oxenstored[5088]: segfault at 405640 ip 0000000000472850 sp 00007ffed0d5acd8 error 7 in oxenstored[400000+b6000]
[ 1645.472839] oxenstored[5174]: segfault at 405640 ip 0000000000472850 sp 00007ffe4ab898a8 error 7 in oxenstored[400000+b6000]
[ 1645.541982] oxenstored[5189]: segfault at 405640 ip 0000000000472850 sp 00007fff8e353e98 error 7 in oxenstored[400000+b6000]
[ 1645.633508] oxenstored[5202]: segfault at 405640 ip 0000000000472850 sp 00007fffa4a22fd8 error 7 in oxenstored[400000+b6000]
[ 1645.761630] oxenstored[5216]: segfault at 405640 ip 0000000000472850 sp 00007fff625b4d78 error 7 in oxenstored[400000+b6000]
[ 1645.912440] oxenstored[5228]: segfault at 405640 ip 0000000000472850 sp 00007ffc6ee10ee8 error 7 in oxenstored[400000+b6000]
[ 1765.416465] oxenstored[5283]: segfault at 405640 ip 0000000000472850 sp 00007ffe166b5b98 error 7 in oxenstored[400000+b6000]
[ 1765.504625] oxenstored[5297]: segfault at 405640 ip 0000000000472850 sp 00007fff3d5ef288 error 7 in oxenstored[400000+b6000]
[ 1765.578264] oxenstored[5310]: segfault at 405640 ip 0000000000472850 sp 00007ffdf9772f28 error 7 in oxenstored[400000+b6000]
[ 1765.716802] oxenstored[5324]: segfault at 405640 ip 0000000000472850 sp 00007ffd3e16f308 error 7 in oxenstored[400000+b6000]
[ 1765.857631] oxenstored[5338]: segfault at 405640 ip 0000000000472850 sp 00007ffce208aa08 error 7 in oxenstored[400000+b6000]
[ 1856.559082] oxenstored[5403]: segfault at 405640 ip 0000000000472850 sp 00007fff87ff24f8 error 7 in oxenstored[400000+b6000]
[ 1856.676249] oxenstored[5419]: segfault at 405640 ip 0000000000472850 sp 00007ffe7e164428 error 7 in oxenstored[400000+b6000]
[ 1856.781293] oxenstored[5433]: segfault at 405640 ip 0000000000472850 sp 00007fff5ee50288 error 7 in oxenstored[400000+b6000]
[ 1856.885715] oxenstored[5448]: segfault at 405640 ip 0000000000472850 sp 00007ffd13eff638 error 7 in oxenstored[400000+b6000]
[ 1857.045093] oxenstored[5463]: segfault at 405640 ip 0000000000472850 sp 00007ffe65874188 error 7 in oxenstored[400000+b6000]
[ 1976.143729] oxenstored[5518]: segfault at 405640 ip 0000000000472850 sp 00007ffcdd02abb8 error 7 in oxenstored[400000+b6000]
[ 1976.245955] oxenstored[5533]: segfault at 405640 ip 0000000000472850 sp 00007ffe70dae408 error 7 in oxenstored[400000+b6000]
[ 1976.337843] oxenstored[5548]: segfault at 405640 ip 0000000000472850 sp 00007fff15bfc5e8 error 7 in oxenstored[400000+b6000]
[ 1976.467495] oxenstored[5562]: segfault at 405640 ip 0000000000472850 sp 00007ffcc66b8278 error 7 in oxenstored[400000+b6000]
[ 1976.609592] oxenstored[5578]: segfault at 405640 ip 0000000000472850 sp 00007fff03bf8348 error 7 in oxenstored[400000+b6000]
[ 2096.420476] oxenstored[5634]: segfault at 405640 ip 0000000000472850 sp 00007ffd07402c98 error 7 in oxenstored[400000+b6000]
[ 2096.508129] oxenstored[5649]: segfault at 405640 ip 0000000000472850 sp 00007ffe26bc62d8 error 7 in oxenstored[400000+b6000]
[ 2096.624823] oxenstored[5663]: segfault at 405640 ip 0000000000472850 sp 00007ffc0e8e4018 error 7 in oxenstored[400000+b6000]
[ 2096.731704] oxenstored[5674]: segfault at 405640 ip 0000000000472850 sp 00007ffde8594618 error 7 in oxenstored[400000+b6000]
[ 2096.866767] oxenstored[5685]: segfault at 405640 ip 0000000000472850 sp 00007ffc285501e8 error 7 in oxenstored[400000+b6000]
[ 2216.639016] oxenstored[5758]: segfault at 405640 ip 0000000000472850 sp 00007ffda2967b68 error 7 in oxenstored[400000+b6000]
[ 2216.738813] oxenstored[5772]: segfault at 405640 ip 0000000000472850 sp 00007ffde05697c8 error 7 in oxenstored[400000+b6000]
[ 2216.809231] oxenstored[5786]: segfault at 405640 ip 0000000000472850 sp 00007ffeb630e458 error 7 in oxenstored[400000+b6000]
[ 2216.914349] oxenstored[5804]: segfault at 405640 ip 0000000000472850 sp 00007fff72e54e28 error 7 in oxenstored[400000+b6000]
[ 2217.032360] oxenstored[5819]: segfault at 405640 ip 0000000000472850 sp 00007ffc758607e8 error 7 in oxenstored[400000+b6000]
[ 2336.915103] oxenstored[5873]: segfault at 405640 ip 0000000000472850 sp 00007fff6205d5f8 error 7 in oxenstored[400000+b6000]
[ 2336.998556] oxenstored[5888]: segfault at 405640 ip 0000000000472850 sp 00007ffe6309e008 error 7 in oxenstored[400000+b6000]
[ 2337.093843] oxenstored[5902]: segfault at 405640 ip 0000000000472850 sp 00007fff27aa2598 error 7 in oxenstored[400000+b6000]
[ 2337.256869] oxenstored[5919]: segfault at 405640 ip 0000000000472850 sp 00007ffd61e63a58 error 7 in oxenstored[400000+b6000]
[ 2337.385723] oxenstored[5930]: segfault at 405640 ip 0000000000472850 sp 00007fffbc968ae8 error 7 in oxenstored[400000+b6000]
[ 2457.148515] oxenstored[6004]: segfault at 405640 ip 0000000000472850 sp 00007ffccb55e308 error 7 in oxenstored[400000+b6000]
[ 2457.239817] oxenstored[6019]: segfault at 405640 ip 0000000000472850 sp 00007ffc98f1c6d8 error 7 in oxenstored[400000+b6000]
[ 2457.321758] oxenstored[6034]: segfault at 405640 ip 0000000000472850 sp 00007ffe00ad07b8 error 7 in oxenstored[400000+b6000]
[ 2457.456819] oxenstored[6047]: segfault at 405640 ip 0000000000472850 sp 00007fff961a62f8 error 7 in oxenstored[400000+b6000]
[ 2457.591563] oxenstored[6061]: segfault at 405640 ip 0000000000472850 sp 00007ffc85461a58 error 7 in oxenstored[400000+b6000]
[ 2577.420475] oxenstored[6124]: segfault at 405640 ip 0000000000472850 sp 00007ffef0da3258 error 7 in oxenstored[400000+b6000]
[ 2577.521907] oxenstored[6139]: segfault at 405640 ip 0000000000472850 sp 00007ffea70deb18 error 7 in oxenstored[400000+b6000]
[ 2577.614027] oxenstored[6151]: segfault at 405640 ip 0000000000472850 sp 00007ffe083ddb98 error 7 in oxenstored[400000+b6000]
[ 2577.713460] oxenstored[6163]: segfault at 405640 ip 0000000000472850 sp 00007fff27740c88 error 7 in oxenstored[400000+b6000]
[ 2577.844358] oxenstored[6176]: segfault at 405640 ip 0000000000472850 sp 00007ffcd6116b88 error 7 in oxenstored[400000+b6000]
[ 2697.651737] oxenstored[6235]: segfault at 405640 ip 0000000000472850 sp 00007ffe9b572fd8 error 7 in oxenstored[400000+b6000]
[ 2697.759061] oxenstored[6249]: segfault at 405640 ip 0000000000472850 sp 00007ffedaedd208 error 7 in oxenstored[400000+b6000]
[ 2697.874265] oxenstored[6264]: segfault at 405640 ip 0000000000472850 sp 00007fffb11b1e18 error 7 in oxenstored[400000+b6000]
[ 2697.982821] oxenstored[6280]: segfault at 405640 ip 0000000000472850 sp 00007ffcaf5ce5a8 error 7 in oxenstored[400000+b6000]
[ 2698.123864] oxenstored[6295]: segfault at 405640 ip 0000000000472850 sp 00007ffd6cf31fa8 error 7 in oxenstored[400000+b6000]
[ 2817.899309] oxenstored[6365]: segfault at 405640 ip 0000000000472850 sp 00007ffce91a8c58 error 7 in oxenstored[400000+b6000]
[ 2818.017188] oxenstored[6378]: segfault at 405640 ip 0000000000472850 sp 00007ffc467f8908 error 7 in oxenstored[400000+b6000]
[ 2818.117164] oxenstored[6392]: segfault at 405640 ip 0000000000472850 sp 00007fffcf60fda8 error 7 in oxenstored[400000+b6000]
[ 2818.247900] oxenstored[6405]: segfault at 405640 ip 0000000000472850 sp 00007ffcd6e4b028 error 7 in oxenstored[400000+b6000]
[ 2818.379269] oxenstored[6418]: segfault at 405640 ip 0000000000472850 sp 00007ffe5603c508 error 7 in oxenstored[400000+b6000]
[ 2938.141872] oxenstored[6505]: segfault at 405640 ip 0000000000472850 sp 00007ffc3be73788 error 7 in oxenstored[400000+b6000]
[ 2938.227502] oxenstored[6519]: segfault at 405640 ip 0000000000472850 sp 00007ffd4bd51d18 error 7 in oxenstored[400000+b6000]
[ 2938.322930] oxenstored[6531]: segfault at 405640 ip 0000000000472850 sp 00007ffedb621188 error 7 in oxenstored[400000+b6000]
[ 2938.495745] oxenstored[6546]: segfault at 405640 ip 0000000000472850 sp 00007fff65ed5bb8 error 7 in oxenstored[400000+b6000]
[ 2938.647489] oxenstored[6558]: segfault at 405640 ip 0000000000472850 sp 00007fff26794eb8 error 7 in oxenstored[400000+b6000]

[Tobias Kreidl]

You may have a hard drive issue, based on the errors shown above, like a failing drive.