Jump to content
Welcome to our new Citrix community!
  • 0

All Windows Server 2008R2 SP1 VMs freeze on XenServer 6.5SP1


Andreas Jantz

Question

Hi,

 

we run a small pool of XenServer 6.5SP1, Patch Level XS65ESP1064 since two years with nearly no downtime nor any problems since yesterday morning.

 

Xenserver (poolmaster) was responsive accessible. All 4 VMs were frozens, no ping no shutdown or restart were possible. It was not psosible to bring the host ordered to a shutdown or restart. After this has happened, everything came back without a problem, Windows was claiming that a unexpected shutdown was made at 05:37:05.

 

I have found in the Daemon.log some sort of problems starting at that time but I do not know what this mean. Maybe someone could bring some light into that matter or direct me to the right place.

Jul  8 05:33:37 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:33:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:33:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:33:51 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:34:22 IT1XENMASTER1 last message repeated 10 times
Jul  8 05:35:36 IT1XENMASTER1 last message repeated 18 times
Jul  8 05:35:37 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:35:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:35:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:35:51 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:36:22 IT1XENMASTER1 last message repeated 10 times
Jul  8 05:37:07 IT1XENMASTER1 last message repeated 13 times
Jul  8 05:37:08 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 3 Submitted requests 
Jul  8 05:37:08 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoPauseDataPath:Target[2] : Waiting for 8 Submitted requests 
Jul  8 05:37:08 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 11 Submitted requests 
Jul  8 05:37:09 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 15 Submitted requests 
Jul  8 05:37:22 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:37:22 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:37:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 1 Submitted requests 
Jul  8 05:37:37 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:37:37 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:37:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:37:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:37:52 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:37:53 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:38:02 IT1XENMASTER1 tapdisk[4047]: tapdisk-syslog: 17 messages, 1622 bytes, xmits: 18, failed: 0, dropped: 0
Jul  8 05:38:02 IT1XENMASTER1 tapdisk[13894]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:02 IT1XENMASTER1 tapdisk[15321]: tapdisk-syslog: 17 messages, 1642 bytes, xmits: 18, failed: 0, dropped: 0
Jul  8 05:38:02 IT1XENMASTER1 tapdisk[20681]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:02 IT1XENMASTER1 tapdisk[23161]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[13729]: tapdisk-syslog: 16 messages, 1463 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[14065]: tapdisk-syslog: 16 messages, 1474 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[15480]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[20891]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[23512]: tapdisk-syslog: 16 messages, 1462 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:04 IT1XENMASTER1 tapdisk[23683]: tapdisk-syslog: 16 messages, 1464 bytes, xmits: 17, failed: 0, dropped: 0
Jul  8 05:38:07 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:38:38 IT1XENMASTER1 last message repeated 10 times
Jul  8 05:38:53 IT1XENMASTER1 last message repeated 5 times
Jul  8 05:39:01 IT1XENMASTER1 tapdisk[4047]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:01 IT1XENMASTER1 tapdisk[13894]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:01 IT1XENMASTER1 tapdisk[15321]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:01 IT1XENMASTER1 tapdisk[20681]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:01 IT1XENMASTER1 tapdisk[23161]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[13729]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[14065]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[15480]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[20891]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[23512]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:04 IT1XENMASTER1 tapdisk[23683]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:39:07 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:39:38 IT1XENMASTER1 last message repeated 10 times
Jul  8 05:39:38 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:39:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:39:43 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|2||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:39:52 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:39:53 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:40:01 IT1XENMASTER1 tapdisk[4047]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:01 IT1XENMASTER1 tapdisk[13894]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:01 IT1XENMASTER1 tapdisk[15321]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:01 IT1XENMASTER1 tapdisk[20681]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:01 IT1XENMASTER1 tapdisk[23161]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[13729]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[14065]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[15480]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[20891]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[23512]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:04 IT1XENMASTER1 tapdisk[23683]: tapdisk-syslog: 0 messages, 0 bytes, xmits: 0, failed: 0, dropped: 0
Jul  8 05:40:07 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:40:08 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoPauseDataPath:Target[0] : 3/3 Submitted requests left (180001 iterrations) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|PdoReset:Target[0] : backend has 3 outstanding requests after a PdoReset 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: disconnecting domid=63, devid=768 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: 63/768, ring=0x1c4b650: disconnect from ring with 3 pending requests 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA800B5FF890 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA800B9FC330 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA800B4F3640 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {64491487-f6e0-4fad-ba22-036574baf16a} 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/63/768) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: received 'sring connect' message (uuid = 6) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: connecting VBD 6 domid=63, devid=768, pool (null), evt 13 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: ring 0x1c4c210 connected 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: sending 'sring connect rsp' message (uuid = 6) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 0 Submitted requests 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoPauseDataPath:Target[0] : 0/0 Submitted requests left (0 iterrations) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: disconnecting domid=63, devid=768 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {64491487-f6e0-4fad-ba22-036574baf16a} 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/63/768) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: received 'sring connect' message (uuid = 6) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: connecting VBD 6 domid=63, devid=768, pool (null), evt 13 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: ring 0x1c4ce10 connected 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[15321]: sending 'sring connect rsp' message (uuid = 6) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-63[15621]: XENVBD|__PdoPauseDataPath:Target[1] : Waiting for 7 Submitted requests 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoPauseDataPath:Target[2] : 8/8 Submitted requests left (180001 iterrations) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|PdoReset:Target[2] : backend has 8 outstanding requests after a PdoReset 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : ENABLED ----> CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: disconnecting domid=60, devid=5632 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: 60/5632, ring=0xae2a10: disconnect from ring with 8 pending requests 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800E70AB50 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800FAFB7C0 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800E4386B0 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA8010D0A390 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800D811810 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA8011EB6010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800EDA2B50 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoCleanupSubmittedReqs:Target[2] : SubmittedReq 0xFFFFFA800DE39010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : CLOSING ----> CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : CLOSED ----> ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|FrontendWriteUsage:Target[2] : NOT_DUMP NOT_HIBER NOT_PAGE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|PdoUpdateInquiryData:Target[2] : VDI-UUID = {64cf2367-2cb5-4bf1-a314-e80a9c69a4ca} 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|FrontendPrepare:Target[2] : BackendId 0 (/local/domain/0/backend/vbd3/60/5632) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|FrontendPrepare:Target[2] : RingFeatures REMOVABLE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state PREPARED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: received 'sring connect' message (uuid = 9) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: connecting VBD 9 domid=60, devid=5632, pool (null), evt 14 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: ring 0xae3610 connected 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[13894]: sending 'sring connect rsp' message (uuid = 9) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|FrontendConnect:Target[2] : VBDFeatures BARRIER  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__FrontendSetState:Target[2] : in state ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-60[14232]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 3 Submitted requests 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoPauseDataPath:Target[0] : 11/11 Submitted requests left (180001 iterrations) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|PdoReset:Target[0] : backend has 11 outstanding requests after a PdoReset 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: disconnecting domid=62, devid=768 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: 62/768, ring=0x1f29e10: disconnect from ring with 11 pending requests 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EAC05B50 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EB403010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EA4C3010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EB402010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EAC0C010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EB415010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EB408010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EAC11010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EAC00010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EC801010 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFE000EAC0D690 -> FAILED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {401c9b95-6cfa-45db-b9f6-a07dae3507fd} 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/62/768) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: received 'sring connect' message (uuid = 2) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: connecting VBD 2 domid=62, devid=768, pool (null), evt 21 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: ring 0x1f2aa10 connected 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: sending 'sring connect rsp' message (uuid = 2) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 0 Submitted requests 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoPauseDataPath:Target[0] : 0/0 Submitted requests left (0 iterrations) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: disconnecting domid=62, devid=768 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {401c9b95-6cfa-45db-b9f6-a07dae3507fd} 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/62/768) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: received 'sring connect' message (uuid = 2) 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: connecting VBD 2 domid=62, devid=768, pool (null), evt 21 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: ring 0x1f2b610 connected 
Jul  8 05:40:09 IT1XENMASTER1 tapdisk[23161]: sending 'sring connect rsp' message (uuid = 2) 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:09 IT1XENMASTER1 qemu-dm-62[23821]: XENVBD|__PdoPauseDataPath:Target[1] : Waiting for 4 Submitted requests 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoPauseDataPath:Target[0] : 15/15 Submitted requests left (180001 iterrations) 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|PdoReset:Target[0] : backend has 15 outstanding requests after a PdoReset 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: disconnecting domid=61, devid=768 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: 61/768, ring=0x1fae650: disconnect from ring with 15 pending requests 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005664B50 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA80022FC1D0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA800211C610 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA80022421D0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA800263BA00 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005342B50 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005B014F0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005BE3B50 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA80024C6B50 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8001E74B50 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA80020E91D0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA80026AA2F0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8002A10010 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005CF7010 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8002A641F0 -> FAILED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {8690747a-19fe-4ece-8cfe-7e7ade991308} 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/61/768) 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: received 'sring connect' message (uuid = 4) 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: connecting VBD 4 domid=61, devid=768, pool (null), evt 13 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: ring 0x1faf210 connected 
Jul  8 05:40:10 IT1XENMASTER1 tapdisk[4047]: sending 'sring connect rsp' message (uuid = 4) 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:10 IT1XENMASTER1 qemu-dm-61[4203]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 1 Submitted requests 
Jul  8 05:40:22 IT1XENMASTER1 mpathalert: [debug|IT1XENMASTER1|1||mscgen] mpathalert=>xapi [label="(XML)"];
Jul  8 05:40:23 IT1XENMASTER1 last message repeated 3 times
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[0] : 1/1 Submitted requests left (180001 iterrations) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|PdoReset:Target[0] : backend has 1 outstanding requests after a PdoReset 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: disconnecting domid=1, devid=768 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: 1/768, ring=0x1140a10: disconnect from ring with 1 pending requests 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoCleanupSubmittedReqs:Target[0] : SubmittedReq 0xFFFFFA8005351010 -> FAILED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {899dce60-7187-4dcc-b1fc-6b30525dc26c} 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/1/768) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: received 'sring connect' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: connecting VBD 0 domid=1, devid=768, pool (null), evt 13 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: ring 0x1142410 connected 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: sending 'sring connect rsp' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[0] : Waiting for 0 Submitted requests 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[0] : 0/0 Submitted requests left (0 iterrations) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|NotifierDpc:Target[0] : Paused, 0 outstanding 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : ENABLED ----> CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: disconnecting domid=1, devid=768 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : CLOSING ----> CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : CLOSED ----> ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendWriteUsage:Target[0] : DUMP NOT_HIBER PAGE 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|PdoUpdateInquiryData:Target[0] : VDI-UUID = {899dce60-7187-4dcc-b1fc-6b30525dc26c} 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[0] : BackendId 0 (/local/domain/0/backend/vbd3/1/768) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[0] : RingFeatures  
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state PREPARED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: received 'sring connect' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: connecting VBD 0 domid=1, devid=768, pool (null), evt 13 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: ring 0x1143010 connected 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20681]: sending 'sring connect rsp' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendConnect:Target[0] : VBDFeatures BARRIER  
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[0] : in state ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[1] : Waiting for 0 Submitted requests 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__PdoPauseDataPath:Target[1] : 0/0 Submitted requests left (0 iterrations) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : ENABLED ----> CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: received 'sring disconnect' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: disconnecting domid=1, devid=832 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: sending 'sring disconnect rsp' message (uuid = 0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state CLOSING 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : CLOSING ----> CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state CLOSED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : CLOSED ----> ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendWriteUsage:Target[1] : NOT_DUMP NOT_HIBER NOT_PAGE 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|PdoUpdateInquiryData:Target[1] : VDI-UUID = {0db4e757-7377-4d17-80f1-b682c8854049} 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[1] : BackendId 0 (/local/domain/0/backend/vbd3/1/832) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendPrepare:Target[1] : RingFeatures REMOVABLE 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state PREPARED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: received 'sring connect' message (uuid = 1) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: connecting VBD 1 domid=1, devid=832, pool (null), evt 14 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: ring 0x2273650 connected 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: sending 'sring connect rsp' message (uuid = 1) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 0: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 9 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:guest_copy2: 1/832, ring=0x2273650: req 7449327995607622699: failed to grant-copy segment 0: -1 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_parse_request: 1/832, ring=0x2273650: req 7449327995607622699: failed to copy from guest: Input/output error 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 0: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 20 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 9 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310017: invalid request type 43 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310059: bad number of segments in request (164) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310017: invalid request type 43 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310059: bad number of segments in request (164) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310017: invalid request type 43 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310059: bad number of segments in request (164) 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 30 times
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 7449327999876310017: invalid request type 43 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 last message repeated 20 times
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|FrontendConnect:Target[1] : VBDFeatures BARRIER  
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state CONNECTED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 
Jul  8 05:40:37 IT1XENMASTER1 qemu-dm-1[21216]: XENVBD|__FrontendSetState:Target[1] : in state ENABLED 
Jul  8 05:40:37 IT1XENMASTER1 tapdisk[20891]: tap-err:tapdisk_xenblkif_make_vbd_request: 1/832, ring=0x2273650: req 4294901760: bad number of segments in request (0) 

 

Thanks

Andreas

Link to comment

7 answers to this question

Recommended Posts

  • 0

How often in those last two years were the servers restarted ? If they were up the full two years it could just simply

be a resource issue that has hopefully since been resolved.  Since 6.5 is no longer supported you need to work on

a migration plan.

 

--Alan--

 

Link to comment
  • 0

Hi Alan,

 

they were running 343 days without a reboot. The VMs more frequent due to Windows Security Patch requirements. However, I would have expected from a reliable VM Host like XenServer to be alerted that a reboot of a host might by necessary due to "XYZ", instead of "crashing" the VMs.

 

Updating to a newer iteration of XenServer is currently no option, but is in sight for next year.

 

BR Andreas

 

By The way, what would be the trigger to decide a reboot would be good?

Edited by Andreas Jantz
Added question...
Link to comment
  • 0

To me, it looks like a storage performance or storage I/O queue issue.  Have you seen any degradation in storage performance? Run "iostat -x 10" to see if you have any obvious bottlenecks as well as "top" to make sure your dom0 instance hasn't run out of memory or CPU resources.  Do you have snapshots that perhaps require being cleaned up?

The error messages could also relate to storage hardware issues, such as bad sectors and failed attempts to perform a rewrite. Disk fragmentation, while less serious in Linux systems, as well as disk integrity (running "fsck" periodically is IMO always a good idea, certainly more than once a year).

 

-=Tobias

Link to comment
  • 0

Now, after the restart of this XenServer host I do not see any  bottleneck. "iostat -x 10" shows me %iowait less than 0.01 and %idle 95.0 in average. We do not take any snapshots from these 4(!) VMs.

 

Regarding disk integrity, are we talking about the dom0 disk(s) to run a fsck or are we talking about the iSCSI virtual disk storage were all VMs are located?

We do run two SSDs as RAID1 for Dom0 and a RAID0 with 5 600GB 10K SAS 12G for the SR. (I know it is a very specific setup, but the RAID0 is fully replicated to a second host as hot Standby.)

 

So which filesystem we should run regularly fsk?

 

Thank you Tobias for your advise. It is always a pleasure for me, that you still support these kind of people like me.

 

Link to comment
  • 0

Andreas,

I'd advise fsck'ing everything on a semi-regular basis, just because if you don't, things can get corrupted over time and you'd never know until you reboot!

you can manipulate the fsck schedule so that it doesn't always happen when you reboot. For some tips on this, see for example: https://linuxconfig.org/how-to-force-fsck-to-check-filesystem-after-system-reboot-on-linux

Note that ext3 file systems will fsck way slower than ext4. Some of our old file systems would literally take hours, hence we converted them to ext4 and it then took just a few minutes. I'm still surprised Citrix hasn't totally switched to ext4 on newer releases.

 

Hey, always glad to lend a hand whenever possible!

 

Cheers,

-=Tobias

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...