Recovering VM after removing host from pool

Jerome Eichler1709155813 · November 11, 2020

Hi folks,

I recently removed one of my hosts from a pool.

During that, the local storage repository, which kept tons of lvms, got deleted.

I was able to recover the lvm structure successfully and all volumes are back and visible in the local storage.

However, on 4 volumes (unfortunately the most important ones), starting the VM is not successful:

Nov 11 14:18:54 xen02 SM: [1388] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:54 xen02 SM: [1388] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:54 xen02 SM: [1390] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:54 xen02 SM: [1390] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:54 xen02 SM: [1388] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:54 xen02 SM: [1390] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:55 xen02 SM: [1388]   pread SUCCESS
Nov 11 14:18:55 xen02 SM: [1390]   pread SUCCESS
Nov 11 14:18:55 xen02 SM: [1388] lock: acquired /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:55 xen02 SM: [1388] LVMCache: will initialize now
Nov 11 14:18:55 xen02 SM: [1388] LVMCache: refreshing
Nov 11 14:18:55 xen02 SM: [1388] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:55 xen02 SM: [1390] Failed to lock /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr on first attempt, blocked by PID 1388
Nov 11 14:18:56 xen02 SM: [1388]   pread SUCCESS
Nov 11 14:18:56 xen02 SM: [1388] lock: released /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:56 xen02 SM: [1390] lock: acquired /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:56 xen02 SM: [1388] Entering _checkMetadataVolume
Nov 11 14:18:56 xen02 SM: [1390] LVMCache: will initialize now
Nov 11 14:18:56 xen02 SM: [1390] LVMCache: refreshing
Nov 11 14:18:56 xen02 SM: [1390] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:56 xen02 SM: [1388] vdi_epoch_begin {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|ff0f00e8-eee2-4667-aec7-faac127f0d9b|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:795141e7-1ba1-47be-bf9a-94e14d722cb1', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'b474dd88-57b5-4d90-9f28-cba56f9facce', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:1d9a0761-4ee2-412a-a7ef-2fdacdb4b0b2', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': 'b474dd88-57b5-4d90-9f28-cba56f9facce'}
Nov 11 14:18:56 xen02 SM: [1390]   pread SUCCESS
Nov 11 14:18:56 xen02 SM: [1390] lock: released /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:56 xen02 SM: [1390] Entering _checkMetadataVolume
Nov 11 14:18:56 xen02 SM: [1390] vdi_epoch_begin {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|d1979848-296f-4c25-b64e-85aed78c8115|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:933db6de-1b3c-4365-9894-9953c790975e', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1f6e939a-5d21-4769-889d-c39ca542396c', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:34f00263-4104-4e2d-ae02-1ac783974c7f', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': '1f6e939a-5d21-4769-889d-c39ca542396c'}
Nov 11 14:18:57 xen02 SM: [1437] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:57 xen02 SM: [1437] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:57 xen02 SM: [1440] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:57 xen02 SM: [1440] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:57 xen02 SM: [1437] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:57 xen02 SM: [1440] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:57 xen02 SM: [1440]   pread SUCCESS
Nov 11 14:18:57 xen02 SM: [1440] Entering _checkMetadataVolume
Nov 11 14:18:57 xen02 SM: [1440] LVMCache: will initialize now
Nov 11 14:18:57 xen02 SM: [1440] LVMCache: refreshing
Nov 11 14:18:57 xen02 SM: [1437]   pread SUCCESS
Nov 11 14:18:57 xen02 SM: [1440] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:57 xen02 SM: [1437] Entering _checkMetadataVolume
Nov 11 14:18:57 xen02 SM: [1437] LVMCache: will initialize now
Nov 11 14:18:57 xen02 SM: [1437] LVMCache: refreshing
Nov 11 14:18:57 xen02 SM: [1437] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:58 xen02 SM: [1440]   pread SUCCESS
Nov 11 14:18:58 xen02 SM: [1440] vdi_attach {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|a5deeab3-ec77-447c-979f-48064f06bf0b|VDI.attach2', 'vdi_ref': 'OpaqueRef:795141e7-1ba1-47be-bf9a-94e14d722cb1', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': 'b474dd88-57b5-4d90-9f28-cba56f9facce', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:40a7a0ce-47f7-49f2-b441-d9a4d7a5fcaf', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_attach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': 'b474dd88-57b5-4d90-9f28-cba56f9facce'}
Nov 11 14:18:58 xen02 SM: [1440] lock: opening lock file /var/lock/sm/b474dd88-57b5-4d90-9f28-cba56f9facce/vdi
Nov 11 14:18:58 xen02 SM: [1437]   pread SUCCESS
Nov 11 14:18:58 xen02 SM: [1437] vdi_attach {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|4fde0d17-08ab-4f28-ab58-7069be1a9e19|VDI.attach2', 'vdi_ref': 'OpaqueRef:933db6de-1b3c-4365-9894-9953c790975e', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': '1f6e939a-5d21-4769-889d-c39ca542396c', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:926f3d1f-0ff7-46e5-96b9-96ba99c9eca3', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_attach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': '1f6e939a-5d21-4769-889d-c39ca542396c'}
Nov 11 14:18:58 xen02 SM: [1437] lock: opening lock file /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:18:58 xen02 SM: [1440] result: {'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEmI0NzRkZDg4LTU3YjUtNGQgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIGI0NzRkZDg4LTU3YjUtNGQ5MC05ZjI4LWNiYTU2ZjlmYWNjZSA=', 'vdi-uuid': 'b474dd88-57b5-4d90-9f28-cba56f9facce', 'mem-pool': 'ffde18af-117b-952c-3401-842915117b31'}}
Nov 11 14:18:58 xen02 SM: [1437] result: {'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEjFmNmU5MzlhLTVkMjEtNDcgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIDFmNmU5MzlhLTVkMjEtNDc2OS04ODlkLWMzOWNhNTQyMzk2YyA=', 'vdi-uuid': '1f6e939a-5d21-4769-889d-c39ca542396c', 'mem-pool': 'ffde18af-117b-952c-3401-842915117b31'}}
Nov 11 14:18:58 xen02 SM: [1542] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:58 xen02 SM: [1542] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:58 xen02 SM: [1542] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:58 xen02 SM: [1548] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:18:58 xen02 SM: [1548] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:58 xen02 SM: [1548] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:58 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:18:58 xen02 SM: [1548] Entering _checkMetadataVolume
Nov 11 14:18:58 xen02 SM: [1548] LVMCache: will initialize now
Nov 11 14:18:58 xen02 SM: [1542]   pread SUCCESS
Nov 11 14:18:58 xen02 SM: [1548] LVMCache: refreshing
Nov 11 14:18:58 xen02 SM: [1542] Entering _checkMetadataVolume
Nov 11 14:18:58 xen02 SM: [1548] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:58 xen02 SM: [1542] LVMCache: will initialize now
Nov 11 14:18:58 xen02 SM: [1542] LVMCache: refreshing
Nov 11 14:18:58 xen02 SM: [1542] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:59 xen02 SM: [1542]   pread SUCCESS
Nov 11 14:18:59 xen02 SM: [1542] vdi_activate {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|f638ba1e-2741-415d-b5e5-8beb54485bd9|VDI.activate', 'vdi_ref': 'OpaqueRef:795141e7-1ba1-47be-bf9a-94e14d722cb1', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': 'b474dd88-57b5-4d90-9f28-cba56f9facce', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:7740a125-6b92-4a21-949a-7e26e7972122', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_activate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': 'b474dd88-57b5-4d90-9f28-cba56f9facce'}
Nov 11 14:18:59 xen02 SM: [1542] lock: opening lock file /var/lock/sm/b474dd88-57b5-4d90-9f28-cba56f9facce/vdi
Nov 11 14:18:59 xen02 SM: [1542] blktap2.activate
Nov 11 14:18:59 xen02 SM: [1542] lock: acquired /var/lock/sm/b474dd88-57b5-4d90-9f28-cba56f9facce/vdi
Nov 11 14:18:59 xen02 SM: [1542] Adding tag to: b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:18:59 xen02 SM: [1542] Activate lock succeeded
Nov 11 14:18:59 xen02 SM: [1542] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:59 xen02 SM: [1542] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:59 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:18:59 xen02 SM: [1548] vdi_activate {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|7d4053a3-cb33-4541-b414-ce9f84fbbf9d|VDI.activate', 'vdi_ref': 'OpaqueRef:933db6de-1b3c-4365-9894-9953c790975e', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': '1f6e939a-5d21-4769-889d-c39ca542396c', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:fcc0097d-11f2-45e8-8dac-355948f67d54', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_activate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': '1f6e939a-5d21-4769-889d-c39ca542396c'}
Nov 11 14:18:59 xen02 SM: [1548] lock: opening lock file /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:18:59 xen02 SM: [1548] blktap2.activate
Nov 11 14:18:59 xen02 SM: [1548] lock: acquired /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:18:59 xen02 SM: [1548] Adding tag to: 1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:18:59 xen02 SM: [1548] Activate lock succeeded
Nov 11 14:18:59 xen02 SM: [1548] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:18:59 xen02 SM: [1548] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:59 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:18:59 xen02 SM: [1548] Entering _checkMetadataVolume
Nov 11 14:18:59 xen02 SM: [1548] LVMCache: will initialize now
Nov 11 14:18:59 xen02 SM: [1548] LVMCache: refreshing
Nov 11 14:18:59 xen02 SM: [1548] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:18:59 xen02 SM: [1542]   pread SUCCESS
Nov 11 14:18:59 xen02 SM: [1542] Entering _checkMetadataVolume
Nov 11 14:18:59 xen02 SM: [1542] LVMCache: will initialize now
Nov 11 14:18:59 xen02 SM: [1542] LVMCache: refreshing
Nov 11 14:18:59 xen02 SM: [1542] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:00 xen02 SM: [1542]   pread SUCCESS
Nov 11 14:19:00 xen02 SM: [1542] LVHDVDI.attach for b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] lock: opening lock file /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:b474dd88-57b5-4d90-9f28-cba56f9facce (0, 1) + (1, 0) => (1, 1)
Nov 11 14:19:00 xen02 SM: [1542] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:b474dd88-57b5-4d90-9f28-cba56f9facce set => (1, 1b)
Nov 11 14:19:00 xen02 SM: [1542] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] ['/usr/bin/vhd-util', 'query', '--debug', '-vsf', '-n', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce']
Nov 11 14:19:00 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:19:00 xen02 SM: [1548] LVHDVDI.attach for 1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] lock: opening lock file /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (0, 0) + (1, 0) => (1, 0)
Nov 11 14:19:00 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (1, 0b)
Nov 11 14:19:00 xen02 SM: [1548] ['/sbin/lvchange', '-ay', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c']
Nov 11 14:19:00 xen02 SM: [1542]   pread SUCCESS
Nov 11 14:19:00 xen02 SM: [1542] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:b474dd88-57b5-4d90-9f28-cba56f9facce (1, 1) + (-1, 0) => (0, 1)
Nov 11 14:19:00 xen02 SM: [1542] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:b474dd88-57b5-4d90-9f28-cba56f9facce set => (0, 1b)
Nov 11 14:19:00 xen02 SM: [1542] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] PhyLink(/dev/sm/phy/ffde18af-117b-952c-3401-842915117b31/b474dd88-57b5-4d90-9f28-cba56f9facce) -> /dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:00 xen02 SM: [1542] lock: opening lock file /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p
Nov 11 14:19:00 xen02 SM: [1542] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p
Nov 11 14:19:00 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:19:00 xen02 SM: [1548] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] ['/usr/bin/vhd-util', 'query', '--debug', '-vsf', '-n', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c']
Nov 11 14:19:00 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:19:00 xen02 SM: [1548] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:00 xen02 SM: [1548] ['/usr/bin/vhd-util', 'scan', '-f', '-m', 'VHD-1f6e939a-5d21-4769-889d-c39ca542396c', '-l', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31', '-a']
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'allocate']
Nov 11 14:19:01 xen02 SM: [1542]  = 0
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'spawn']
Nov 11 14:19:01 xen02 SM: [1542]  = 0
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'attach', '-p', '1753', '-m', '9']
Nov 11 14:19:01 xen02 SM: [1542]  = 0
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40']
Nov 11 14:19:01 xen02 SM: [1548]   pread SUCCESS
Nov 11 14:19:01 xen02 SM: [1548] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:01 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (1, 0) + (0, 1) => (1, 1)
Nov 11 14:19:01 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (1, 1b)
Nov 11 14:19:01 xen02 SM: [1548] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:01 xen02 SM: [1548] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:01 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (1, 1) + (-1, 0) => (0, 1)
Nov 11 14:19:01 xen02 SM: [1548] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (0, 1b)
Nov 11 14:19:01 xen02 SM: [1548] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:01 xen02 SM: [1548] PhyLink(/dev/sm/phy/ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c) -> /dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:01 xen02 SM: [1548] lock: opening lock file /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p
Nov 11 14:19:01 xen02 SM: [1548] Failed to lock /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p on first attempt, blocked by PID 1542
Nov 11 14:19:01 xen02 SM: [1542]  = 2
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'detach', '-p', '1753', '-m', '9']
Nov 11 14:19:01 xen02 SM: [1542]  = 0
Nov 11 14:19:01 xen02 SM: [1542] ***** ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40'] failed: status=2, pid=1755, errmsg=No such file or directory: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40'] failed: status=2, pid=1755, errmsg=No such file or directory
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 819, in launch_on_tap
Nov 11 14:19:01 xen02 SM: [1542]     TapCtl.open(pid, minor, _type, path, options)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 426, in open
Nov 11 14:19:01 xen02 SM: [1542]     cls._pread(args=args, input=input)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 295, in _pread
Nov 11 14:19:01 xen02 SM: [1542]     tapctl._wait(quiet)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 284, in _wait
Nov 11 14:19:01 xen02 SM: [1542]     raise self.CommandFailure(self.cmd, **info)
Nov 11 14:19:01 xen02 SM: [1542]
Nov 11 14:19:01 xen02 SM: [1542] ['/usr/sbin/tap-ctl', 'free', '-m', '9']
Nov 11 14:19:01 xen02 SM: [1542]  = 0
Nov 11 14:19:01 xen02 SM: [1542] Exception in activate/attach
Nov 11 14:19:01 xen02 SM: [1542] Removed host key host_OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5 for b474dd88-57b5-4d90-9f28-cba56f9facce
Nov 11 14:19:01 xen02 SM: [1542] ***** BLKTAP2:<function _activate_locked at 0x7fb821b3ced8>: EXCEPTION <class 'blktap2.TapdiskFailed'>, Tapdisk(vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce): ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40'] failed: status=2, pid=1755, errmsg=No such file or directory
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 84, in wrapper
Nov 11 14:19:01 xen02 SM: [1542]     ret = op(self, *args)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1625, in _activate_locked
Nov 11 14:19:01 xen02 SM: [1542]     dev_path = self._activate(sr_uuid, vdi_uuid, options)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1671, in _activate
Nov 11 14:19:01 xen02 SM: [1542]     self._get_pool_config(sr_uuid).get("mem-pool-size"))
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1323, in _tap_activate
Nov 11 14:19:01 xen02 SM: [1542]     options)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 850, in launch_on_tap
Nov 11 14:19:01 xen02 SM: [1542]     raise TapdiskFailed(cls.Arg(_type, path), ctl)
Nov 11 14:19:01 xen02 SM: [1542]
Nov 11 14:19:01 xen02 SM: [1542] lock: released /var/lock/sm/b474dd88-57b5-4d90-9f28-cba56f9facce/vdi
Nov 11 14:19:01 xen02 SM: [1542] ***** tapdisk failed exception: Tapdisk(vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce): ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40'] failed: status=2, pid=1755, errmsg=No such file or directory: EXCEPTION <class 'blktap2.TapdiskFailed'>, Tapdisk(vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce): ['/usr/sbin/tap-ctl', 'open', '-p', '1753', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce', '-t', '40'] failed: status=2, pid=1755, errmsg=No such file or directory
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Nov 11 14:19:01 xen02 SM: [1542]     return self._run_locked(sr)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Nov 11 14:19:01 xen02 SM: [1542]     rv = self._run(sr, target)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/SRCommand.py", line 269, in _run
Nov 11 14:19:01 xen02 SM: [1542]     writable, caching_params)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1565, in activate
Nov 11 14:19:01 xen02 SM: [1542]     if self._activate_locked(sr_uuid, vdi_uuid, options):
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 84, in wrapper
Nov 11 14:19:01 xen02 SM: [1542]     ret = op(self, *args)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1625, in _activate_locked
Nov 11 14:19:01 xen02 SM: [1542]     dev_path = self._activate(sr_uuid, vdi_uuid, options)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1671, in _activate
Nov 11 14:19:01 xen02 SM: [1542]     self._get_pool_config(sr_uuid).get("mem-pool-size"))
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 1323, in _tap_activate
Nov 11 14:19:01 xen02 SM: [1542]     options)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/blktap2.py", line 850, in launch_on_tap
Nov 11 14:19:01 xen02 SM: [1542]     raise TapdiskFailed(cls.Arg(_type, path), ctl)
Nov 11 14:19:01 xen02 SM: [1542]
Nov 11 14:19:01 xen02 SM: [1542] Raising exception [453, tapdisk experienced an error [opterr=No such file or directory]]
Nov 11 14:19:01 xen02 SM: [1542] ***** Local VHD on LVM: EXCEPTION <class 'SR.SROSError'>, tapdisk experienced an error [opterr=No such file or directory]
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/SRCommand.py", line 372, in run
Nov 11 14:19:01 xen02 SM: [1542]     ret = cmd.run(sr)
Nov 11 14:19:01 xen02 SM: [1542]   File "/opt/xensource/sm/SRCommand.py", line 125, in run
Nov 11 14:19:01 xen02 SM: [1542]     os.strerror(e.get_error().get_error_code()))
Nov 11 14:19:01 xen02 SM: [1542]
Nov 11 14:19:01 xen02 SM: [1548] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p
Nov 11 14:19:02 xen02 SM: [1548] ['/usr/sbin/tap-ctl', 'allocate']
Nov 11 14:19:02 xen02 SM: [1548]  = 0
Nov 11 14:19:02 xen02 SM: [1548] ['/usr/sbin/tap-ctl', 'spawn']
Nov 11 14:19:02 xen02 SM: [1548]  = 0
Nov 11 14:19:02 xen02 SM: [1548] ['/usr/sbin/tap-ctl', 'attach', '-p', '1838', '-m', '9']
Nov 11 14:19:02 xen02 SM: [1548]  = 0
Nov 11 14:19:02 xen02 SM: [1548] ['/usr/sbin/tap-ctl', 'open', '-p', '1838', '-m', '9', '-a', 'vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c', '-t', '40']
Nov 11 14:19:02 xen02 SM: [1548]  = 0
Nov 11 14:19:02 xen02 SM: [1548] Set scheduler to [noop] on [/sys/dev/block/254:9]
Nov 11 14:19:02 xen02 SM: [1548] tap.activate: Launched Tapdisk(vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c, pid=1838, minor=9, state=R)
Nov 11 14:19:02 xen02 SM: [1548] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/lvchange-p
Nov 11 14:19:02 xen02 SM: [1548] DeviceNode(/dev/sm/backend/ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c) -> /dev/xen/blktap-2/tapdev9
Nov 11 14:19:02 xen02 SM: [1548] lock: released /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:19:03 xen02 SM: [1925] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:19:03 xen02 SM: [1925] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:19:03 xen02 SM: [1925] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:03 xen02 SM: [1928] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:19:03 xen02 SM: [1928] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:19:03 xen02 SM: [1928] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:03 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:03 xen02 SM: [1928] Entering _checkMetadataVolume
Nov 11 14:19:03 xen02 SM: [1928] LVMCache: will initialize now
Nov 11 14:19:03 xen02 SM: [1928] LVMCache: refreshing
Nov 11 14:19:03 xen02 SM: [1928] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:03 xen02 SM: [1925]   pread SUCCESS
Nov 11 14:19:03 xen02 SM: [1925] Entering _checkMetadataVolume
Nov 11 14:19:03 xen02 SM: [1925] LVMCache: will initialize now
Nov 11 14:19:03 xen02 SM: [1925] LVMCache: refreshing
Nov 11 14:19:03 xen02 SM: [1925] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:04 xen02 SM: [1925]   pread SUCCESS
Nov 11 14:19:04 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:04 xen02 SM: [1925] vdi_detach {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|b2db6c62-209d-47e4-b03e-9b231690dd1c|VDI.detach', 'vdi_ref': 'OpaqueRef:795141e7-1ba1-47be-bf9a-94e14d722cb1', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'b474dd88-57b5-4d90-9f28-cba56f9facce', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:cee14085-3868-4829-9b3c-36b2f4ed8336', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': 'b474dd88-57b5-4d90-9f28-cba56f9facce'}
Nov 11 14:19:04 xen02 SM: [1925] lock: opening lock file /var/lock/sm/b474dd88-57b5-4d90-9f28-cba56f9facce/vdi
Nov 11 14:19:04 xen02 SM: [1928] vdi_deactivate {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|45de6595-b93b-416e-bba9-5892e661e495|VDI.deactivate', 'vdi_ref': 'OpaqueRef:933db6de-1b3c-4365-9894-9953c790975e', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1f6e939a-5d21-4769-889d-c39ca542396c', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:d542b763-9b7c-44f8-93e1-c0aa22989d8a', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': '1f6e939a-5d21-4769-889d-c39ca542396c'}
Nov 11 14:19:04 xen02 SM: [1928] lock: opening lock file /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:19:04 xen02 SM: [1928] blktap2.deactivate
Nov 11 14:19:04 xen02 SM: [1928] lock: acquired /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:19:04 xen02 SM: [1928] ['/usr/sbin/tap-ctl', 'close', '-p', '1838', '-m', '9']
Nov 11 14:19:04 xen02 SM: [1928]  = 0
Nov 11 14:19:04 xen02 SM: [1928] ['/usr/sbin/tap-ctl', 'detach', '-p', '1838', '-m', '9']
Nov 11 14:19:04 xen02 SM: [1928]  = 0
Nov 11 14:19:04 xen02 SM: [1928] ['/usr/sbin/tap-ctl', 'free', '-m', '9']
Nov 11 14:19:04 xen02 SM: [1928]  = 0
Nov 11 14:19:04 xen02 SM: [1928] tap.deactivate: Shut down Tapdisk(vhd:/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c, pid=1838, minor=9, state=R)
Nov 11 14:19:04 xen02 SM: [1928] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:19:04 xen02 SM: [1928] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:05 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:05 xen02 SM: [1928] Entering _checkMetadataVolume
Nov 11 14:19:05 xen02 SM: [1928] LVMCache: will initialize now
Nov 11 14:19:05 xen02 SM: [1928] LVMCache: refreshing
Nov 11 14:19:05 xen02 SM: [1928] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:05 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:05 xen02 SM: [1928] LVHDVDI.detach for 1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] lock: opening lock file /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (0, 1) + (1, 0) => (1, 1)
Nov 11 14:19:05 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (1, 1b)
Nov 11 14:19:05 xen02 SM: [1928] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] ['/usr/bin/vhd-util', 'query', '--debug', '-vsf', '-n', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c']
Nov 11 14:19:05 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:05 xen02 SM: [1928] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:05 xen02 SM: [1928] ['/usr/bin/vhd-util', 'scan', '-f', '-m', 'VHD-1f6e939a-5d21-4769-889d-c39ca542396c', '-l', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31', '-a']
Nov 11 14:19:06 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:06 xen02 SM: [1928] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:06 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (1, 1) + (-1, 0) => (0, 1)
Nov 11 14:19:06 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (0, 1b)
Nov 11 14:19:06 xen02 SM: [1928] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:06 xen02 SM: [1928] lock: acquired /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:06 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c (0, 1) + (0, -1) => (0, 0)
Nov 11 14:19:06 xen02 SM: [1928] Refcount for lvm-ffde18af-117b-952c-3401-842915117b31:1f6e939a-5d21-4769-889d-c39ca542396c set => (0, 0b)
Nov 11 14:19:06 xen02 SM: [1928] ['/sbin/lvchange', '-an', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c']
Nov 11 14:19:07 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:07 xen02 SM: [1928] ['/sbin/dmsetup', 'status', 'VG_XenStorage--ffde18af--117b--952c--3401--842915117b31-VHD--1f6e939a--5d21--4769--889d--c39ca542396c']
Nov 11 14:19:07 xen02 SM: [1928]   pread SUCCESS
Nov 11 14:19:07 xen02 SM: [1928] lock: released /var/lock/sm/lvm-ffde18af-117b-952c-3401-842915117b31/1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:07 xen02 SM: [1928] Removed host key host_OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5 for 1f6e939a-5d21-4769-889d-c39ca542396c
Nov 11 14:19:07 xen02 SM: [1928] lock: released /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi
Nov 11 14:19:07 xen02 SM: [2097] lock: opening lock file /var/lock/sm/ffde18af-117b-952c-3401-842915117b31/sr
Nov 11 14:19:07 xen02 SM: [2097] LVMCache created for VG_XenStorage-ffde18af-117b-952c-3401-842915117b31
Nov 11 14:19:07 xen02 SM: [2097] ['/sbin/vgs', '--readonly', 'VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:08 xen02 SM: [2097]   pread SUCCESS
Nov 11 14:19:08 xen02 SM: [2097] Entering _checkMetadataVolume
Nov 11 14:19:08 xen02 SM: [2097] LVMCache: will initialize now
Nov 11 14:19:08 xen02 SM: [2097] LVMCache: refreshing
Nov 11 14:19:08 xen02 SM: [2097] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31']
Nov 11 14:19:08 xen02 SM: [2097]   pread SUCCESS
Nov 11 14:19:08 xen02 SM: [2097] vdi_detach {'sr_uuid': 'ffde18af-117b-952c-3401-842915117b31', 'subtask_of': 'DummyRef:|74b8f8c7-b7ee-42d5-b629-d0ad561d80d5|VDI.detach', 'vdi_ref': 'OpaqueRef:933db6de-1b3c-4365-9894-9953c790975e', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1f6e939a-5d21-4769-889d-c39ca542396c', 'host_ref': 'OpaqueRef:a9245f28-b347-40c0-a346-32fef75eade5', 'session_ref': 'OpaqueRef:359e3080-d9a1-4474-9e88-4d7d2d3cf0a6', 'device_config': {'device': '/dev/disk/by-id/scsi-3600050e0a4eacc00b9610000c21e0000-part3', 'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:c5cd8c8a-c873-4bb6-91a8-d20c460007d6', 'vdi_uuid': '1f6e939a-5d21-4769-889d-c39ca542396c'}
Nov 11 14:19:08 xen02 SM: [2097] lock: opening lock file /var/lock/sm/1f6e939a-5d21-4769-889d-c39ca542396c/vdi

I see that the device/file in question is shows as "no such file or directory", however, when I look for that one, it is there:

[14:20 xen02 ~]# ls -alh /dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce
lrwxrwxrwx 1 root root 113 11. Nov 13:23 /dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-b474dd88-57b5-4d90-9f28-cba56f9facce -> /dev/mapper/VG_XenStorage--ffde18af--117b--952c--3401--842915117b31-VHD--b474dd88--57b5--4d90--9f28--cba56f9facce
[14:21 xen02 ~]#

Could someone please provide a hint about what I can do to get that volume back working as well?

Thanks!

Tobias Kreidl · November 11, 2020

Could try to do an emergency power reset (xe command) on those VMs to see if that might help. If the boot disk got somehow corrupted, that's a who;e separate issue. LVM repairs can be very tricky, alas, and not always successful.

-=Tobias

Jerome Eichler1709155813 · November 11, 2020

Hi Tobias,

it's luckily not the boot disk. It's a second disk containing data. The boot disk works fine and the server itself boots up fine.

Tobias Kreidl · November 11, 2020

Is it a Linux VM? If so, it may need a fsck file system check.

Jerome Eichler1709155813 · November 11, 2020

Hi Tobias,

yes, it's linux, however, when I attach this second disk to that VM it does not boot up and results in the messages given in my initial post.

If I remove the disk, booting works properly.

I tried to mount that one manually on the host (hopefully correct):

[16:18 xen02 ~]# mount /dev/VG_XenStorage-718ed1a5-ed9c-ebad-2bd4-e621d3b7bffc/VHD-5450b065-2336-4f16-a3d7-dee3ad684951 /mnt
mount: unknown filesystem type '(null)'
[16:18 xen02 ~]#

[16:18 xen02 ~]# mount /dev/VG_XenStorage-ffde18af-117b-952c-3401-842915117b31/VHD-1f6e939a-5d21-4769-889d-c39ca542396c /mnt
mount: unknown filesystem type '(null)'
[16:18 xen02 ~]#

Hopefully that container is not lost? Any suggestions or things I could try?

Tobias Kreidl · November 11, 2020

If you can boot the VM, run "fsck -l" and see what other disks show up. If the data disk does, run "fsck /dev/..." on whatever disk device corresponds to the data disk and see if it can be repaired. Sorry, am going on a long drive and won't be available most of the rest of the day.

Jerome Eichler1709155813 · November 11, 2020

Hi Tobias,

like I said, the VM does not boot with the second disk attached to it. So I cannot do what you suggested, unfortunately, because, which disk shall I do an fdisk on, if it cannot be attached to the VM?

Tobias Kreidl · November 12, 2020

OK, that would indicate something's really wrong with the storage integrity. I guess I'd explore LVM command like vgchange to see id something got deactivated.

Sign In

Recovering VM after removing host from pool

Question

Jerome Eichler1709155813

Link to comment

7 answers to this question

Recommended Posts

Tobias Kreidl

Link to comment

Jerome Eichler1709155813

Link to comment

Tobias Kreidl

Link to comment

Jerome Eichler1709155813

Link to comment

Tobias Kreidl

Link to comment

Jerome Eichler1709155813

Link to comment

Tobias Kreidl

Link to comment

Create an account or sign in to comment

Create an account

Sign in

Discussions

Netscaler

Citrix

Technical Articles

Tech Insights

Community Articles

Resources

Events

Education