Jump to content
Welcome to our new Citrix community!
  • 0

pbd-plug failed with SR backend failure


Reg Dancy1709159709

Question

Hello,

 

I tried plugging my SR back since it shows in Xen Center that it needs repair.

 

xe pbd-list sr-uuid=0cfed699-3cb2-b9c8-96aa-fc184a7242c6 host-uuid=dc25b112-d348-44d1-be85-c239bf90ba4d
uuid ( RO)                  : 22ef1a9b-de43-bc75-de54-8a999d30d3ab
             host-uuid ( RO): dc25b112-d348-44d1-be85-c239bf90ba4d
               sr-uuid ( RO): 0cfed699-3cb2-b9c8-96aa-fc184a7242c6
         device-config (MRO): multihomelist: 172.31.225.101:3260,172.31.225.102:3260,172.31.225.103:3260,172.31.225.100:3260; SCSIid: 3624a93701ab79778588c412e0001103f; targetIQN: iqn.2010-06.com.purestorage:flasharray.3c9bf12bd7c53cf3; target: 172.31.225.101; port: 3260
    currently-attached ( RO): false

 

After grabbing the uuid I did the following:

 

xe pbd-plug uuid=22ef1a9b-de43-bc75-de54-8a999d30d3abThere was an SR backend failure.
status: non-zero exit
stdout:
stderr: Traceback (most recent call last):
  File "/opt/xensource/sm/LVMoISCSISR", line 590, in <module>
    SRCommand.run(LVHDoISCSISR, DRIVER_INFO)
  File "/opt/xensource/sm/SRCommand.py", line 351, in run
    sr = driver(cmd, cmd.sr_uuid)
  File "/opt/xensource/sm/SR.py", line 147, in __init__
    self.load(sr_uuid)
  File "/opt/xensource/sm/LVMoISCSISR", line 320, in load
    LVHDSR.LVHDSR.load(self, sr_uuid)
  File "/opt/xensource/sm/LVHDSR.py", line 199, in load
    self._undoAllJournals()
  File "/opt/xensource/sm/LVHDSR.py", line 1131, in _undoAllJournals
    self._undoAllInflateJournals()
  File "/opt/xensource/sm/LVHDSR.py", line 1151, in _undoAllInflateJournals
    self.lvActivator.activate(uuid, vdi.lvname, False)
  File "/opt/xensource/sm/lvmanager.py", line 63, in activate
    self.lvmCache.activate(self.ns, uuid, lvName, binary)
  File "/opt/xensource/sm/lvmcache.py", line 49, in wrapper
    ret = op(self, *args)
  File "/opt/xensource/sm/lvmcache.py", line 147, in activate
    self.activateNoRefcount(lvName)
  File "/opt/xensource/sm/lvmcache.py", line 49, in wrapper
    ret = op(self, *args)
  File "/opt/xensource/sm/lvmcache.py", line 200, in activateNoRefcount
    lvutil.activateNoRefcount(path, refresh)
  File "/opt/xensource/sm/lvutil.py", line 556, in activateNoRefcount
    text = cmd_lvm(cmd)
  File "/opt/xensource/sm/lvutil.py", line 157, in cmd_lvm
    stdout = pread_func([os.path.join(LVM_BIN, lvm_cmd)] + lvm_args, *args)
  File "/opt/xensource/sm/util.py", line 189, in pread2
    return pread(cmdlist, quiet = quiet)
  File "/opt/xensource/sm/util.py", line 182, in pread
    raise CommandException(rc, str(cmdlist), stderr.strip())
util.CommandException: Input/output error

 

While I ran this command I tailed the SMlog:

 

SM: [21385] Setting LVM_DEVICE to /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f
Dec  4 08:55:22 xs4 SM: [21385] lock: opening lock file /var/lock/sm/0cfed699-3cb2-b9c8-96aa-fc184a7242c6/sr
Dec  4 08:55:22 xs4 SM: [21385] LVMCache created for VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6
Dec  4 08:55:22 xs4 SM: [21385] ['/sbin/vgs', 'VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6']
Dec  4 08:55:22 xs4 SM: [21385]   pread SUCCESS
Dec  4 08:55:22 xs4 SM: [21385] lock: acquired /var/lock/sm/0cfed699-3cb2-b9c8-96aa-fc184a7242c6/sr
Dec  4 08:55:22 xs4 SM: [21385] LVMCache: will initialize now
Dec  4 08:55:22 xs4 SM: [21385] LVMCache: refreshing
Dec  4 08:55:22 xs4 SM: [21385] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6']
Dec  4 08:55:22 xs4 SM: [21385]   pread SUCCESS
Dec  4 08:55:22 xs4 SM: [21385] ['/usr/bin/vhd-util', 'scan', '-f', '-c', '-m', 'VHD-*', '-l', 'VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6']
Dec  4 08:55:22 xs4 SM: [21385]   pread SUCCESS
Dec  4 08:55:22 xs4 SM: [21385] Found inflate journal 13e6adde-1a4c-4fdd-a109-51b2f433cd00, deflating /dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/VHD-13e6adde-1a4c-4fdd-a109-51b2f433cd00 to 26902265856
Dec  4 08:55:22 xs4 SM: [21385] lock: opening lock file /var/lock/sm/lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/13e6adde-1a4c-4fdd-a109-51b2f433cd00
Dec  4 08:55:22 xs4 SM: [21385] lock: acquired /var/lock/sm/lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/13e6adde-1a4c-4fdd-a109-51b2f433cd00
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 (0, 0) + (1, 0) => (1, 0)
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 set => (1, 0b)
Dec  4 08:55:22 xs4 SM: [21385] ['/sbin/lvchange', '-ay', '/dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/VHD-13e6adde-1a4c-4fdd-a109-51b2f433cd00']
Dec  4 08:55:22 xs4 SM: [21385] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdak not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sds
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdak which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sds
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdg not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdak
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdg which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdak
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdy not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdg
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdy which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdg
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdm not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdy
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdm which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdy
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdae not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdm
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdae which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdm
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdau not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdae
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdau which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdae
Dec  4 08:55:22 xs4 SM: [21385]   Found duplicate PV HwzLBFWyWxMqcZOwlFg0n01pu630hZ1h: using /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdav not /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdau
Dec  4 08:55:22 xs4 SM: [21385]   Using duplicate PV /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdav which is last seen, replacing /dev/disk/by-scsid/3624a93701ab79778588c412e0001103f/sdau
Dec  4 08:55:22 xs4 SM: [21385]   device-mapper: resume ioctl on (253:52) failed: Invalid argument
Dec  4 08:55:22 xs4 SM: [21385]   Unable to resume VG_XenStorage--0cfed699--3cb2--b9c8--96aa--fc184a7242c6-VHD--13e6adde--1a4c--4fdd--a109--51b2f433cd00 (253:52)
Dec  4 08:55:22 xs4 SM: [21385] '
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 (1, 0) + (-1, 0) => (0, 0)
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 set => (0, 0b)
Dec  4 08:55:22 xs4 SM: [21385] lock: released /var/lock/sm/lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/13e6adde-1a4c-4fdd-a109-51b2f433cd00
Dec  4 08:55:22 xs4 SM: [21385] lock: released /var/lock/sm/0cfed699-3cb2-b9c8-96aa-fc184a7242c6/sr
Dec  4 08:55:22 xs4 SM: [21385] lock: opening lock file /var/lock/sm/lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/13e6adde-1a4c-4fdd-a109-51b2f433cd00
Dec  4 08:55:22 xs4 SM: [21385] lock: acquired /var/lock/sm/lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/13e6adde-1a4c-4fdd-a109-51b2f433cd00
Dec  4 08:55:22 xs4 SM: [21385] WARNING: decrementing normal refcount of 0
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 (0, 0) + (-1, 0) => (0, 0)
Dec  4 08:55:22 xs4 SM: [21385] Refcount for lvm-0cfed699-3cb2-b9c8-96aa-fc184a7242c6:13e6adde-1a4c-4fdd-a109-51b2f433cd00 set => (0, 0b)
Dec  4 08:55:22 xs4 SM: [21385] ['/sbin/lvchange', '-an', '/dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/VHD-13e6adde-1a4c-4fdd-a109-51b2f433cd00']
Dec  4 08:55:23 xs4 SM: [21385]   pread SUCCESS
Dec  4 08:55:23 xs4 SM: [21385] ['/sbin/dmsetup', 'status', 'VG_XenStorage--0cfed699--3cb2--b9c8--96aa--fc184a7242c6-VHD--13e6adde--1a4c--4fdd--a109--51b2f433cd00']
Dec  4 08:55:23 xs4 SM: [21385]   pread SUCCESS

 

I did some digging into lvm and performed the following:

 

lvremove /dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/inflate_13e6adde-1a4c-4fdd-a109-51b2f433cd0

This didn't work.

 

Any suggestions would be much appreciated.

 

 

Link to comment

13 answers to this question

Recommended Posts

  • 0

Yes I did. This was working fine until recently. However I did just take a peak at multipath.conf file and it seems it might be missing the config. Yet, multipath appears to be running anyway:

 

multipath -ll
3624a93701ab79778588c412e00011036 dm-3 PURE    ,FlashArray      
size=20T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 19:0:0:251 sdag 66:0   active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 18:0:0:251 sdaa 65:160 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 20:0:0:251 sdan 66:112 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 14:0:0:251 sdc  8:32   active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 16:0:0:251 sdo  8:224  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 17:0:0:251 sdu  65:64  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 21:0:0:251 sdam 66:96  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 15:0:0:251 sdi  8:128  active ready running
3624a93701ab79778588c412e00011035 dm-0 PURE    ,FlashArray      
size=20T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 18:0:0:253 sdab 65:176 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 19:0:0:253 sdah 66:16  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 21:0:0:253 sdao 66:128 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 14:0:0:253 sdd  8:48   active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 15:0:0:253 sdj  8:144  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 16:0:0:253 sdp  8:240  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 17:0:0:253 sdv  65:80  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 20:0:0:253 sdap 66:144 active ready running
3624a93701ab79778588c412e00011034 dm-1 PURE    ,FlashArray      
size=20T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 20:0:0:252 sdat 66:208 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 14:0:0:252 sdf  8:80   active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 17:0:0:252 sdx  65:112 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 15:0:0:252 sdl  8:176  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 16:0:0:252 sdr  65:16  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 21:0:0:252 sdar 66:176 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 18:0:0:252 sdad 65:208 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 19:0:0:252 sdaj 66:48  active ready running
3624a93701ab79778588c412e00011033 dm-4 PURE    ,FlashArray      
size=20T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 19:0:0:254 sdal 66:80  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 21:0:0:254 sdax 67:16  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 18:0:0:254 sdaf 65:240 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 16:0:0:254 sdt  65:48  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 17:0:0:254 sdz  65:144 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 14:0:0:254 sdh  8:112  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 20:0:0:254 sdaw 67:0   active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 15:0:0:254 sdn  8:208  active ready running
3624a93701ab79778588c412e0001103c dm-2 PURE    ,FlashArray      
size=20T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 18:0:0:250 sdac 65:192 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 19:0:0:250 sdai 66:32  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 20:0:0:250 sdas 66:192 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 14:0:0:250 sde  8:64   active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 15:0:0:250 sdk  8:160  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 17:0:0:250 sdw  65:96  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 21:0:0:250 sdaq 66:160 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 16:0:0:250 sdq  65:0   active ready running

 

 

Link to comment
  • 0

I tried removing the inflate disk at first but I wasn't successful, however after several attempts it finally worked, thanks to one of my colleagues. Here are the steps he used:

 

  lvscan | grep inflate
  lvremove /dev/VG_XenStorage-0cfed699-3cb2-b9c8-96aa-fc184a7242c6/inflate_13e6adde-1a4c-4fdd-a109-51b2f433cd00_26902265856
  pvs
  vgchange -ay --config global{metadata_read_only=0}
  pvs
  lvs
  partprobe /dev/mapper/VG_XenStorage--0cfed699--3cb2--b9c8--96aa--fc184a7242c6-VHD--3ebf5571--81c8--48c3--ad3f--1df826219021
  vgchange -ay --config global{metadata_read_only=0}
  xe sr-scan uuid=0cfed699-3cb2-b9c8-96aa-fc184a7242c6
  partprobe /dev/mapper/VG_XenStorage--0cfed699--3cb2--b9c8--96aa--fc184a7242c6-VHD--3ebf5571--81c8--48c3--ad3f--1df826219021
  ls /etc/lvm/backup
  xe sr-scan uuid=0cfed699-3cb2-b9c8-96aa-fc184a7242c6
  lvscan | grep inflate
  lvscan |grep MGT
  xe sr-scan uuid=0cfed699-3cb2-b9c8-96aa-fc184a7242c6
  xe pbd-plug uuid=22ef1a9b-de43-bc75-de54-8a999d30d3ab
  xe sr-scan uuid=0cfed699-3cb2-b9c8-96aa-fc184a7242c6

 

Thanks for your suggestions guys!

Link to comment
  • 0

```

 xe pbd-list
uuid ( RO)                  : ab95dcb9-db62-f8f6-dffb-51cab5104fa7
             host-uuid ( RO): 954536bd-2c94-4ca2-aff9-3d5906d489a0
               sr-uuid ( RO): 2a647b48-ab45-6500-a55c-c1ef856a55d2
         device-config (MRO): device: /dev/md3
    currently-attached ( RO): false

 

```

xe pbd-plug uuid=ab95dcb9-db62-f8f6-dffb-51cab5104fa7
There was an SR backend failure.
status: non-zero exit
stdout:
stderr: Traceback (most recent call last):
  File "/opt/xensource/sm/LVMSR", line 2130, in ?
    SRCommand.run(LVHDSR, DRIVER_INFO)
  File "/opt/xensource/sm/SRCommand.py", line 343, in run
    sr = driver(cmd, cmd.sr_uuid)
  File "/opt/xensource/sm/SR.py", line 142, in __init__
    self.load(sr_uuid)
  File "/opt/xensource/sm/LVMSR", line 200, in load
    self._undoAllJournals()
  File "/opt/xensource/sm/LVMSR", line 1136, in _undoAllJournals
    self._handleInterruptedCloneOps()
  File "/opt/xensource/sm/LVMSR", line 885, in _handleInterruptedCloneOps
    self._handleInterruptedCloneOp(uuid, val)
  File "/opt/xensource/sm/LVMSR", line 922, in _handleInterruptedCloneOp
    self._undoCloneOp(lvs, origUuid, baseUuid, clonUuid)
  File "/opt/xensource/sm/LVMSR", line 972, in _undoCloneOp
    vhdInfo = vhdutil.getVHDInfo(basePath, lvhdutil.extractUuid, False)
  File "/opt/xensource/sm/vhdutil.py", line 109, in getVHDInfo
    ret = ioretry(cmd)
  File "/opt/xensource/sm/vhdutil.py", line 99, in ioretry
    errlist = [errno.EIO, errno.EAGAIN])
  File "/opt/xensource/sm/util.py", line 284, in ioretry
    return f()
  File "/opt/xensource/sm/vhdutil.py", line 98, in <lambda>
    return util.ioretry(lambda: util.pread2(cmd),
  File "/opt/xensource/sm/util.py", line 185, in pread2
    return pread(cmdlist, quiet = quiet)
  File "/opt/xensource/sm/util.py", line 178, in pread
    raise CommandException(rc, str(cmdlist), stderr.strip())
util.CommandException: Invalid argument

 

```

 

Please, help..

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...