Reclaim free space broken in 7.0

[Florin Baca]

We have 3 XenServers in our infra, the oldest one running a fresh install of 6.5, newest running a fresh install of 7.0 and one which I have upgraded yesterday from 6.5 to 7.0. There were no upgrade issues, aside from guest tools not installing on Windows hosts. Storage is local on all servers.

 

Cleaning the upgraded server, I tried to 'reclaim free storage' which failed with:

 

blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

I looked in SMlog and I can see the blkdiscard runs after the lvcreate command:

 

Sep  5 07:20:57 xen2 SM: [11766] do_trim: {'sr_uuid': '54fcd1f8-ef26-d32f-1a18-db9e16169231'}
Sep  5 07:20:57 xen2 SM: [11766] lock: opening lock file /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr
Sep  5 07:20:57 xen2 SM: [11766] lock: tried lock /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr, acquired: True (exists: True)
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:57 xen2 SM: [11766] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv"
Sep  5 07:20:57 xen2 SM: [11766] '
Sep  5 07:20:57 xen2 SM: [11766] Ignoring exception for LV check: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv !
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvcreate', '-n', '54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv', '-l', '100%F', 'VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported
Sep  5 07:20:58 xen2 SM: [11766] '
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/dmsetup', 'status', 'VG_XenStorage--54fcd1f8--ef26--d32f--1a18--db9e16169231-54fcd1f8--ef26--d32f--1a18--db9e16169231_trim_lv']

 

For testing, I ran the same on the freshly installed XenServer 7.0 but I am seeing the exact same error so it's not caused by the upgrade. I don't know if this can be ignored (since it shows in XenCenter as well) but the virtual allocation reported by XenCenter is larger than the actual size of the disk.

 

As a reference, I also reclaimed the free space on the old 6.5 server. The operation succeeds and there is no blkdiscard command in the SMlog.

 

Has anyone else seen these issues with 7.0?

 

Thanks,

Florin

[Jason Rasmussen]

No, I need a solution that actually works, not a behind the curve hypervisor riddled with unresolved known issues that are blamed on users such as this issue.

[Daniel Andrade]

1 hour ago, Jason Rasmussen said:

I have moved all my XenServer VMs to VMWare VSAN. Migrating from XenServer was a pain due to non-standard 'open' format XenCenter exports, and mainly the problematic XenTools that embed themselves in Windows as they don't uninstall cleanly and conflict in other VM environments, but it can be done (reply if you want a how-to). Things just work in VMware and are much more reliable. I have retired all my XenServers. This is the solution.

 

What is the license value of this solution that you started using?

I use Dell R720 and Dell R730 with 256GB of Ram and 20TB of HDD SAS with 2 Intel Xeon processors with total of 20 physical cores and 40 in HT.

Do you have any idea of a free solution to replace XenServer? If not, can you tell me how much I would spend with VMWare?

[Jason Rasmussen]

Sometimes, you get what you pay for. Goodbye XenServer. You will not be missed.

[Thiago Natanael]

I have the same problem in XenServer 7.2, when i try to reclaim free space, i get this error:

 

BLKDISCARD ioctl failed: Operation not supported

 

and the disk usage continues high.....

 

I have an SCSI card that does not support TRIM, i know the issue is because of this, but, i need to resolve the problem...

 

Any suggestions would be apreciable! Thanks!

[Net Admin1709157341]

We have the same problem and we are getting below error on XenCenter:

Bulkdiscard ioctl failed: operation not supported

 

have you got any update on this?

 

Thanks,

[Florin Baca]

Unfortunately this is still unresolved as I haven't had too much time to look into it.

[tahir abbas]

I have same problem too...

[Francisco Rafael de Assis E. Fonseca]

I have the same problem, any solution? Thanks!

[Sysadmins Linux]

Same here!

[Tobias Kreidl]

Are these SSD drives? If so, the TRIM operation doesn't work on them as that's not currently supported on XenServer.

Otherwise, try a coalesce leaf run with:

/opt/xensource/sm/cleanup.py -u <UUID of the SR> -x

 

Note that VMs must be shut down for a coalesce of this nature to work on them, if I recall correctly.

 

-=Tobias

[Florin Baca]

They are not SSDs, all servers are using SAS or SATA hard drives. They are also different drive models. The third of our server has also been upgraded to XenServer 7 and it shows the same errors.

 

Unfortunately I can't shutdown the VMs right now, but I should give that a try when we have our next maintenance window.

[Tyson Moore]

Same issue here. All the drives are 2.5" SAS. HP and IBM drives. Server is HP DL360 Gen8 with HP Smart Array P420i 1GB. This issue also happened on our last server: HP DL360 G5.

 

https://support.citrix.com/article/CTX217613

https://support.citrix.com/article/CTX216146

 

Have found out that the SAS Controller likely doesn't support TRIM.

 

The issue we had last time was that the space filled up and we couldn't reclaim it, even after deleting snapshots. This led to what was a rather critical situation... we were unable to take snapshots as there wasn't enough space. So if we had a VM crash, or an update stopped the websites from running (we run websites from the VM's), we would be unable to do anything. Thankfully when I found the issue again about 2 - 3 days ago, I came back to the issue the next day and there was just enough space to take a snapshot.

 

So now there is just the issue of trying to claim the space back, as I know it will get to the point where it will be critical to take a snapshot, and I can't delete all of them and be left with nothing to restore from if it doesn't let me take one.

[Tobias Kreidl]

Lack of TRIM would likely cause this condition, regardless whether involving an SSD or spinning drive.

 

-=Tobias

[Jerome Eichler1709155813]

Hey Guys,

 

we have the same problem on our XenServer 7.1 installation.

 

Our SR is a Synology appliance and connected with ISCSI to XenServer.

The SR is nearly filled up to 100%, but the VMs themselves have just the half of it, around 450-500GB.

 

Trying to reclaim the free space results in the following error according to /var/Log/SMlog:

 

May  2 16:10:15 xen04 SM: [2932] do_trim: {'sr_uuid': '47fbe998-bb64-4b86-6f55-811f8947fb62'}

May  2 16:10:15 xen04 SM: [2932] lock: opening lock file /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:15 xen04 SM: [2932] lock: tried lock /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr, acquired: True (exists: True)

May  2 16:10:15 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:16 xen04 SM: [2932] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv"

May  2 16:10:16 xen04 SM: [2932] '

May  2 16:10:16 xen04 SM: [2932] Ignoring exception for LV check: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv !

May  2 16:10:16 xen04 SM: [2932] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:16 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:16 xen04 SM: [2932] ['/sbin/lvcreate', '-n', '47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv', '-l', '100%F', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv: BLKDISCARD ioctl failed: Operation not supported

May  2 16:10:17 xen04 SM: [2932] '

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/dmsetup', 'status', 'VG_XenStorage--47fbe998--bb64--4b86--6f55--811f8947fb62-47fbe998--bb64--4b86--6f55--811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] lock: released /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:17 xen04 SM: [2932] lock: closed /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

 

The disks inside the Synology are not SSD disks.

 

How can we get back our free space?

It is quite annoying as we currently cannot do any snapshots which makes our backup strategy quite unusable for the moment.

 

Any help would be appreciated.

 

Thank you!

[Chandrika Srinivasan]

Hi Florin, 

 

We made the change in XenServer 7.0 because TRIM was failing silently on 6.5 when it wasn't supported on the disk/array. On your 6.5 setup, when TRIM "succeeds", can you verify if it actually freed up space on the backend?

 

Thanks,

Chandrika

[Florin Baca]

Hi Srinivasan,

 

Thanks for the reply! Unfortunately our environment is now all upgraded to 7.0 so I am unable to test if this actually worked in the past but I do not remember to have had issues.

 

-Florin

[Chandrika Srinivasan]

Thanks Florin!

[Jason Rasmussen]

Hello,

 

The submitted bug remains unresolved. I cannot migrate any VMs to/from this server, and am concerned that upgrading my functional 6.5 to 7.2 will break the SSD reclaim free space.

 

Thank you

[Tobias Kreidl]

Unfortunately, probably the best fix would be to export or migrate all the VMs and their storage off that SR, reinitialize it, and move th econtents back again. There have been times that space reclamation has been so frustrating that this was the only recourse I could find that worked.

 

-=Tobias

[Jason Rasmussen]

Hi All,

 

I am running XenServer 7.1 on one of my servers and tried live migrating some VMs to it which all failed migrating. Now my local storate RAID6 SSD array is showing 91% used. It seems the failed migrations are eating up space and I can't find their ghosts. Trying to reclaim freed space I receive the following error:

 

Reclaiming freed space on SR 'Local storage'
blkdiscard: /dev/VG_XenStorage-5e3bb883-fe1b-e4d0-401c-4006fe1efeb2/5e3bb883-fe1b-e4d0-401c-4006fe1efeb2_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

Any suggestions how to fix this error?

 

Thank you

[Tobias Kreidl]

Again, alas, your best bet is to export/delete/import or your VMs or move them at least temporarily to a different SR so you can clean it off altogether and start over.

[Jason Rasmussen]

Hi Tobias,

 

So I have to shutdown each VM, export them all, reformat the SR (what is the easiest way to do this?), then re-import all VMs for this host which will not cleanup its space? Is this issue fixed in 7.2?

 

Thanks

[Jason Rasmussen]

Hi All,

 

I setup a temporary NFS share on our FreeNAS server to migrate the VM images to. After the VMs are migrated off the local storage, what is the best way to reformat the local XenServer storage with the ghost migrations eating up all the space so I can then move the VMs back?

 

Thank you

[Alan Lantz]

I've always reinstalled XenServer. But if you don't want to do that I wonder if unplugging and forgetting the SR then re-adding it back

with sr-introduce would be the method.

 

--Alan--

 

[Daniel Andrade]

Hello,

 

I have the same problem here, I have servers with XenServer 7 that the command presents the error, and identical machines running version 6.5 with no problem.

I am migrating my machines and I will try to format a host and put XenServer 6.5 again for testing, if this same hardware that presents the TRIM error running with version 6.5 I will reuse 6.5 on all my servers.

 

You can not use XenServer if you can never recover space when you remove VM when you remove snapshot. My virtual machines have HDDs of 600GB, every time I have this problem I lose 600GB. I am prone to migrating to another virtualization solution.

 

Reading this article I was shocked, maybe because I did not have fluent English.

 

Could you please help me: https://support.citrix.com/article/CTX217613 I understood that Citrix is working to change the displayed error phrase and not to solve the problem.