Jump to content
Welcome to our new Citrix community!
  • 0

XenServer 7 Problem clearing Space


Michael Ohlman

Question

Hi There, 

 

I am not very skilled with XenServer as I was given a job of figuring out why Xackup had stopped working.

 

It is very similar if not the same as this discussion here

https://discussions.citrix.com/topic/391289-xen-server-7-reclaim-disk-space/

 

I just want to make sure this is the proper issue.  The sr is at 97% full and will not create a snapshot. I have done a bit of digging but I am afraid to remove or delete anything important. 

 

My question is should I shut down vms and try a rescan, when I perform the scan it says there are no changes then continues to scan a network share sr. I just want to make sure there is something hiding or something I can remove to clear up space.

 

I have a full server report that I can upload just not sure the proper method.

 

Thanks.,

 

Mike

Link to comment

6 answers to this question

Recommended Posts

  • 0

You will need to clean out old snapshots and such, which with an sr-scan can take up to 24 hours. However, with not hardly any space left, this can also get to the point where you can no longer rely on a coalescing operation to be able to work.

 

Your options then are:

1) Move the storage of the VM to another SR so things get cleaned up in the process, and local storage can be used, if available

and large enough

2) Go through the painful VM export/delete/import process for at least some of the VMs

3) move the VM and its storage to another pool, giving you the chance to clean up the old SR

 

-=Tobias

Link to comment
  • 0

Check the /var/log/SMlog file when you request a scan of the SR. This will start the cleanup tasks which may produce errors if there isn't enough free space to copy data from child nodes of the delta tree to their parents. The cleanup temporarily needs extra space as while it's working the same data will be in two places while it is copied from child to parent and then the child is deleted once the parent is a complete superset of the child. The cleanup process has a 5 minute delay once started so that the cleanup doesn't run during a VM bootstorm.

Link to comment
  • 0

So I ran sr scan and this is the output. It is running commands on the ISO share. The storage in question is the underlined which says no changes. Not sure what else I should be looking for. This past weekend Another technician did a manual backup of the vms so we have that. 

 

This is the SR scan and output. I have also posted below the output when I click reclaim free space

 

 

pread SUCCESS
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] SR d76d ('Local storage') (9 VDIs in 3 VHD trees):
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *c3abc671[VHD](100.000G//31.395G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             979b5600[VHD](100.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             40e8ff6a[VHD](100.000G//100.203G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *6929243a[VHD](450.000G//128.152G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             5a7aa359[VHD](450.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             4f8b928c[VHD](450.000G//450.887G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *0dc0e5c6[VHD](100.000G//90.445G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             c7fe7cf2[VHD](100.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             61e01b20[VHD](100.000G//100.203G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]
Dec 17 12:49:52 xenserver-cosmic SM: [24180] lock: released /var/lock/sm/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/sr
Dec 17 12:49:52 xenserver-cosmic SM: [24180] ['/usr/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa']
Dec 17 12:49:52 xenserver-cosmic SM: [24180]   pread SUCCESS
Dec 17 12:49:52 xenserver-cosmic SM: [24180] ['/usr/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa']
Dec 17 12:49:52 xenserver-cosmic SM: [24180]   pread SUCCESS
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] Got sm-config for *c3abc671[VHD](100.000G//31.395G|ao): {'read-caching-enabled-on-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'false', 'read-caching-reason-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'LICENSE_RESTRICTION', 'vdi_type': 'vhd', 'vmhint': 'OpaqueRef:e32b2d2a-a666-f6ca-1df1-adaebacc085a', 'vhd-blocks': 'eJzt1CEOgDAMheHuJEgsFrWjbcfCcwCuwBGQTSDpBkEtQUwQCPxf0oq24qma2ThI5pbcGitoW05Oq02m4WL5DvHpAADqaF93v38t8bdEAT5gky6o+FmOAgAAAAAA+BmXAEYMzlA='}
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] Got sm-config for *6929243a[VHD](450.000G//128.152G|ao): {'read-caching-enabled-on-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'false', 'read-caching-reason-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'LICENSE_RESTRICTION', 'vdi_type': 'vhd', 'vmhint': '91142d8b-65c5-d462-cf3c-85a84c0710f3', 'vhd-blocks': 'eJzt2c0JwlAMAOBXBL0ILiDoCN2gq7iJjtZV3MCjB/HnmIBPqhUV+b7bIyRN0vb0Lvu+lNJd/sLpNkqZfLuLn1OAR2bf/kcBAAAAAAC44xCvdOb1+51dDG3ioa1WmMRDk8r1MbSKkVO1QhlY4YE0RRcjx/roL6mPPl56Z4vRFab1jaclr+NhmZJS8bTkbYykJQ9tPPW6GpiUGm+rjz3HSFNvHD6m3z6bkT5jAAAAAAAAAAAAAAAAAAAAAAAA4H2aK/S42ro='}
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] Got sm-config for *0dc0e5c6[VHD](100.000G//90.445G|ao): {'read-caching-enabled-on-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'false', 'read-caching-reason-834667c2-cdfe-42bc-b3b3-fceebfae2bd9': 'LICENSE_RESTRICTION', 'vdi_type': 'vhd', 'vmhint': 'OpaqueRef:8ac4e0cf-3d4a-772b-3cfc-a89d44d13398', 'vhd-blocks': 'eJzt2DEOgkAQBdC/oaAxobbiKNyM5Rh2HgXsvYPbWUolmiAjI0qISLKNQcx/CZNhszvMhG5F1AYwJYBY3h1HK4uRzd0AERH9sfwR7fPlqnE3UytERERERLQodVR/uIT7ddZnU4OwQaRZASmMJhfENRIH65J+GwZHEpkDVun0P6g8CuTtc3O2jbUOmHZTf19pnM0gDvvJLa/2GwSSBZrZYQFtd731/2JRCcLx8klDFgV9zdi/pK9cw6HrAlbkrKPrHTaRN3MHG/x6uw=='}
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] No work, exiting
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] SR d76d ('Local storage') (9 VDIs in 3 VHD trees): no changes
Dec 17 12:50:14 xenserver-cosmic SM: [24329] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|d7e71139-2bf6-bcd6-ee1b-4d6efec61044|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:c7f07c93-b8c0-45c3-e3db-a58a1e68de56', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:50:44 xenserver-cosmic SM: [24458] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|6111a577-3d43-c5a2-27c1-68c1c60eee56|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:798b2d8a-221f-76d8-37ab-639d34bf52a3', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:51:15 xenserver-cosmic SM: [24613] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|8a1ba5f8-3803-a4ba-8bb0-86202ebd5e5b|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:9e9bfc3a-ba50-3639-4cae-55fe6464ea10', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:51:45 xenserver-cosmic SM: [24740] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|16a2937d-045c-75e9-35ff-08b49595df9f|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:6d73e527-6aaa-b6b1-1303-8272695b9b83', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:52:15 xenserver-cosmic SM: [24897] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|b1badd2b-db7a-9d65-0f68-41d649485805|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:24b5a6a9-baaa-4318-63a8-cbd552e25c37', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:52:45 xenserver-cosmic SM: [25024] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|8357e9f9-aec5-0cdf-40b5-49520a510152|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:9af5d277-1f08-c3c5-b550-49e438507916', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}
Dec 17 12:53:15 xenserver-cosmic SM: [25179] sr_scan {'sr_uuid': '00f6c6b4-55f3-3b80-beab-c2dc72b63d9b', 'subtask_of': 'DummyRef:|f9dbb77d-1470-2a4a-6549-ed490a7ba834|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:056257eb-8900-b6bf-477e-a050f4b339dd', 'session_ref': 'OpaqueRef:5cf61201-a69c-56c5-2bb0-2187fde3b878', 'device_config': {'username': 'biznet', 'type': 'cifs', 'SRmaster': 'true', 'cifspassword_secret': '612d84f2-2973-e976-80c0-3caf8e49268f', 'location': '//192.168.1.249/ISO'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:278badca-1d1f-9077-8469-a2402d3c4bda'}

 

 

Reclaim Free Space output:

 do_trim: {'sr_uuid': 'd76d87f6-6c27-e4cb-b4b0-49f1d56f82aa'}
Dec 17 13:53:58 xenserver-cosmic SM: [10273] lock: opening lock file /var/lock/sm/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/sr
Dec 17 13:53:58 xenserver-cosmic SM: [10273] lock: tried lock /var/lock/sm/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/sr, acquired: True (exists: True)
Dec 17 13:53:58 xenserver-cosmic SM: [10273] ['/usr/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa_trim_lv']
Dec 17 13:53:58 xenserver-cosmic SM: [10273] FAILED in util.pread: (rc 5) stdout: '', stderr: '  One or more specified logical volume(s) not found.
Dec 17 13:53:58 xenserver-cosmic SM: [10273] '
Dec 17 13:53:58 xenserver-cosmic SM: [10273] Ignoring exception for LV check: /dev/VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa_trim_lv !
Dec 17 13:53:58 xenserver-cosmic SM: [10273] ['/usr/sbin/lvcreate', '-n', 'd76d87f6-6c27-e4cb-b4b0-49f1d56f82aa_trim_lv', '-l', '100%F', 'VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa']
Dec 17 13:53:59 xenserver-cosmic SM: [10273]   pread SUCCESS
Dec 17 13:53:59 xenserver-cosmic SM: [10273] ['/usr/sbin/lvremove', '-f', '/dev/VG_XenStorage-d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa_trim_lv', '--config', 'devices{issue_discards=1}']
Dec 17 13:53:59 xenserver-cosmic SM: [10273]   pread SUCCESS
Dec 17 13:53:59 xenserver-cosmic SM: [10273] ['/sbin/dmsetup', 'status', 'VG_XenStorage--d76d87f6--6c27--e4cb--b4b0--49f1d56f82aa-d76d87f6--6c27--e4cb--b4b0--49f1d56f82aa_trim_lv']
Dec 17 13:53:59 xenserver-cosmic SM: [10273]   pread SUCCESS
Dec 17 13:53:59 xenserver-cosmic SM: [10273] Trim on SR: d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa complete.
Dec 17 13:53:59 xenserver-cosmic SM: [10273] lock: released /var/lock/sm/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/sr
Dec 17 13:53:59 xenserver-cosmic SM: [10273] lock: closed /var/lock/sm/d76d87f6-6c27-e4cb-b4b0-49f1d56f82aa/sr
 

Thanks for any suggestions you have.

 

 


 

Link to comment
  • 0

Based on the following

14 hours ago, Michael Ohlman said:

Dec 17 12:49:52 xenserver-cosmic SMGC: [24180] SR d76d ('Local storage') (9 VDIs in 3 VHD trees):
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *c3abc671[VHD](100.000G//31.395G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             979b5600[VHD](100.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             40e8ff6a[VHD](100.000G//100.203G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *6929243a[VHD](450.000G//128.152G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             5a7aa359[VHD](450.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             4f8b928c[VHD](450.000G//450.887G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]         *0dc0e5c6[VHD](100.000G//90.445G|ao)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             c7fe7cf2[VHD](100.000G//8.000M|n)
Dec 17 12:49:52 xenserver-cosmic SMGC: [24180]             61e01b20[VHD](100.000G//100.203G|ao)

 

There is no space to be freed. You have three virtual disk trees, 2 x 100GB + 1 x 450GB. All three trees have a snapshot - the 8M nodes. The parent nodes of the trees (the read-only shared pieces of the snapshots and the live writeable disks) total 250G and then you have the additional 650GB for the writeable disks to give a total of ~ 900GB.

 

The reclaim space option that you have run is a command to the SAN to reclaim no longer used space in the SAN storage but from the looks of the above you really are using all the space.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...