Garbage Collection > Unexpected bump in size, compared to minimum acheived > Leaf-coalesce failed

[Emanuele Tonelli]

On our 8.2 LTSR hypervisor, some running VM (all of them are Windows 2012R2) cannot recover the space used for deleted snapshots (our linux VMs never show this problem).

They get a message like this one:

SMGC: [29036] Iteration: 6 -- Initial size 172425728 --> Final size 212349440

SMGC: [29036] Unexpected bump in size, compared to minimum acheived

What does it mean?

How can the size increase (from 172425728  to 212349440) instead of decreasing?

Is this the reason for the Leaf-coalesce failure?

 

If the VM is shut down when the Garbage Collection starts, then no error is received and the space is reclaimed correctly.

Why the difference between a running VM and a stopped VM?

 

SMGC: [29036] No progress, attempt: 1
SMGC: [29036] Aborted coalesce
SMGC: [29036] Iteration: 1 -- Initial size 3811787264 --> Final size 1122189824
SMGC: [29036] Iteration: 2 -- Initial size 1122189824 --> Final size 338424320
SMGC: [29036] Iteration: 3 -- Initial size 338424320 --> Final size 309006848
SMGC: [29036] Iteration: 4 -- Initial size 309006848 --> Final size 222855680
SMGC: [29036] Iteration: 5 -- Initial size 222855680 --> Final size 172425728
SMGC: [29036] Iteration: 6 -- Initial size 172425728 --> Final size 212349440
SMGC: [29036] Unexpected bump in size, compared to minimum acheived
SMGC: [29036] Starting size was         3811787264
SMGC: [29036] Final size was            212349440
SMGC: [29036] Minimum size acheived was 172425728
SMGC: [29036] Removed leaf-coalesce from 51054e79[VHD](101.000G/202.512M/101.203G|a)
SMGC: [29036] *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
SMGC: [29036]          ***********************
SMGC: [29036]          *  E X C E P T I O N  *
SMGC: [29036]          ***********************
SMGC: [29036] leaf-coalesce: EXCEPTION <class 'util.SMException'>, VDI 51054e79-6cb0-4a9e-8603-2c1f813aff34 could not be coalesced
SMGC: [29036]   File "/opt/xensource/sm/cleanup.py", line 1644, in coalesceLeaf
SMGC: [29036]     self._coalesceLeaf(vdi)
SMGC: [29036]   File "/opt/xensource/sm/cleanup.py", line 1919, in _coalesceLeaf
SMGC: [29036]     .format(uuid=vdi.uuid))
SMGC: [29036]
SMGC: [29036] *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
SMGC: [29036] Leaf-coalesce failed on 51054e79[VHD](101.000G/202.512M/101.203G|a), skipping
 

[Tobias Kreidl]

How full is the SR on which those VMs have their storage? I take it these VMs have XenTools all installed OK, right?

[Emanuele Tonelli]

We have 4 TB used and 6 TB free....

Anyway the problem is the 9.3.0 tools on Windows 2012 R2. With the 9.2.3 it works perfectly...

[Tobias Kreidl]

Perhaps try to uninstall and re-install XenTools. Hard to understand why thin provisioning would not let to coalesce unless it's something like 90% full.

Worst case, if you have the space, you could storage-migrate the VMs to different SRs and start from scratch with an empty SR. Of course, storage migration will probably not work if the VMs do not have the tools installed properly.

 

-=Tobias

[Mark Syms]

This means that the VM is continuously writing data to its virtual disk at a rate faster than the garbage collection process can consolidate the data into the parent node in the VHD tree. As you can see the GC has several attempts and then gives up marking the VDI as not leaf coalescable.

As the GC process has to read data from one VHD node and write to another it is unfortunately quite common that a VM with a moderately high IO throughput will win the race as that only has to write data to the disk and not perform the secondary read.

[Tobias Kreidl]

Try to do a powerstate reset on the VM then?

 

-=Tobias

[Mark Syms]

Unlikely to help doing a powerstate reset. Shutting the VM down for a period and running a scan of the SR while it is stopped may address the issue, or reducing the amount of data being written.

[Tobias Kreidl]

@Mark, if the VM was hung, is what I meant, but yes, best to shut it down if possible.

 

-=Tobias

[Mark Syms]

If the VM were hung it would not be producing Write IOs at a rate faster than what the GC process can consolidate them. That it is gives a pretty good indication that the VM is live and active.

[Tobias Kreidl]

Good point!

 

-=Tobias