Jump to content
Welcome to our new Citrix community!
  • 0

Shown usage of local storage is double from the allocated disks


Peter VARGA1709157052

Question

This are the allocated disks on a HV 8.1 with all updates:

 

image.png.e4ef30becc255893ad55cd04eef66757.png

 

In total 417 MB.

 

And this is displayed in the overview for the local storage:

image.thumb.png.0a833fbf1dbb5e2e884f99555c6b4521.png

 

More than the double of the size of all disks. Why is this?

I checked each VM, no one has a snapshot [which would be anyway visible in the Virtual Disks Overview. I tried also "Reclaim freed space".

What am I doing/understanding wrong?

Link to comment

17 answers to this question

Recommended Posts

  • 1

If you're down to 68% then there should be less reason for the coalesce not to work. Whilst you might not have a snapshot, deleting one is not instantaneous and requires the coalesce to complete for it to be done. As an example

          A
         / \
        B   S1
       / \
      C   S2
     / \
    D   S3


In this example A is the disk originally created with the VM. When snapshot S1 is taken, A is marked readonly and two new VHDs are created, B & S1. The VM is updated to write to B. This process is repeated as S2 & S3 are taken, also producing C & D.

 

When we delete S1, S2 & S3 we no longer need to have all 4 of A, B, C & D, but we do need all of the data that we can see if we look through D->C->B->A. Some of the data in A, B or C may not be accessible anymore as it has also been re-written into a sub-node. The snapshot removal operation only removes the Sn VHD node and schedules the coalesce process. This explains why, although you no longer have any visible snapshots, you have more space being consumed than just that defined by the disks. The coalesce process will initially merge B->A, remove B and then merge C->A and remove C leaving just A & D.

 

This last bit is then the difficult bit, at least when the VM is running. We want to copy all the data in D into A but the VM is still writing into D so whilst we're copying data from D to A there is a good chance the VM might re-write a data block that's already been copied. If this block update doesn't get copied again later we'd experience data loss. To avoid this, the final part of the process pauses the VMs I/O path so that it doesn't write anymore and copies the data from D to A and then points the VM at A and unfreezes its I/O path. However, this pause can only be for a very short time as otherwise the VM might crash. So, if D is larger than a certain threshold, the coalesce process will take an internal snapshot of the current state and give the VM a new empty delta disk to write to. Once the snapshot is coalesced, if the VMs live node is small enough it is paused and the copy completed. If it is not yet small enough, because the VM wrote a lot of data while that coalesce was happening, the process is repeated until the criteria is met or it becomes clear that the coalesce cannot make progress - i.e. if after every snapshot coalesce the delta, which started empty, is bigger than the previous generation then the process cannot win against the VMs IO rate and it will give up trying to do so.

 

If you look at /var/log/SMlog you should see logging from the coalesce process.

  • Like 2
Link to comment
  • 0

Clearly, you have thin provisioning on this SR.  If it gets too full, the coalesce process will not be able to work. That typically happens when you reach around the 90% actual storage allocation level. One option would be -- if you have extra space somewhere -- to migrate the storage to another SR to empty it out and then move the storage back again once it has been zeroed out. Otherwise, you may have to do a full export/import of the VMs,which would be quite time-consuming and also reqire downtime on the VMs.

 

-=Tobias

Link to comment
  • 0
4 hours ago, Tobias Kreidl said:

Clearly, you have thin provisioning on this SR.  If it gets too full, the coalesce process will not be able to work. That typically happens when you reach around the 90% actual storage allocation level. One option would be -- if you have extra space somewhere -- to migrate the storage to another SR to empty it out and then move the storage back again once it has been zeroed out. Otherwise, you may have to do a full export/import of the VMs,which would be quite time-consuming and also reqire downtime on the VMs.

 

-=Tobias

 

 Hi Tobias,

 

I don't think the SR is thin provisioned:

 

uuid ( RO)                    : c11e63eb-0802-15af-1367-46243b5fa5af
              name-label ( RW): Local storage
        name-description ( RW):
                    host ( RO): APA92
      allowed-operations (SRO): VDI.enable_cbt; VDI.list_changed_blocks; unplug; plug; PBD.create; VDI.disable_cbt; update; PBD.destroy; VDI.resize; VDI.clone; VDI.data_destroy; scan; VDI.snapshot; VDI.mirror; VDI.create; VDI.destroy; VDI.set_on_boot
      current-operations (SRO):
                    VDIs (SRO): 1861ff23-0f56-4fe3-868f-19c554b30036; 5535af17-be5f-4b2f-8dc2-d90c46db2609; 8d73b60a-055f-4584-be66-dba713035320; ee54bb7d-470f-417d-aac1-0a7275583f52; aab2db72-6779-495b-a929-af552d7fb0b8; 5771a738-3d5d-4851-a9ee-c0c55f432d93; f4caf398-f4ea-4a7c-8a40-8030defbdf47; eb1f659b-25f4-4571-9912-47baf285c405; bfc1156d-d026-41f0-99d9-a3b5d79714cc; afeec223-1909-446e-ae14-67a0a8b18de4; a912d665-afaa-4dfd-a97a-b16e42799dd6
                    PBDs (SRO): 58bf360c-7fb4-2a87-b4d0-493dab753e84
      virtual-allocation ( RO): 665722028032
    physical-utilisation ( RO): 657549426688
           physical-size ( RO): 955076575232
                    type ( RO): lvm
            content-type ( RO): user
                  shared ( RW): false
           introduced-by ( RO): <not in database>
             is-tools-sr ( RO): false
            other-config (MRW): trim_last_triggered: 1592043134.76; i18n-original-value-name_label: Local storage; i18n-key: local-storage
               sm-config (MRO): allocation: thick; use_vhd: true; devserial: scsi-3600605b008b62560228e88a9261883bc
                   blobs ( RO):
     local-cache-enabled ( RO): false
                    tags (SRW):
               clustered ( RO): false

 

Link to comment
  • 0

I'm curious how the virtual allocation (1.1 TB) exceed the physical allocation (837 GB)? I guess if it's iSCSI and LVM and a local SR, it would not be thinly provisioned. However, it's still 94% full hence coalescing isn't going to work well once that full. I assume you tried an SR scan, Peter?

Link to comment
  • 0
12 hours ago, Tobias Kreidl said:

I'm curious how the virtual allocation (1.1 TB) exceed the physical allocation (837 GB)? I guess if it's iSCSI and LVM and a local SR, it would not be thinly provisioned. However, it's still 94% full hence coalescing isn't going to work well once that full. I assume you tried an SR scan, Peter?

 

I have no clue but this morning the used percentage has substantially decreased - see the below screenshot. I don't know what is going on.

Can it be the NAUBACKUP can cause it as there was a backup running and it failed for a 200 GB VM. I had to start it again and it looks to me like that some of the used space was caused by the snapshot. Any idea regarding the backup?

 

However, the total of visible used space by the disks is 417 GB but the shown usage is 200 GB more. This would be still the size of the VM which failed to backup? Could a HV restart help or is there an other way how to reclaim the space.

 

image.thumb.png.848a1fef80990816ca59c95fc6ecc41d.png

Link to comment
  • 0

Peter,

If there is a coalesce process, that can take up to a day, so maybe that was it. And indeed, not sure what I was thinking but it's clearly not thin provisioned. The extra allocation may have been because the coalesce process had not kicked in or finished yet. As I understand it a thick provisioned SR should have the same allocated and used space showing, at least under normal circumstances!

You could look for extra VDI storage to see if a failed snapshot is taking up space or maybe it's visible in XenCenter, even.

 

-=Tobias

Link to comment
  • 0

Would snapshots alter these sizes ?  I know some earlier versions of XenServer did weird storage allocation stuff, but most of that

should have been addressed by now. If its coalesce you may have to export a VM to give some room for that to happen. That 

doesn't address root cause though.

 

--Alan--

 

Link to comment
  • 0

There have been times where the only way I could fix this was to export all the VMs and import them back again or just destroy the whole SR and re-create it before importing the VMs back. Moving VM storage (Storage Xenmotion) would be a better option, if space is available as the VMs could remain running the entire time.

 

-=Tobias

Link to comment
  • 0

All SRs that support snapshots have some level of thin provisioning (the snapshots are stored truncated).

 

As Tobias says, when the SR is very full (and 90% is a good guideline figure) there may not be sufficient free space for the coalesce phase of the garbage collection process to work. Snapshots are stored in a delta tree on the storage with read only parents and writeable leaf nodes. When a snapshot is removed the distinction between the former snapshot's parent and it's child is no longer required and the two nodes can be coalesced. In order to do this the size of the parent must be increased in order to be able to contain the superset of the data contained in the parent and the child (some of the child data may replace data in the parent so it's not as simple as just adding the size of the child to the parent). The child data can then be copied to the parent and once complete the child can be deleted which will free any space that was consumed by duplicated data.

 

The important thing here is that there must be enough available free space to temporarily allow for this data to be duplicated whilst copying.

 

Mark.

Citrix  Hypervisor, Storage Engineering.

Link to comment
  • 0
On 6/15/2020 at 10:30 AM, Mark Syms said:

All SRs that support snapshots have some level of thin provisioning (the snapshots are stored truncated).

 

As Tobias says, when the SR is very full (and 90% is a good guideline figure) there may not be sufficient free space for the coalesce phase of the garbage collection process to work. Snapshots are stored in a delta tree on the storage with read only parents and writeable leaf nodes. When a snapshot is removed the distinction between the former snapshot's parent and it's child is no longer required and the two nodes can be coalesced. In order to do this the size of the parent must be increased in order to be able to contain the superset of the data contained in the parent and the child (some of the child data may replace data in the parent so it's not as simple as just adding the size of the child to the parent). The child data can then be copied to the parent and once complete the child can be deleted which will free any space that was consumed by duplicated data.

 

The important thing here is that there must be enough available free space to temporarily allow for this data to be duplicated whilst copying.

 

Mark.

Citrix  Hypervisor, Storage Engineering.

 

Dear Mark, thank you very much for your very detailed answer. Even I understand your explanation I don't understand why the usage of the SR can grow up to 94% when the total of all used disks == 417 MB and NO snapshot does exist on the SR.

 

Yesterday I installed the two latest HV 8.1 updates and I had to restart the HV. The used percentage is still at 68% which means, 200 GB are more used than the total of all disks.

 

However, what about this possible solution: I create from a VM a snapshot and run in XenCenter "New VM from Snapshot". Once the VM has been created I delete the base. As I have anyway backups there is no danger of losing data. May be then the 200 GB will become available once the VM physically doesn't exist.

 

Thx, Peter

Edited by Peter VARGA
relevant typo in the size
Link to comment
  • 0
On 6/16/2020 at 10:47 AM, Mark Syms said:

....

This last bit is then the difficult bit, at least when the VM is running.

....

 

This was the key to the solution. I shutdown all VM on the host for about 90 minutes and now the allocated space matches exactly the size of all VDI.

 

Thank you Mark. Anyway, may I remark that this could be solved better as not always all VM on a host can be shutdown in order to clean up all snapshots properly?

 

image.thumb.png.7bd4831b54e4d5642998bcc2b4a6785d.png

Link to comment
  • 0

Of course that is desirable and there have been a number of improvements in this area to reduce the incidence of VMs that just will not complete the coalesce but, ultimately, there will always be some that are just writing data so fast that it is not possible to coalesce the data up into the parent node.

Link to comment
  • 0
On 7/19/2020 at 10:57 AM, Mark Syms said:

Of course that is desirable and there have been a number of improvements in this area to reduce the incidence of VMs that just will not complete the coalesce but, ultimately, there will always be some that are just writing data so fast that it is not possible to coalesce the data up into the parent node.

 

Few days later the usage is again by 80%. I consider it as a design mistake. No offence.

The best is, I didn't create any snapshot since then...

Link to comment
  • 0
1 minute ago, Mark Syms said:

I don't disagree, unfortunately it's an architectural and ancient mistake and one which we are paying dearly for and will continue to do so until we can convince the powers that be to fund a major overhaul of the entire storage subsystem in the hypervisor.

 

Here I found a considerable bound amount which can be transferred to the new development:

 

image.png.714683a8883203e8bd3db1bc3c5b323f.png

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...