Jump to content
Welcome to our new Citrix community!
  • 0

GFS2 Performance?


Christof Giesers

Question

Did anyone here make benches with GFS2 installed?

How does it perform when using several VMs (parallel, random IO), compared to LVM?

Any experience with fragmentation/performance when having several, growing, VMs yet?

 

Having the option of a thing provisioned storage on our SAN would be great, especially since it would allow to make snapshot of even very big VMs (like our ~8 TB fileserver), which won't be possible with LVM and the missing of thinprovisioned, growing, snapshots (which are possible, just not implemented on XS!).

 

Regards

 - Christof

Link to comment

21 answers to this question

Recommended Posts

  • 1
11 hours ago, Mark Syms said:

That doesn't align with our internal testing data. There will obviously always be some overhead to GFS2 as it's a filesystem (and a distributed one at that) but in general we've seen only about 5-10% degradation.

 

On a quick test on our testing environment this morning I obtained the following results. Windows 10, on GFS2, 10g Intel X520 NIC to Storage with 2x10g bonded NIC accessing 12x SSDs in RAID0. Using your criteria of 8k, 60% random, 35% write I get this -

 

image.thumb.png.373f8f3ae956c895004e46b32c188198.png

 

The only difference I think is that the workers have 32 outstanding requests instead of the default 1.

 

So, there looks like something wrong in your environment. Can you run top in the XenServer Dom0 whilst the IOMeter is running and see what the memory usage and CPU load is like, and if there are any processes hitting 100%?

 

Thanks,

 

Mark.

Citrix Hybrid Cloud Platforms

 

Mark, I'm also curious what IOMeter results you get with the same SSD RAID 0 array over 12 SSDs, as I would have thought your IOPS would be at least 300,000 with that hardware.

  • Like 1
Link to comment
  • 0

Tobias, I think you read the wrong thing there. GFS2 (the filesystem, not the XenServer usage of it) has a 100TB max file size (not file system), the maximum theoretical file system size is 8EB.

 

XenServer currently only certify use of GFS2 with up to 2TB virtual disk files but we are working to increase that in a future release.

 

Theoretical limits and supported (either by Citrix or the upstream developers) are obviously different.

Link to comment
  • 0

Right, sorry to have misstated that; corrected now above.  More than 2 TB on any storage would be very welcome (aside from special units, which have higher limits already: Dell EqualLogic (15 TB limit) and NetApp (12 TB limit).

 

BTW, is it true that you currently can only storage motion from a GFS2 to another GFS2 SR? => Currently apparently yes, but to be addressed at a later date.

 

Thanks, @Mark.

-=Tobias

Link to comment
  • 0

Worse than that, you can't Storage Motion from or to GFS2 at all even to/from another GFS2 SR.

 

Live VM migration between hosts is fully supported but storage migration is not and will not work due to the new storage architecture. That's the next thing to be worked on.

 

Disks can be offline copied/moved between non-GFS2 and GFS2 SRs (or even between GFS2 SRs).

Link to comment
  • 0

Ah, thanks for the clarification,. @Mark. That will make this hard to use in some environments until more flexible storage migration is working. Full speed ahead on getting that implemented! On the other hand, the locking mechanism should be better on GFS2 and hence we should hope for fewer issues with things like orphan VDIs. And it should work fine for usage within a given pool.

 

-=Tobias

Link to comment
  • 0
1 hour ago, Alan Lantz said:

Absolutely thanks for the clarification, I mis-read that VM migration wasn't supported when in fact it is. That might be enough to sway me 

to try it sometime in the future. A larger than 2TB size would be a welcome as well. 

 

--Alan--

 

 

 

Worth testing on a local pool with shared storage, for sure!

Link to comment
  • 0

I have some testing results from a (2) node Starwind cluster to share, but keep in mind there are several layers of caching involved. So, the relative results are what's useful, rather than the raw numbers...

 

I compared disk I/O performance for (2) identical Windows 10 VMs using the following IOMeter access spec:

  • IO size: 8k
  • % random: 60%
  • % write: 35%

I created (2) identical 1.3TB Starwind HA image disks (LUNs), each with 2GB of L1 RAM cache and 100GB of L2 SSD cache. I created an LVM based SR and a GFS2 based SR, each 1.3TB in size, from the Starwind LUNs (using software iSCSI initiator of XenServer)

 

For the LVM based SR, IOPS averaged around 3200 and throughput was on average around 29 MBps. These results were pretty consistent, although they did improve as the cache hit rate improved.

 

For the GFS2 based SR, the best IOPS I briefly saw was 759 at 6.3 MBps. Also, max I/O latency was almost 2400 ms! Average latency was around 100ms on most tests. Also, the performance metrics were a lot more erratic on the GFS2 based SR.

 

I confirmed that in all tests, the iSCSI sessions were using MPIO and that network traffic on the (2) pNICs of each XenServer host were load balancing the iSCSI session traffic roughly 50/50.

 

So, it would seem that performance with GFS2 is pretty terrible.

 

Link to comment
  • 0

That doesn't align with our internal testing data. There will obviously always be some overhead to GFS2 as it's a filesystem (and a distributed one at that) but in general we've seen only about 5-10% degradation.

 

On a quick test on our testing environment this morning I obtained the following results. Windows 10, on GFS2, 10g Intel X520 NIC to Storage with 2x10g bonded NIC accessing 12x SSDs in RAID0. Using your criteria of 8k, 60% random, 35% write I get this -

 

image.thumb.png.373f8f3ae956c895004e46b32c188198.png

 

The only difference I think is that the workers have 32 outstanding requests instead of the default 1.

 

So, there looks like something wrong in your environment. Can you run top in the XenServer Dom0 whilst the IOMeter is running and see what the memory usage and CPU load is like, and if there are any processes hitting 100%?

 

Thanks,

 

Mark.

Citrix Hybrid Cloud Platforms

Link to comment
  • 0

Hi Mark,

 

I used a value of 16 outstanding IO requests for all tests. I also used (4) workers rather than the (2) you used.

 

I'm not sure if I'll get a chance to redo the testing, as I'm going on holiday in a couple of days and I've got a long list of things to take care of before I go. I may not post back until early April...

Link to comment
  • 0

Well... 12 SSDs in RAID0

2x 10 GE -> probably only 1.1 GB/s possible (bonding dosn't scale for that, MPIO is what we need)

Resulting in:

 

25k IOPS and 210 MB/s, wich is roughly a fifth of what should be possible - IOPS-wise it should be even like 10 times.

 

So the already rather slow storage system is even slower with GFS2. I hoped for thin provisioning on block storage, but I'm not sure if that way is an option for us.

I'll try to find time for benchmarks on a test machine over here, also curious to see what XS 8 brings, when it finally comes out - it's not even 2 weeks until Q1 is over, and if you want to keep your 'deadline'. Hope this or soon another release will also be LTSR again.

Link to comment
  • 0

Ok, I've captured a screenshot of TOP while running an IOMeter test, as previously described, with the VM located on a GFS2 LUN. Performance was just as bad as before, so it seems to be consistently poor.

 

FWIW, here's the TOP screenshot:

 

image.thumb.png.3c9be187d2bad724b99e9d941df927e3.png

 

It doesn't seem like there are any performance bottlenecks with respect to CPU/RAM utilization.

Link to comment
  • 0

I have tried to GFS2, actually forced cause network team refused to give me the enough size for operation.

The hardware I am on is
bl460c G8 Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Speed: 2700 MHz, 256 GB RAM

Storage is HPE 3Par 8400 the lun is all SSD with deduplication active and it's a thin lun from storage array side I like to note the lun is on RAID 5.

 

These servers are used for VDI machines, it's a cluster of 6 identical servers. the lun is a shared lun.

 

I did the test using the active enviroment, one of the machie so it should mimic almost the real word performance we have.

 

I did 2 tests 1 with 2 works and the other has 1 worker idel.

The chriteria is 8K, 65% Read, 60% Random.

 

I also added the result of the test i you are interested performance I did before switching to GFS2 has a little of overhead I guess that is expected but not that much.

for signle worker on LVM the result was 24 MOIPS while on GFS2 it became 15. as a control I ran the same test on citrix VDI demo server hosted on citrix datacenter I think I uploaded the result for a single worker for that it has a better performance than my LVM but that is expected they might be using Hyperconverg or something at that level.

 

in my view the performance is great specially if we are in great need for storage and storage team quite refuses to give is more line, even though they have dedupe active.

Here is our usage of space from our side I know 3par can give better dedup performance but what can you do storage admins :/

333.5 GB used of 2 TiB total (8.7 TiB allocated)

 

 

iometer_result.PNG

iometer_result_2.PNG

results.csv

results2.csv

citrix_windows_10_1io_worker.PNG

Link to comment
  • 0

Version 8.0 (Citrix Hypervisor)

When I used the 7.6 with GFS2 Server was struggling with RAM no matter how much I increase dom0 ram it needs more.

Now it's way better.

also I would like to mention the connection is 8gb fibre channel.

 

Before connecting to the 3Par we used Eva4400 it wasn't nice we keep getting corruption with the VDI image I guess GFS2 needs a good storage array (latest stuff).

on the EVA it was not even able to detect the storage size or the deduplicated size.

 

Link to comment
  • 0
On 19.3.2019 at 11:30 AM, Mark Syms said:

On a quick test on our testing environment this morning I obtained the following results. Windows 10, on GFS2, 10g Intel X520 NIC to Storage with 2x10g bonded NIC accessing 12x SSDs in RAID0. Using your criteria of 8k, 60% random, 35% write I get this -

 

image.thumb.png.373f8f3ae956c895004e46b32c188198.png

 

 

If I wouldn't know better, I'd guess you're joking.

 

25.7k IOPS out of 12 SSDs in RAID 0 should give several 100.000s of IOPS (if storage and network can handle).

I recommend to run that again on a clean environment and post again, because this is... something between ridiculous and embarrassing.

Even a single (enterprise) SSD should do more than your result (HPE usually gives numbers of between 60 and (for SAS) >100k IOPS for their SSDs and (also for SAS) > 1 GB/s).

 

Here's my result with our settings (as far as you've given the parameters) on LVMoHBA after running for 5 minutes:

grafik.thumb.png.206b22800e90ba0341d04a8a34c95d88.png

Ig does ~300 MB/s with ~36.5 k IOPs on a poor HPE MSA2040 SAN/FC with 2x FC16G on tiered SAS SSD/HDD storage (high priority, so should run on RAID 6 with 8 SAS SSDs).

Bottleneck is the MSA2040, as it's not a high-performance storage. HPE EVA and 3Par should add a digit there, if not busy with other loads.

 

I can't say what your bottleneck is, if it's GFS2 (that would make it an absolute no-go for any production) of somewhere else.

I expect you use 'slow' NFS, as bonding is nonsense for iSCSI and ('slow') usual bonding is for availability but doesn't help NFS performance.

 

Regards!

Christof

 

Link to comment
  • 0
15 minutes ago, Christof Giesers said:

 

If I wouldn't know better, I'd guess you're joking.

 

25.7k IOPS out of 12 SSDs in RAID 0 should give several 100.000s of IOPS (if storage and network can handle).

I recommend to run that again on a clean environment and post again, because this is... something between ridiculous and embarrassing.

Even a single (enterprise) SSD should do more than your result (HPE usually gives numbers of between 60 and (for SAS) >100k IOPS for their SSDs and (also for SAS) > 1 GB/s).

 

Here's my result with our settings (as far as you've given the parameters) on LVMoHBA after running for 5 minutes:

grafik.thumb.png.206b22800e90ba0341d04a8a34c95d88.png

Ig does ~300 MB/s with ~36.5 k IOPs on a poor HPE MSA2040 SAN/FC with 2x FC16G on tiered SAS SSD/HDD storage (high priority, so should run on RAID 6 with 8 SAS SSDs).

Bottleneck is the MSA2040, as it's not a high-performance storage. HPE EVA and 3Par should add a digit there, if not busy with other loads.

 

I can't say what your bottleneck is, if it's GFS2 (that would make it an absolute no-go for any production) of somewhere else.

I expect you use 'slow' NFS, as bonding is nonsense for iSCSI and ('slow') usual bonding is for availability but doesn't help NFS performance.

 

Regards!

Christof

 

The bottleneck is the single 10g network card in the host which is saturated.

Link to comment
  • 0
10 minutes ago, Mark Syms said:

The bottleneck is the single 10g network card in the host which is saturated.

 

If you know that's the bottleneck and your benchmark is (for comparison) useless: Why do you publish it?

Go grab some hardware, that's suitable and repeat it. I'm curious, but for now I doubt GFS2 is the way to go.

We stay on 7.1 LTSR for a while (probably until next LTSR or we switch to another solution), so I can't test that.

 

Small math says: Even with 10GE you should get a rough 150k IOPs with 8k size per IO - and thus hitting a good 1 GB/s. If your NIC is saturated with 210 MB/s and 27k IOPS, it shouldn't be about the NIC, rather drivers, firmware, CPU... whatver else.

 

But feel free to explain and fix my math.

 

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...