Post to Twitter |
Comments (13) |
Views (36731) |
Under the guise of science (lots of graphs, configuration parameters and techno speak must mean they are impartial, right?) the "performance team" at VMware has published compelling performance data for ESX 3.5 versus XenServer 5.0 and native, for virtualized XenApp workloads.
Congratulations are in order. The VMware team has done a fabulous job of searching to find a single instance of a set of parameters for ESX & XenServer that, under a carefully crafted set of "simulated user behaviors" shows ESX outperforming XenServer for the XenApp workload.
As a former academic, I'd give this mumbo jumbo an F grade. Bad science, bad scientists, uneven playing field:
- First, the VMware claims are not independently reproducible. Like every claim on performance that VMware makes, only they can make it and nobody can refute it, because nobody else can publish results for comparative tests between VMware and any other product.Their EULA forbids it. So, these results are true by definition, from your pals at the VMware ministry of truth. By contrast, an open, independent set of tests run by Project Virtual Reality Check, a benchmarking project conducted jointly by two Citrix/VMware solution providers in the Netherlands finds results wholly at odds with VMware's. Our own performance tests have also been independently validated by the Tolly Group (to whom VMware also denied permission to publish comparative results against ESX). Project VRC concludes that
- XenServer supports between 118-128% more users per host than VMware ESX for XenApp VMs configured with 1 vCPU. For example: Test 5 on XenServer and Test 8 on ESX (which were identical tests) tested 4 VMs with 1 vCPU and 4Gb of memory per VM, and shows that XenServer's optimal user workload is 86.5 users whereas ESX is just 38 users
- XenServer supports between 42-68% more users per host than VMware ESX for XenApp VMs configured with 2 vCPUs. For example Test 8 on XenServer and Test 16 on ESX (which were identical tests) tested 4 VMs with 2 vCPUs and 4Gb of memory per VM, and shows that XenServer's optimal user workload is 124.5 users whereas ESX is just 82.5 users.
- Second, the VMware "study" is not a thorough exploration of a valid set of parameters for the Terminal Services / XenApp workload. Instead, it is a narrow look at a particular set of configurations which are not reasonable in practice:
- No test of 32 bit workloads - the primary candidates for server consolidation for this workload because a 32 bit OS exhausts its memory at 4 GB and a modern server can pack hundreds of GB and many cores. Our work in this area has shown a compelling benefit to virtualizing TS/XenApp 32 bit workloads on XenServer, and an equally compelling set of reasons not to use ESX for this purpose.
- Unrealistic configuration - The server used in the tests is certainly punchy - the machine had 64 GB RAM and 4 processors--each with 4 cores (16 total processor cores). Anyone familiar with 64b TS/XenApp knows this machine could easily support hundreds of XenApp sessions. But the "scientists" at VMware don't. They instead chose to run exactly one VM (with only 2 vCPU's and using only 25% of the available memory) and XenApp at minimal levels of concurrency (i.e. 10-40 users). No multi-VM scenarios, no tests at useful user-counts. Based on their measurements they appear to gleefully extrapolate deeper into the realm of fiction to proudly pronounce their horse the winner.
** In our own work in this area, we found XenServer and other virtualization platforms to be roughly equal for this rather absurd set of parameters. But for high user counts, the numbers are radicaly different. We suitably anonymized the non-XenServer results, which are reproduced from the Tolly report. Quiz of the day, which result do you think is ESX?:
- Third, even VMware's users and partners are challenging their "results". This study is so one sided that the majority of the blog followups on VMware's site from its partners and customers point out how ridiculous they are. Since the "performance team" may well redact them, here are a couple, saved for posterity:
VMware's continued blunders in the performance arena are nothing short of embarrassing. So I've decided to issue an open challenge to VMware CTO Stephen Herrod: Steve, it's time to rein in the monkeys behind the keyboard, end VMware's indefensible EULA restrictions and allow independent performance comparisons of your products with others, by third parties with a vested interest in accuracy and independence. This sort of nonsense does nothing for VMware's brand credibility, its customers, channel partners or competitors other than give us all a hearty laugh at your expense.
Comments (13)
Feb 03, 2009
Anonymous says:
Simon, This reminds me of the Oracle war between Tom Kyte and some folks ...Simon,
This reminds me of the Oracle war between Tom Kyte and some folks at Burlusen Consuting (Don is the CEO) there a couple of years back on Askto.oracle.com . Tom is undefeatable [and remains] as he really dished out some really cool analogies and independent test results. We all need honest results. Our customers are screaming for answers.
Eventually the customers/clients that use Citrix farms to consume and be productive, must stay that way, whether they are on VMware, Xen or bare-metal.
Lets see the independent results.
Tarry
Feb 03, 2009
Todd Hsu says:
vmware's EULA is ridiculous, and so is their ELA (enterprise licensing agreement...vmware's EULA is ridiculous, and so is their ELA (enterprise licensing agreement). If Citrix has asked their clients to prepaid their subscription advantages 3 years up front. I am sure Citrix's revenue will be more than 2 billion dollars. But is that the right thing to do?
Simon, what you've done here is the reason why Citrix has the best channel program and the best partners in the industry. Technology driven to slove real life customer challenges. Which is what we are all here to do. I will not be suprised vmware's revenue suffers from their own poison this year. Meanwhile, let the truth speak for itself. We need more scientist like you in this industry.
Feb 03, 2009
Anonymous says:
The issue I am having with the vendor bitchslapping is HOW is Citrix XenApp bett...The issue I am having with the vendor bitchslapping is HOW is Citrix XenApp better on Xensource not WHY. Im not particulary interested in the marketting goop that both you and Vmware are guilty of, to be fair at least
I lay this to you sir, maybe you need to walk before you can run and actually perform tests like vmware on your own products first before going into a public contest.
Feb 03, 2009
Anonymous says:
So this is the kind of response the CTO of Citrix offers: 10% (vague) technical ...So this is the kind of response the CTO of Citrix offers: 10% (vague) technical rebuttal and 90% bitch-slapping? If so, I don't think it is your competitors that are the more likely to be redacting posts on their site.
And with respect to VMware's EULA restrictions, you say you want the EULA lifted, but I suspect either with or without a EULA it serves your purposes quite well. Without one, you could publish all kinds of technical drivel that VMware would be forced to rebut. But with the EULA, you can claim that the EULA is the only obstacle to your ability to showcase the "superiority" of your technology. I wonder whether VMware's rejection of your publication was due to the quality of the technical comparison or because it would reflect poorly on VMware's product. You would have us believe the latter, but you offer nothing to preclude (or even diminish) the possibility of the former.
For some of your readers virtualization technology may be their religion. And for them, this type of "we are so good and our competitors are so evil" screed may be therapeutic. But for the rest of us, this is really just more crap for us to wade through.
Feb 03, 2009
Anonymous says:
Sorry my Anonymous but Simon is actually right. The complains about VMware's EU...Sorry my Anonymous but Simon is actually right.
The complains about VMware's EULA have been commentet upon many many times from other vendors as well. Please remember, that it is not only VMWare and Citrix that has virtualisations capable software. And you dont have to see any test results to realise that XenServer is much faster and should also be able to handle more load. I recommend you to download XenServer and just play around for a while and then do the same with ESX.
Feb 03, 2009
Anonymous says:
Well microsoft seem to have done the same as VMware and EULA for years....MS hav...Well microsoft seem to have done the same as VMware and EULA for years....MS have the same policy over most products for benchmarking .
I dont personally think that it's something that citrix, micky mouse or donald duck can do anything about even if they wanted to its there choice and they are protecting the integrity of there product from people bending the truth (i wonder who) which is a tatic unfortunately exists today in the world of lame topics like this post.Would Citrix have complained about the situation if Xensource wasn't bought and built purely out of inbred innovation like ESX?
Good luck to Vmware Vroom and the team, they give good solid advice on HOW there products are tested and don't drivel something that a lot of vendors need to take note of....
Feb 04, 2009
Anonymous says:
With respect to VMWare's EULA here is the part that people complain about... "Y...With respect to VMWare's EULA here is the part that people complain about...
"You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review."
To the people complaining about it have you ever sent a request into VMware or better yet ever had one rejected? I've seen benchmark reports where VMware has not come out on top, so unless you are doing something what is being accused of VMware I'm pretty sure you will be able to publish them.
Mar 02, 2009
Anonymous says:
According to Citrix, VMware denied a request by The Tolly Group to publish some ...According to Citrix, VMware denied a request by The Tolly Group to publish some benchmark results between ESX and XenServer. Knowing how well regarded and detailed the Tolly people conduct their testings, it seems to me that the complaint against VMware is legitimate. Even more when they don't subject themselves to the same standard they force on others.
Mar 02, 2009
Anonymous says:
Whenever I click on the thumbnail image of the vmware comments shown above, I ge...Whenever I click on the thumbnail image of the vmware comments shown above, I get an invalid image error. Was the .GIF pulled?
Mar 02, 2009
Simon Crosby says:
Just tested it - works fine for me. Possibly a browser setting? SimonJust tested it - works fine for me. Possibly a browser setting?
Simon
Apr 01, 2009
Anonymous says:
I'm glad to see the commentary about the childishness of this and many other pos...I'm glad to see the commentary about the childishness of this and many other posts against VMWare. It's the kind of wording i expect to over a 12 pack of Bud light and shots of cuervo. Unfortunate because as a Citrix customer, while I was impressed with some interviews on DABCC, the writing takes down my opinion a couple notches.
Both companies have some great areas of expertise. Both companies were first to market with respective advantages. Perhaps if you want to proliferate in the DC space, maybe start up an ASPLA program or something that will help the new world of on demand absorb more of your product line. Giving away a gutted Hypervisor and trying to compare it to ESX 3.5 seems more like FUD than anything. Goes right in line with cloaking yourself as a virtualization expert because you bought ZenSource, and renamed your flagship product line to somehow potray Citrix invented the Hypervisor? FUD FUD FUD.
BTW, i could not find the post about VMWare not running XenApp, but I can tell you it runs great, and when you look at ROI and TCO with Real Estate, People, Hardware, and Software in mind, the war over the cost of the hypervisor and/or mgmt tools is about 10% of the issue. But if anything its in the mgmt tools. Please let us know when you start giving away citrix essentials.
Put the war to bed, get creative, build on your great brand and product line in the space you know best and the other will come along.
May 24
Anonymous says:
Think what you like of the technologies, my criticism goes to VMWare for closing...Think what you like of the technologies, my criticism goes to VMWare for closing their forums within 2 weeks of their posting. Despite the unfavourable comments, Citrix has kept this forum open for 3 months after this posting.
Clearly Citrix Systems has shown through this action that it is happy to be subjected to criticisms from vendor land and the consumer, while VMWare is not.
If you are going to dole out the criticism about your competitors products then you should be able to take the heat from us (consumers / vendors). Hence, one for Citrix for doing this and a big zero to VMWare (whose forum I followed to here) for being a bunch of cowards.
Jul 27
Anonymous says:
Interesting, the VMware comments were about other tests that others had performe...Interesting, the VMware comments were about other tests that others had performed and how they didn't line up with the results published there. No bashing of products or people. Of course they closed comments 11 days after posting and shunted the rest to a discussion forum while this is still up...
Posters, get some data about YOUR test results in here if you have a problem with Citrix winning in the XenApp race. Prove to us how you can outpace Citrix on Citrix by using ESX.
We have also validated the test results of Citrix, the Tolly Group and PVRC with VMware getting 173 users, Hyper-V getting 177 and XenServer getting 283 on identical hardware. This was a script from an actual document creation in word and was CPU bound on all three platforms. VM's were configured acording to vendor best practices for XenApp or Terminal Services and running 2vCPUs and 4GB of RAM each on 32bit Windows Server 2003.
Add Comment