Post to Twitter |
Comments (10) |
Views (22670) |
Last week was a good one for XenServer. Within just a few days of each other:
- Virtualization Review published benchmark performance test results that showed XenServer to be "the porsche of hypervisors" while acknowledging the excellent runner-up performance of Hyper-V (no surprise there - same architecture, smart chaps, large team) and the rather pokey performance of ESX
- An announcement that SAP has selected XenServer for virtualization of its XenApp farms, in a deployment over 500 servers
- An announcement thatTesco PLC is using XenServer to virtualize its mission critical point of sales transaction software in a deployment of over 500 HP servers.
(Oh, and XenServer Virtual Infrastructure, including all management is 100% free. Get it here)
Combined with the detailed and thorough independent benchmarks of Project Virtual Reality Check, these recent market validations place XenServer at the very forefront of virtualization, and are a tremendous validation of the hard work of the XenServer crew, and the incredible commitment by the powerful Xen community, which develops the engine of this Porsche.
The Tesco announcement is personally important to me: Some 40% of food in the UK is purchased at Tesco, and all those point of sale transactions cross XenServer in real time. So when my mum swipes her card at the check-out, XenServer needs to do its thing! Personal too, in that this deployment, targeted at 1,500 servers, plays a key role in enabling Tesco to reach its commitments for a reduced carbon footprint. Sure, XenServer does that every day, in countless enterprises and clouds, but this one is a little more personal.
The stand taken by Keith Ward and team at Virtualization Review is also personal. Though Keith has been quick to point out that his team did not violate the VMware EULA in its performance benchmarking of XenServer, Hyper-V and ESX, FUD from the VMware side appears to allude to the fact that this may be so. Hats off to this courageous team which attempted to provide a thoroughly unbiased comparison of the performance of the three hypervisors, and continually consulted VMware for guidance to ensure that they were using valid use cases:"The Porsche of hypervisors? XenServer. Raise your hand if you saw that coming. It outperformed Hyper-V and ESX in most categories. The pokiest? ESX. Again, not at all what I expected. In fact, even in the few tests ESX came out on top, it barely edged out the competition. Microsoft did well across the board, and is definitely a fine product."
Of course, I immediately wondered what my friends at VMware would do to spoil the party. Remember, this is the organization that challenged IDC and Yankee group on their empirical research that showed VMware was vulnerable to Microsoft Hyper-V and Xen, and losing share to both.
The VMware benchmarking team are smart folks. They appear eminently reasonable:"Benchmarking is a difficult process fraught with error and complexity at every turn. It's important for those attempting to analyze performance of systems to understand what they're doing to avoid drawing the wrong conclusions or allowing their readers to do so. For those that would like help from VMware, we invite you to obtain engineering assistance from benchmark@vmware.com. And everyone can benefit from the recommendations in the Performance Best Practices and Benchmarking Guidelines paper. Certainly the writers at Virtualization Review can."
However Eric Horschmann slaps Virtualization Review with the conclusion that "We're Not Against Benchmarking - We're Only Against Bad Benchmarking" implying that VMware had no input into or approval over the results, and moreover asserting that benchmarking is such a deep science that it clearly is only accessible to a small VMware clique whose methodology is not to be shared with mere mortals.
In Horschmann's rather lengthy but shallow critique of the work done by the Virtualization Review team, he only mildly alludes to the fact that VMware engineers were in full approval of the methodology and tests used. Keith Ward confirms that"We talked extensively with VMware during the process, and an engineer in the benchmarking department approved our methodology before we went to press."
Moreover"To ensure the validity of our test results and testing environment, we enlisted the help of Stuart Yarost to formulate and validate the test plan. Yarost is an ASQ Certified Software Quality Engineer and Certified Quality Engineer with more than 22 years' experience in the software and quality fields. Yarost currently holds the position of Vice Chair of Programs for the ASQ Software Division. A special thanks to Yarost for his help."
Now, given that XenServer is doing so well, I'm inclined to be really positive to the folks in the corner with the black eye. So let's assume Eric Horschmann and the lads at the VMware Ministry of Truth are right. That means
- Keith Ward, Rick Vanover, Stuart Yarost and team (including engineer from VMware who approved of the methodology), you're all idiots.
- Benchmarking hypervisors requires deep science, and ESX as the industry leader is profoundly deep, requiring such profoundly, super deep expertise to tune and benchmark that it is just not possible for ordinary humans (hence the VMware EULA that forbids publications of comparative benchmarks - and Horschmann clearly states that this is why the EULA is so restrictive). Virtualization Review was foolish to think that even with a staff of trained VMware engineers, an independent consultant, and advice and approval from VMware, that it would be possible to reproduce the fine art that is uniquely owned by the VMware benchmarking team. (I've commented on this previously in "VMware Wins! (Bad Science Required)". What they didn't know of course is that results are meaningless - this is just spin.
- Since rational, well disposed folks trying to make their ESX installation work as well as possible failed utterly in the attempt (that is, XenServer and Hyper-V clobbered ESX on performance), we conclude that probably no normal user could get ESX to perform either. It really is a super complex (expensive) hypervisor, and it's so difficult to get to work that if you're merely a well intended VMware user, you have no hope.
- Therefore, probably most VMware installations run extremely badly, since they are run by mere mortals who could never understand how deep and fickle the beast called ESX is. And even if you've spoken to a VMware engineer about how to get it right, you're probably still doing it wrong.
Wake up VMware. Your response rings hollow, and we are afforded yet another laugh at your expense.
Comments (10)
Mar 16, 2009
Anonymous says:
reply to post 4: a virtual infrastructure should not be treated as "...reply to post 4: a virtual infrastructure should not be treated as "a default fits for all". What do I mean? sure you can go through a default install and configuration, and in a matter of minutes you will have VMs up and running. but, in a Datacenter (small, medium, big), you should take into consideration a full design for servers, networking, storage, security, etc. And please, believe when I say, it is not a cup of tea.
regards
Jose Ruelas
Anonymous replies:
Mar 17, 2009
Anonymous says:
Simon, I have to take issue with the Benchmark article at the beginning of this ...Simon, I have to take issue with the Benchmark article at the beginning of this post, as after reading it over I would definitely not conclude that XenServer is the "Porsche of hypervisors". Granted that's supposedly Virtualization Review's assertion, but I think their testing is flawed just like everyone else's (including VMware's). The only benchmark they published that I might actually trust is the SQL Job Completion time, as the Average CPU and RAM operations can be erratic even with VMs on local storage (we've found that out from our own testing). In that benchmark, Hyper-V wins, you guys and ESX are close. After seeing the results from our internal testing, we decided to scrap performance as a testing measure for VM platforms, and go for Management, Support, and other metrics instead. I call bullshit on all y'all (Citrix, Microsoft, and VMware).
Mar 17, 2009
Simon Crosby says:
1. We didn't have anything to do with the Virtualization Review selection of how...1. We didn't have anything to do with the Virtualization Review selection of how/what/when to do their testing. I got the results the same day the rest of the world did. So there's nothing to call Citrix on here.
2. You're right that you should be able to benchmark whatever is important to you, openly, and publish the results. You can with XenServer, you can't with ESX.
3. I support vociferous, voluble benchmarking so that if there are any issues in any product, the vendor gets to hear about it, and fix it, fast.
Mar 17, 2009
Anonymous says:
Simon, why you are so angry at VMware? What do you mean: "Therefore, probably mo...Simon, why you are so angry at VMware? What do you mean: "Therefore, probably most VMware installations run extremely badly".
Do you mean that all 500 Fortune companies have extremely badly running VMware installations? Are they all idiots and you are "virtualization God"? What you are trying to prove - XenServer is better than VMware. Maybe, but nobody knows, because nobody uses it. Just an example of how XenServer works - we moved storage to a completely different location (physically), however, VMs (which were sitting on that storage) are still running and XenServer shows that storage is connected!!!! Does it require patch or hotfix? Anyway, I am not agains XenServer, I just don't like so much acidity in your post. Relax!
Mar 17, 2009
Anonymous says:
Err, he's not angry. He's just pointing out the absurdity of VMware's posi...Err, he's not angry. He's just pointing out the absurdity of VMware's position, and showing that if you start with their assumptions, that is the only logical end result - that nobody can actually configure VMware to achieve good performance.
Since that's plainly rediculous, it follows that the initial premise (vmware being the only people capable of benchmarking ESX) is false.
Mar 17, 2009
Simon Crosby says:
Indeed. Not angry at VMware at all, other than the absurdity of their posi...Indeed. Not angry at VMware at all, other than the absurdity of their position.
best
Simon
Mar 18, 2009
Anonymous says:
Very, very limited benchmarking, testing a very limited set of capabilities whic...Very, very limited benchmarking, testing a very limited set of capabilities which Keith confirms:
From Keith Ward:
"... This test only determined one aspect of these hypervisors. To make a buying decision based on this article would be foolish in the extreme."
ESX has some many unique features, including performance ones (Large pages support, Jumbo Frames, and many others), management ones, such as Storage VMotion, that Xen cannot beat.
The same Virtualization review site states ESX stands alone on the "Enterprise Ready" category.
This benchmarking only proves Xen and Hyper-V can grant decent performance, when stressed by "PC-Burning like" software. This benchmark was not done simulating real usage scenarios.
It is more or less the same thing of comparing a single Terminal Server with a single XenApp server, and conclude they perform almost identically.
For the SQL Server test, Large Pages, available on ESX server only, can change everything on the results.
Mar 31, 2009
Anonymous says:
To add something: I run in this conversation. I really do not care VMWARE or Ci...To add something:
I run in this conversation. I really do not care VMWARE or Citrix or I care about perfomanse being small shop of 30+ servers. I don't like rocket science. I like to get it working in 10 minutes and leave it working. I found I can do this with VMWARE as well as with Citrix. The only thing where Citrix is below WMWARE is converter tools. I noticed that On single QC server with 2GB of RAM (yes, 2GB of ram), I can run more VM on Citrix then on VMWARE. Simple as that. If I find something where I can run more VM then on Citrix, then I will go for it. And Citrix is free with live migration. Tested, Working. WMWARE is ripping people off. They can do this since they are CISCO. That's reason we don't use CISCO but HP PROCURVE. SImple, you get more bang for the buck. I thing if you exclude Fortinet x00 companies, when comes to small shopes, somebody will win the battle. So far there is Citrix in lead. VMWARE just because price is behind. And this is the fact not who is faster nanosec or if you can run one VM more on particular hypervisor
)). So chill out, let Citrix to bring VMWARE price down since they are REALLY overcharging now ....
Apr 01, 2009
Anonymous says:
You sure do blog alot, are you sure you're a CTO and not a professional VMware b...You sure do blog alot, are you sure you're a CTO and not a professional VMware basher? Why don't you talk up the strengths of Citrix instead of spreading FUD.
Jul 27
Anonymous says:
I am a Independent consultant and certified on all 3 VMware/Citrix & MS and ...I am a Independent consultant and certified on all 3 VMware/Citrix & MS and being in virtualization space for 5years, I have to say, there is lot for Citrix & MS to catch up !!! Having said so, they have own strenghts but nothing significant in comparision to VMware, not even close.
Add Comment