Post to Twitter |
Comments (12) |
Views (35277) |
Memory is a big concern for XenApp on a 32bit operating system like Windows 2003 Server. In the default state, Windows 2003 can only "see" 4GB of memory, which is split up into two equal parts: Kernel Memory (2GB) and User Memory (2GB). Kernel Memory is further broken down into 4 other parts:
- Paged Pool: Memory space used by the system and kernel level components that can be paged out of physical RAM and into the page file
- Non Paged Pool: A section of memory guaranteed to always reside in physical RAM and is used by the operating system for certain kernel level processes
- System Page Table Entry : An index table that tells the operating system where the virtual memory actually resides in physical RAM or on the page file
- System Cache: Maps open files in memory for better performance. This is where the registry hives are located as well
Once the system has started, the different sections of kernel memory cannot be re-allocated. The system tries to allocate these 4 areas appropriately, but they might require "tweaking". However, the four areas cannot all be set to the maximum level as that would go over the 2GB limit of kernel memory.
Many of you are probably saying, "But I can use the PAE switch on Windows 2003 to go above the 4GB limit". You are correct, you can go above the 4GB limit, but are you aware of the consequences of this action?
- You must be using Windows 2003 Enterprise or Data Center. This setting does not function in Windows 2003 Standard.
- The PAE Switch does NOT change the kernel memory limitations of 2GB
- To use the extra RAM, more System Page Table Entry memory is used
- If you have more System Page Table Entries, you will end up with less Paged Pool, System Cache and Non Paged Pool
Talk about being between a rock and a hard place. Adding more RAM and enabling the PAE switch "might" give you more scalability but at a great cost for a more expensive operating system, more RAM and special optimization configuration analysis and implementation. The reason I said "might" give you more scalability is because you will now likely run out of kernel memory before you run out of user memory. So you just bought a more expensive operating system and more RAM that will sit there wasted.
Now I know some of you will add a comment saying something to the effect that you are using the PAE switch and ended up increasing single server scalability by 60, 70, 80 or even 90%. All I can say is congratulations and I applaud you 
. You are lucky as you have the right set of apps for this to work as well as it has. But I want you to think about going down a completely different route. Virtualization...
Keep using Windows 2003 Standard but virtualize it with XenServer. Upgrade the RAM on the physical servers so it can support 2-4+ virtual servers. In the end, you will end up with a system that is more flexible, scalable and easier to manage.
If you interested in learning more about sever virtualization for XenApp, then take a look at the following:
- TechTalk Recording: Make Server Virtualization work for XenApp (http://www.citrix.com/English/NE/events/event.asp?eventID=1679445)
- White Papers
Daniel
Homer Quote of the Blog: "To be loved, you have to be nice to others EVERYDAY! To be hated, you don't have to do squat."
Comments (12)
Aug 04, 2008
Anonymous says:
I like the idea of distributing the load and avoiding some of the memory pitfall...I like the idea of distributing the load and avoiding some of the memory pitfalls, but when we're talking about TCO, more servers even if they're virtual isn't the answer. The additional support and management cost associate with managing 2-4 times as many servers not to mention the added cost for your favorite virtualization software (including staff and training) has to equal or exceed the cost of installing Enterprise Edition and optimizing utilization on a single server. I'd like to see a more detailed comparison that takes into account the TCO for both models.
Aug 06, 2008
Daniel Feller says:
Agreed about the cost justifications and TCO analysis for either going to enterp...Agreed about the cost justifications and TCO analysis for either going to enterprise for Windows 2003 or for virtualizing. Each virtual server you add is just like a real server, it must be patched, managed, maintained and fixed when issues occur. So, what if we could reduce the number of servers you have to manage?
Manage a role instead of a server. In the XenApp world, you only have a few roles (XenApp application group 1, XenApp application group 2, Data Collector, Web Interface, License Server, Data Store). In most XenApp environments, you have 6-10 unique roles, which can equate to 20-1000+ servers. If you look into Provisioning Server, you can now manage the role. I maintain and patch the role and all of the servers that use this role will be automatically updated. Much easier to maintain. Watch this webinar if you want to know more: https://www1.gotomeeting.com/en_US/island/webinar/provideEmail.tmpl?Action=rgoto&_sf=2
Aug 05, 2008
Helge Klein says:
The article contains several errors. Here are the corrections: 1) On Wind...The article contains several errors. Here are the corrections:
1) On Windows Server 2008 memory can in fact be taken away from an underused kernel area like non-paged pool and reassigned to an overused area like system PTEs. The feature works fully automatic and is called Dynamic Kernel Address Space. Read more about it here:
http://blogs.sepago.de/helge/2007/07/30/windows-server-2008-architectural-changes-relevant-to-terminal-services-1/
2) Applications need in fact not be specifically designed to use RAM above 4 GB. This is all done auto-magically by enabling the PAE boot switch, which is enabled implicitly on all systems that have DEP turned on. With PAE 5 processes can consume 2 GB each on a system with 12 GB RAM.
Only if one single 32-bit process needs more than 2 GB RAM it needs to be programmed with support for AWE. But that is not the case on terminal servers, the type of server Daniel Feller is referring to.
I have written another blog article about how and when to use more than 4 GB RAM on 32-bit Windows:
http://blogs.sepago.de/helge/2008/05/25/x64-my-terminal-servers-run-just-fine-with-32-bits-and-81216-gb-ram/
Aug 05, 2008
Daniel Feller says:
Helge Great points. You are correct, Windows 2008 can reassign these memory are...Helge
Great points. You are correct, Windows 2008 can reassign these memory areas, but Windows 2003 or 2000 cannot.
Also, to be able to address the higher levels of memory, you end up using more page table entries. In a XenApp environment, I've seen numerous implementations stuck with low utilization of the servers because they either ran out of page table entries, paged pool or non paged pool. Unfortunately, PAE will not help you out there as PAE only helps in the user part of memory and not the kernel part. So, adding more RAM and enabling PAE won't do much.
Now, of course, there are cases as well were the PAE switch will help and I bet numerous people have seen it work. But in cases where it doesn't work because of the lack of kernel space, virtualization is a good option.
Aug 07, 2008
Anonymous says:
"Keep using Windows 2003 Standard but virtualize it with XenServer. Upgrade the ..."Keep using Windows 2003 Standard but virtualize it with XenServer. Upgrade the RAM on the physical servers so it can support 2-4+ virtual servers." No, don't use Standard Serve with XenServer. A Windows Enterprise License allows 4 VM per host and a DataCenter License provides Unlimited VMs per host. Using Standard, you will only be allowed to install 1 VM per standard license.
Anyways, just like in the physical world you can use the PAE switch you can also leverage the PAE switch in the virtual world. The advantage of the PAE switch in the virtual world is now you can easily tweak RAM assigned to the VM to workload. Maybe it's only 4 GB before you run out or kernel space or maybe it's 8 GB.
Yes, PAE does work and yes it has draw backs but I suspect the answer is a balance of the two... If you have a host with 32 gigs of RAM and your TS Workload taps out a 8GB of RAM then you should create 4 VMs and not 8 VMs with just 4GB each.
Joe Shonk
Jan 16, 2009
Mikael Laine says:
He didn't mean to use Standard as a host OS, XenServer won't even run on a Winte...He didn't mean to use Standard as a host OS, XenServer won't even run on a Wintel host.
IE, use XenServer on your hardware and 4 versions of Windows Standard edition as guests to optimize the memory usage.
Mikael Laine
Oct 10, 2008
Tord Bergset says:
OK... I guess I represent one of the people that you refer to in your...OK...
I guess I represent one of the people that you refer to in your original post - we use /PAE and are loving it
Prior to setting up our Citrix farm on new hardware we did extensive testing and found the following (if anyone would like to follow..):
HP Proliant BL465c (Blades) with 2,4 dual-core CPU's, 12GB (or 16GB) RAM.
Windows 2003 Enterprise edition with /PEA switch added to the boot.ini (removed /3GB switch)
This config easily gives us 100+ connections per server (we have set the limit to 110 sim connections)...
During testing we found that the OS (!) started to perform porely around 120 sim connections - hw was still OK!
...and here i sthe fun part - we run 200+ different apps on each server, including all the known "killers" like Lotus Notes, SAP etc.
In short - saying that /PAE isn't to be used is the "dumbest" thing I hear these days.
We have been running this config for more than a year and have seen absolutely no issues.
On average our servers run about 75-80 sim connections, still with apps "flying" and logon times less than 30 secs.
My point - Don't listen to what people tell you about the /PEA, just try it in your environment and I bet you'll have something to celebrate the following weekend
Tord Bergset
Jan 16, 2009
Mikael Laine says:
"In short - saying that /PAE isn't to be used is the "dumbest" thing I hear thes..."In short - saying that /PAE isn't to be used is the "dumbest" thing I hear these days."
So you're calling everyone that runs out of PTE's etc liers? =) Pretty daring statement. And yes you're right, it should be evaluated if more RAM and PAE will provide better performance, but it won't work in many cases. It completely depends on the scenario as stated in the article. However, using XenServer as a plattform will probably allow you to pump even more sessions out of your hardware that you're doing now. IE, that should also be evalutated =P
Another tip, running 200+ applications on each server sounds like a poor design decision, but whatever works for you =P
Jan 16, 2009
Neil Burton says:
I agree with Tord on this. Having previously been of the conventional scho...I agree with Tord on this. Having previously been of the conventional school of thought (PAE + Citrix = don't do it) I've done some testing recently and realised that it can make a vast difference to the number of logons supported, obviously if your servers are becoming constrained for user memory before kernel memory. In fact in my current environment on C-Class blades I am seeing a 15-20% increase in scalability just by enabling PAE on our standard edition / 4GB blades thus regaining 0.75GB of memory previously lost to the PCI-E hole.
In comparison I have been testing 4 x XenServer VMs per blade with 2 x vCPU and 4GB RAM each and the results are dire. It seems as though a 2 x vCPU XenApp VM becomes very stressed when subjected to user logons when already under a moderate (say 50%) user load and the resulting CPU spikes result in session degradation for all users. I'm about to try scaling the concept upwards to BL680's (24 core) with 6 VMs per host with 4vCPU / 8GB RAM (PAE) and FC SAN attached disk as I believe this will yeild better results, but then again each blade would be vastly more expensive.
Also with regards to 200 applications per server - I personally think this is an excellent design decision if it can be made to work. We are doing a similar thing here although all apps are virtualised with Softgrid - how do you know Tord isn't doing the same?
Jan 19, 2009
Mikael Laine says:
"obviously if your servers are becoming constrained for user memory before kerne..."obviously if your servers are becoming constrained for user memory before kernel "
Guess you missed when i agreed that it should be evaluated.
"Also with regards to 200 applications per server - I personally think this is an excellent design decision if it can be made to work. We are doing a similar thing here although all apps are virtualised with Softgrid - how do you know Tord isn't doing the same?"
I wouldn't say that i'm running 200+ applications per server if they are not up and running on the server. Or well, i would if i actually streamed and had userload on the applications simultaniously. But i'm all in if someone can load 100+ users, 200+ applications on those blades and still have a good user experience. Just find it a bit hard to belive. That's why i read it as loading the apps localy.
Regarding your xenserver tests, did you try with single vCPU's and more machines / blade?
Jan 21, 2009
Neil Burton says:
Mikael - sorry if I misinterpreted the point you were making. We didn't bother ...Mikael - sorry if I misinterpreted the point you were making.
We didn't bother testing 1vCPU - although I am aware others have had success with this - as we have some good reasons for wanting to minimise the number of logical XenApp servers in the environment - many due to some per-server licensing costs. Even if virtualising allows us to have twice as many users per blade, if the number of logical XenApp servers quadruples then we're barely any better off in cost terms and we end up having to support an overall more complex environment.
I'm just running a few more tests but it is highly likely we will end up running on physical machines with Enterprise, 8GB RAM and PAE. In order to mitigate the environmental disadvantage of not virtualising, I am proposing we look at trialling PowerSmart as all our blades have ILO2.
Neil
Jun 09
Anonymous says:
What is the correct syntax for using the /PAE switch in the boot.ini?What is the correct syntax for using the /PAE switch in the boot.ini?
Add Comment