Wan Optimization Survey:

Take this quick survey to tell us more about the solutions your organization uses to optimize your WAN.

1. Do you currently have a WAN optimization solution in your IT environment?	Choose
Yes (go to Q2)
No (go to Q3)

2. Which, if any, of the following WAN optimization solutions does your organization currently use? (check all that apply)	Choose
Citrix WAN optimization product(s) (WANScaler, Branch Repeater, Branch Repeater with Windows Server)
Riverbed
Blue Coat
Expand
Other (please specify in a comment)
I don't know

3. Approximately what average percentage of your organization's overall network traffic is via XenApp (ICA)?	Choose
0%
1-25%
26-50%
51-75%
76-99%
100%
I don't know

Thank you If there is anything else you would like to tell us, please leave a comment.

On Oct 15th we released HRP03 for PS 4.5 on Windows Server 2003 and XenApp 5 on Windows Server 2003. Traditionally XenApp HRPs (Hotfix Rollup Pack) are a bundle of bug fixes and hence not that exciting. But in HRP03, in addition to XenApp bug fixes we included IMA optimizations that provide the following benefits.

Improved Performance

• Faster load balancing (up to 65% in >250 server farms)
• Time to bring up a large global farm with multiple sites is dramatically reduced (the farm-wide start up time was improved by 25% on physical machines and 59% on VMs in >50 server farms)
• Improved replication of session and load data over slow WAN links

Higher availability

• Provide user access resiliency (avoids periods when new users are denied application access for several minutes)
• Fewer data store corruptions
• Data collector election resiliency (69% improvement in election time with 2,000 applications and 95% improvement with 50,000 sessions)
• Better diagnostics
• Optimized caching of user and group information on XML brokers to better handle environments with tens of thousands of users

These updates are further explained within CTX118658 and CTX118659. Also check out the 10 minute webinar explaining the technical details of these optimizations.

You won't be surprised to hear that we spend a lot of time improving XenApp performance when running on XenServer. Although there are some good benchmark comparisons available (such as the Tolly Group report), I still get a lot of customers asking about what the "secret sauce" is. I sat down with George Dunlap, the lead XenServer performance engineer to chat about the very first optimisation we did back in XenServer 4.0 last year.

Before we dive in, we first need to explain how a normal operating system handles memory. George explains:

"Modern desktop and server processors don't access memory directly using its physical address. They use 'virtual memory' to separate the addresses that processes use to read and write memory from the actual memory itself. This allows operating systems to hide from processes all the dirty details of how much memory there is, where in physical memory the process needs to write to, and so on."

"However, the actual processor still needs to translate from a virtual address to the physical memory address in order to actually read and write any memory. This translation is done with something called page tables."

"Page tables are used to implement virtual memory by mapping virtual addresses to physical addresses. The operating system constructs page tables using physical memory addresses, and then puts the physical address of the "top-level" page table into a hardware register called the 'base pointer'. Then the processor will read these page tables to translate virtual addresses to physical addresses as needed, before reading and writing to physical memory."

Most modern processor types have some sort of paging mechanism, although XenServer is specifically tuned for x86-64 CPUs. An excellent book on the general topic is Modern Operating Systems by Andrew Tanenbaum. When XenServer creates Windows VMs, it takes advantage of the virtualization extensions in modern CPUs, which requires special memory handling in Xen. George explains this further:

"When we create a virtual machine, we virtualize the memory as well; that means that the guest operating system's idea of physical memory does not match up to real physical memory on the host. Traditionally, what the guest thinks of as physical memory is called "physical memory", and what the hypervisor thinks of as physical memory is called "machine memory". Since this terminology is a bit confusing, Xen tends to call what the guest thinks of as physical memory as "guest physical" memory, just to help make things more clear."

"This means that any fully-virtualized operating system, like Windows, will create page tables using guest physical memory, and will point the base pointer at the guest physical address of the top-level page table. Unfortunately, the hardware still needs to map from virtual memory address to machine addresses, not guest physical addresses."

"In order to allow this to happen, the hypervisor sets up shadow page tables. These page tables are generated by the hypervisor are copies of the guest page tables, but with the guest physical addresses converted into machine physical addresses. The guest cannot access them directly, and they don't reside in the guest's physical memory; they're generated out of a pool of memory that the hypervisor allocates when a VM is created, called "shadow page table memory"."

"What this means is that whenever the guest operating system wants to map some new memory, after it writes the data into the page table but before it can actually use it, the hypervisor needs to translate the change to the guest page table into changes to the shadow page table. So any workload that involves a lot of this will necessarily involve the hypervisor a lot, which causes overhead."

So shadow page tables are our mechanism of giving a guest an interface which is identical to real hardware (so it doesn't need to be modified), but still intercepting changes before they reach the real hardware. You can find more details from the XenSummit 2006 talk or from the 2005 NSDI paper. So how is this all relevant to XenApp performance? Back to George...

"The hypervisor allocates a certain amount of memory for each VM to use for shadow page tables; this is called shadow page table memory. As new page tables are created and old ones aren't used anymore, the hypervisor cycles through this shadow page table memory. When it needs a new page and there isn't enough, it will 'unshadow' the guest page tables that haven't been used for the longest time to reclaim shadow memory, so that it can use more."

"We don't know ahead of time how much shadow memory a given workload will use, but we can estimate based on the amount of memory that the VM has. We allocate enough shadow memory for each page to be mapped once, more or less, then add an extra 50% to have some slack. For all the workloads we've tested, that's been enough – except XenApp."

"XenApp is the one workload we've found that requires more shadow page table memory than our standard default. Because XenApp generally starts hundreds of copies of the same process, the same memory ends up mapped in hundreds of different processes. What happens when all of those processes are active is that XenServer is continually unshadowing one process' page tables in order to shadow another process' pagetables; only to have to re-shadow the original ones a second or two later when it runs again! This is called thrashing, when there's not enough of a limited resource."

Once the bottleneck was discovered, the solution was simple. In XenServer 4.1, we created a special XenServer application template called "Citrix XenApp", which has an increased shadow multiplier that reserves more shadow memory for the guest when it starts. This is also a good example of how templates hide the complexities of performance tuning from the user, but still permitting custom modifications if they are required. For example, on your XenServer host with a VM called "XenApp", you could view the shadow multiplier by using the CLI:

# xe vm-list name-label=XenApp params=HVM-shadow-multiplier
HVM-shadow-multiplier ( RW) : 4.000

The same value is also available from XenCenter in the Optimization pane, but of course do remember that the default value was chosen through extensive testing and doesn't need to be changed. Most of the other templates in XenServer also have carefully tuned settings (e.g. the hardware platform flags) to ensure smooth running, or in the case of Linux templates, to support para-virtual installation. This is why it's so important that you not use the "Other Install Media" template in preference of a more specialised one!

I mentioned at the beginning of this post that this was the first of many XenApp optimisations. We've just released the public beta of the latest XenServer ("Orlando") which is even faster. The story of what those improvements are, and the tools which George and his team uses to analyze the inner workings of Xen, are a topic for a future post. For now, get downloading XenServer and start virtualizing your XenApp installations! Or if you're feeling inspired, go over to xen.org, check out the source, and get coding...

Presentation server 4.5 added some cool features to optimize ICA performance. The improvement is most visible over slow links. To set up a test environment to see the difference requires multiple machines and some ways to control the network characteristics such as bandwidth, latency, packet loss rate etc. Products such as Shunra can be used to simulate the network. http://www.shunra.com/vedesktop desktop version costs $499 dollars though the last time I checked.

I just found out that vmware workstation has the team feature that allows virtual machines to be grouped together and connect using a virtual LAN. You can specify the virtual LAN bandwidth and packet loss rate. You can change the latency of the virtual LAN though. The ability to change latency would be a nice feature to add I think. This allows a relatively quick way to conduct some comparison testing in a controlled environment.

am using vmware workstation 6.0. I am not sure about vmware workstation 5.5.

is an example of what I tried.

a new team

an XP VM to the team

a CPS 4.5 VM to the team

a new virtual LAN for the team

both vms to use the virtual LAN

Since vmware workstation doesn provide automatic DHCP server for the virtual LAN, I configured static IP addresses for both VMs. You can configure static IP addresses in the Guest OS. Make sure you set the network mask and default gateway correctly.

vms can now communicate with each other. They are not directly connected to the outside network. Now you have an isolated environment to conduct your test. Optionally you can enabled shared folders or/and host only networking to move data between your vms and your host machine.

an ICA connection from the XP VM to the CPS VM. Launch a test application. I used google earth from http://earth.google.com/. It worked really well for me. When the internet connection is not available, google earth will use cached data. If you would like to run your test in an isolated environment to reduce variables caused by internet connection, you may want to enable only the private network. Google earth worked really well for me. You can try rotating the 3D globe. Such operation is typically a challenge for remote display protocols.

Shut down the team, change the virtual LAN characteristics and try again. Note that you will need to restart the team before the changes to the virtual LAN takes effect. I wish VMWARE would remove this restriction in the future.

make some changes to CPS settings (create new policies for example) to see for yourself how well the new features work! Please note that you will need to open the old Citrix management console to configure policies. I heard the development team is still working on moving all features to the MMC based Access management console.

also possible to add another virtual network card to allow one or more VMs to gain access to external networks. I will try to write more on that topic in a separate post.

also tried RDP connections and compared the results.

it yourself and share your experience with the rest of us...