For those of you who attended the TechTalk on XenDesktop Technical Dive, I wanted to post the videos maintenance videos. 

Remember, a virtual desktop solution must be able to simplify maintenance or else you are simply moving the administrative problem from remote sites to the data center. The first video shows how easy it is to patch the Hypervisor (XenServer).  The running virtual machines are automatically moved to another available XenServer without impacting the users. 

XenServer Update Video:

The second video shows how thousands of users' desktops can be patched easily without requiring a significant amount of time or expense with the use of Provisioning Server. 

Provisioning Server OS Images Update Video:

These are just two examples of maintenance for XenDesktop. The incorporation of XenApp and application streaming greatly simplifies the maintenance of application delivery.  If you want to hear more, take a listen to the recording of the TechTalk which can be accessed from here.

Thanks

Daniel

Homer Simpson Quote of the Blog (What do we need a psychiatrist for? We know our kid is nuts.)

We have seen the materials, at a high-level, on how the XenDesktop solution fits together and the benefit it can provide.  Are you interested in understanding more detail of the end-to-end solution? 

In this 60 minute webinar, I will provide you with a very quick overview of the complete solution and then spend the majority of the time discussing the different components, what they are for, how they work and how virtual desktops are managed by the solution.  We will cover the integration of the following components: 

• Desktop Receiver
• Access Gateway
• XenDesktop Controller
• XenServer
• Provisioning Server
• XenApp
• Citrix User Profile Manager. 

It is sure to be a great time where we will all learn a lot.  And I might even explain to you on how XenDesktop relates to a Simpsons episode. 

By the end of the webinar, we will all be able to understand the following song:

Desktop Receiver connected to the Access Gateway, Access Gateway connected to the Web Interface, Web Interface connected to the XML Broker, XML Broker connected to the IMA Service, IMA service connected to the Data Collector, Data Collector connected to the Pool Service, Pool Service connected to the XenServer and that's how the whole thing works

See you there and don't forget to register here.

Daniel 

Like I said in the recent TechTalk on server virtualization for XenApp, because there were so many questions, i was going to post answers to them all in a blog.  And this is the blog. 

First, many of you wanted the addresses for the reference materials i identified in the webinar. Here they are:

• TechTalk Webinar Recording: 
• Reference Architecture
• Design Considerations
• Implementation Guide

http://xenserver.citrix.vivoconcepts.com/prg/form/Citrix_runningxenapponxenserver_080225.cfm 

Q: Is all this done on a Citrix appliance or is it all software based and we provide the hardware?

A: Running XenServer is all software based.  You install XenServer, which takes roughly 10 minutes, on a physical server.  From there you can split up the physical server into any number of virtual servers.  A free version of XenServer Express and an evaluation version of XenServer Enterprise can be downloaded here: http://www.citrix.com/site/SS/downloads/results.asp?productID=683148* *\\

Q: What is the best resource for researching the possibilities of XenApp?

A: With regards to virtualization and recommendations, I would suggest the following materials, which covers different types of configuration, practices, considerations, how to do it, and much more. 

• TechTalk Webinar Recording: 
• Reference Architecture
• Design Considerations
• Implementation Guide
• Optimizing XenApp Performance with XenServer 4.1.0 Enterprise Edition
• Performance Evaluation of XenApp with XenServer
• Benefits of Virtualizing XenApp with XenServer* *
  

Q: What about network utilization with regards to Provisioning Server?

A: Network utilization is important for Provisioning Server in that the operating system image is streamed down to the virtual server.  With a base Windows 2003 Server, the install size is roughly a few GB.  However, Provisioning Server does NOT stream that entire image to the virtual server.  Provisioning Server ONLY streams materials as needed.  In fact, booting a Windows 2003 Server only streams a fraction of the multi-GB actually used in the install.

Q: Is Network Storage iSCSI or Fiber connection?

A: When you virtualize the disk with Provisioning Server, you essentially do not have any storage assigned to the virtual server.  Yes, you read that right, you don't assign storage to the virtual server because the image is streamed on-the-fly.  It is actually pretty wild to think about. Provisioning Server should be on an enterprise storage solution like a NAS or a SAN for high-availability and high speed of delivery to the virtual server. 

I know the first time I had discussions about Provisioning Server I was like, what do you mean there is no disk.  But it is true.  If you stream to the physical server, you can completely unplug the hard drives. In the virtual server world, you just don't assign disks to the server. With this type of solution, you end up not having to worry about GBs and GBs of storage required for a virtualized XenApp solution.  In fact, I've seen customers use Provisioning Server to help them migrate to newer versions of XenApp.  Right now, let's say you are running XenApp 4.5 installed on physical servers. When the next release of XenApp arrives, you create your image with Provisioning Server and stream the image to the servers (physical or virtual).  If you run into challenges with the new version of XenApp, your fallback procedure is to use the hard drives again, which still contains the XenApp 4.5 installation. Pretty cool if you ask me. 

Q: Would XenServer bundle with P2V tool for free? Or we have to buy PlateSpin P2V tool?

A: The P2V tool, when it is released, will be free. You won't need to buy any third-party tools to do P2V conversions. 

Q: We have VMware ESX Enterprise already. Should we get XenServer for our XenApp farm? What are the advantages?

A: I'm not a Sales person so I don't recommend products just because it is what we sell. So when talking about virtualizing XenApp, first understand that XenApp is a unique beast.  It doesn't behave like other systems within the data center. It must support hundreds of users simultaneously.  This requires lots of memory, lots of context switching, lots of disk access. It is different than let's say Exchange or SQL Server.  Before XenServer 4.1, I would have been hard pressed to recommend XenServer as a viable solution for XenApp. In fact, most virtualization solutions would not have dealt with XenApp effeciently.  But look what happens when XenSource became part of Citrix.  Our engineers (XenApp and XenServer) worked together to re-architect the hypervisor to perform remarkably better for XenApp virtual machines as compared to the 4.0 version of XenServe. That being said, XenServer is optimized for the XenApp workload.   Instead of making you perform some low-level funky "tweaks" the hypervisor, we just have you select the type of workload the virtual server is doing.  In this case, you select XenApp. This option changes how XenServer deals with memory to align better with XenApp requirements. 

Now, when you look at XenServer Platinum the picture becomes even better with Provisioning Server.  Without Provisioning Server you must still manage each virtual server as if it was a physical server. This is regardless if you are using XenServer, Hyper-V or even VMware.  Provisioning Server lets you focus on the role and not the server.  There are fewer roles in the data center than there are servers. Easier to manage and maintain, a huge savings if you ask me.  And you did   But I did only touch on a two areas.  Take a look at the documents (especially the reference architecture) I put at the beginning of this blog for additional information.

Q: what were your server specs for your example?

A: The scalability testing completed with XenServer and XenApp were done on a Dell PowerEdge 1950 (1 Quad-core 1.6GHz, 8GB-16GB RAM). 

Q: What about users that are logged into an app, and the server is rebooted

A: A physical server, virtual server or a server receiving the image from Provisioning Server, those users are disconnected and their sessions are gone.  Now if the physical XenServer fails, the virtual XenApp servers can be moved to another XenServer, a feature we call XenMotion.  In this circumstance, a user might see a slight pause in their session, all depends on the current situation. But the point is in this situation, the users session and data is intact.

Q: You mentioned doing P2V of Citrix servers throughout your presentation.  Are there any items to be aware of when doing this?  Any resources to help with this process?

A: Well, the first is an upcoming P2V tool that will let you convert a physical server to a virtual server for XenServer.  If you only use XenServer and not Provisioning Server, the only other item is to set the optimization setting for the virtual server to Optimized for XenApp.  This was discussed earlier in this blog.  If you are also going to stream the system with Provisioning Server, you will want to build the "golden image" how you want it to be for each server. You then must run the integration utility, which will take care of all the other configuration items.  If you want instructions on how to do the Provisioning Server aspect, take a look at the Implementation Guide identified at the beginning of this blog.

Q: Did you use Provisioning Server w/ the test load, or just straight XenApp on XenServer?

A: The scalability testing was just XenApp on XenServer.  I can bet your next question will be what impact on scalability with Provisioning Server have.  And might I say it is a great question if I do say so myself.  Unfortunately, I'm not aware of any scalability testing that shows the impact to single server scalability with Provisioning Server. 

Q: How is XenApp rated on VMware ESX vs. on Provisioning Server on XenServer?

A: Unfortunately, due to VMware's end user license agreement, we are not able to publish scalability numbers for VMware ESX.  No one can except VMware.  We did tests against a number of virtualization vendors and found that XenServer allowed roughly 70% more users than others when running 64Bit XenApp.

Q: How large would a server image be with Provisioning Server?

A: The size of the Provisioning Server virtual disk, which I call a role, can be pretty much any size.  However, you don't want to go wild with the image size.  If you create a 10GB image and a 100GB image, the 100GB image will take a lot longer to build and optimize, plus it will waste space and we are all trying to conserver power, space, cooling, etc.

Q: What is the best client to use - PN, PNA or WI?

A: You tell me   It really depends on what you need.  Most administrators prefer PN as it allows them to make connections as they need to support the environment.  Users prefer PNA or WI.  PNA is great in that you don't have to go to a web page to get to your applications, so it is faster from a user perspective, but WI allows you to integrate the published resources in other pages like SharePoint. I personally use Web Interface and my favorite color is green.

Q: How can one discern how much RAM/CPU is being used on a daily basis?  Does Access Suite Console give that info?  (Am on PS 4.0 and use VMWare)

A: Within the XenApp Access Management Console, you can generate reports for your XenApp servers to give you all kinds of information about the overall utilization of the servers. The reports are in the Report Center. Also, you can use Resource Manager or EdgeSight to get even more detailed information.

Q: Is there a release date for the P2V tool?

A: All I can tell you is the beta is expected soon.  I would log onto MyCitrix and see if you can see it in the download section. Also, Roger Klorese has been blogging about the next version of XenServer (Project Orlando). I recommend taking a look at his blogs.

Q: Is there a guideline for application roles that are not suited for XenServer virtualization?

A: Hmmm, I'm trying to see if I can think of an application that is not suited for XenServer, but I'm having trouble.  Before XenServer 4.1, I would have probably said XenApp due to the overhead, but now that the overhead has been drastically reduced, I can't say that anymore. 

Q: What file system do you recommend for the storage partitions on a NAS or SAN?  (I think VMware has a proprietary clustered file system, Novell uses OCFSv2).

A: This is what I love about XenServer and Citrix.  You can use anything you want.  NAS, SAN, NFS, iSCSI.  If you already use something like NetApp, use it.  If you use a SAN solution, use it for XenServer. 

Q: Nice to see how memory issues can be addressed with virtualization. What about CPU, network, and disk I/O being the bottleneck?

A: Excellent question. 

• CPU:  Not sure what issues are around CPU in the XenApp world except for CPU underutilization because of memory bottlenecks and memory limits.  Virtualizing lets you completely use what you paid for. 
• Network: The networking aspect is interesting.  Because the physical server is now hosting multiple virtual servers, you want to make sure you have adequate bandwidth going into and out of the physical server. The network component is critical to XenApp, but the data transferred is fairly minimal due to the use of ICA.  Now on the backend, the XenApp applications require data from their source. And if Provisioning Server is being used to stream the operating system, more network bandwidth is required.  But these should still be within the limits of the current standard server hardware of 1GB NICs.  However, I would still recommend mulitple NICs to a single XenServer. You don't want a Homer Simpson tripping over a network cable and dropping all users from a XenServer.
• Disk I/O: In a enterprise design, I would recommend you use some type of fast storage like a NAS (regardless if you use Provisioning Server or just plain XenServer).  These devices are specialized hardware optimized for file sharing.  I have had customers tell me that their XenApp environments actually run faster because of XenServer and the integrated NAS/SAN.  

Q: is the benefit presented on this slide in the fact that Disaster recovery is improved by virtualizing?

A: Disaster recovery is improved. With XenServer you can move a running virtual server to another XenServer. Provisioning Server also helps in the DR scenario as you can quickly re-provision systems to take on a new workload with a simple reboot. 

Q: You don't have to have 32-bit apps to run on 64-bit OS.  That's where you get your scalability on XenApp

A: True, you can continue to run your 32bit apps on a 32bit OS like Windows 2003.  The problem is that you have a memory limit with 32bit OS.  In more cases than not, you will max your RAM before you get close to maximizing your CPU. 

Q: We have XenApp 4.5 running on a dev/test environment in VMware. Session connection times seem to be slow for an app to open up. What kind of things should I be looking at to find the source of the problem outside of adding hardware to the VM. thanks!

A: Well, first I would say try using XenServer  (I know, I had to say it)    But seriously, take a look at the storage situation with your VMware implementation. What performance numbers are you getting from the I/O system in your setup?  With XenServer, we recommend you use either a NAS or SAN type solution which provides the fastest possible disk performance.  The faster your disks run, the faster apps load because they are coming from disk.

Q: What technology are you using to determine user count?

A:  We are just performing scalability tests with tools like EdgeSight for Load Testing and then to get the metrics, we utilize perfmon counters and log files to analyze the results and make comparisons.

Q: I'm a bit new to XenApp but your numbers for concurrent users seemed very high. If your Visio app is using 1Gb of RAM just for you, doesn't' that mean that a max of 15 people could use Visio on a XenApp server?

A: In that example, yes.  However, it all depends on the apps.  For example, the scalability numbers I presented for roughly 300 concurrent users on a physical server were working with Excel only.  This was used to determine overhead. Your concurrency numbers will vary based on workloads. The scalability numbers are meant to give you an idea of the XenServer overhead.

Q: Another point is disk utilization... we are often disk bound

A: Yes, disks can be a problem. Sure you can implement array controllers, use 15K RPM drives, but you are still relying on the local system to manage the file system. When you integrate with a SAN or a NAS type solution, those devices are optimized for file hosting.  Optimization=Speed

Q: I have already heard that you can not clone XenApp servers... where can I learn more about this?

A: Read the Reference Architecture document identified at the beginning of this rather long blog. It talks about the Provisioning Server Integration Utility for XenApp.

Q: How do you get 200 users in 4GB of RAM?

A: By running Excel only.  This workload is used to show the overhead impact between 64bit physical, virtual and 32bit.  You workloads and your concurrency numbers will be different. This is to give you an idea of the expected overhead.  I've actually seen other people get their XenApp servers into the upper 100s by using bigger applications than Excel. It all depends on the apps and users. That is the main problem with scalability tests, they only reflect a single type of workload and do not represent your environment. They are only meant to give you an idea of what the overhead is and comparisions against other products.

Q: Any tool like ESX Ranger becoming available?

A: ESX Ranger has many different features.  I know you would be able to use something like Workflow Studio to help manage the environment from user-based, event-based or schedule-based triggers.  As this product is still in beta, it is hard to tell what functionality and integrated components will be available at release.

Q: Isn't XenServer only supported on 64-bit platforms?  How then would we virtualize a 32-bit Server with your scenarios?

A: XenServer is a 64bit hypervisor, but it can virtualize 32 and 64bit systems. 

Q: Did you reference a 32bit version of XenServer?

A: XenServer only exists as 64bit.  There is no 32bit code in the XenServer.

Q: What about PAE on 32-bit systems?  This allows more than 4gb of ram to be addressed. 

A: I wondered if someone was going to ask about that.  Congratulations.  You can use more than 4GB of RAM on a 32bit system, but there are a lot of things you must be aware of.  Instead of making this blog even longer, I created another entry that discuss the PAE setting, which can be found here:

Q: Why would you keep a backup data store if you can just motion it instead?

A: In the event the live data store is corrupted and is unrecoverable. If it is, I hope you have a backup. 

Q: What are your thoughts on virtualizing Provisioning Server?

A: The great answer, it depends.  Virtualizing Provisioning Server will induce latency into the stream as it must go through a virtual network and then onto the physical network to the device.  However, being able to hot-move the Provisioning Server to another server and easily add capacity makes virtualizing a very sound solution.  I haven't seen any numbers yet as to what virtualizing Provisioning Server would do to the scalability.

Q: Running published desktops. Can I virtualize these?

A: Published desktops on XenApp, yes you can.  If you are talking about XenDesktop, desktop virtualization, VDI, whatever else they call it these days, you can as well. In fact, Citrix XenDesktop is also based on the XenServer hypervisor.

Thanks

Daniel

Homer Simpson Quote of the Blog "If you really want something in this life, you have to work for it. Now quiet, they're about to annouce the lottery numbers!"

Memory is a big concern for XenApp on a 32bit operating system like Windows 2003 Server.  In the default state, Windows 2003 can only "see" 4GB of memory, which is split up into two equal parts: Kernel Memory (2GB) and User Memory (2GB). Kernel Memory is further broken down into 4 other parts:

• Paged Pool: Memory space used by the system and kernel level components that can be paged out of physical RAM and into the page file
• Non Paged Pool:  A section of memory guaranteed to always reside in physical RAM and is used by the operating system for certain kernel level processes
• System Page Table Entry : An index table that tells the operating system where the virtual memory actually resides in physical RAM or on the page file
• System Cache: Maps open files in memory for better performance. This is where the registry hives are located as well
  

Once the system has started, the different sections of kernel memory cannot be re-allocated. The system tries to allocate these 4 areas appropriately, but they might require "tweaking". However, the four areas cannot all be set to the maximum level as that would go over the 2GB limit of kernel memory.

Many of you are probably saying, "But I can use the PAE switch on Windows 2003 to go above the 4GB limit". You are correct, you can go above the 4GB limit, but are you aware of the consequences of this action?

1. You must be using Windows 2003 Enterprise or Data Center.  This setting does not function in Windows 2003 Standard.
2. The PAE Switch does NOT change the kernel memory limitations of 2GB
3. To use the extra RAM, more System Page Table Entry memory is used
4. If you have more System Page Table Entries, you will end up with less Paged Pool, System Cache and Non Paged Pool
   

Talk about being between a rock and a hard place. Adding more RAM and enabling the PAE switch "might" give you more scalability but at a great cost for a more expensive operating system, more RAM and special optimization configuration analysis and implementation. The reason I said "might" give you more scalability is because you will now likely run out of kernel memory before you run out of user memory. So you just bought a more expensive operating system and more RAM that will sit there wasted. 

Now I know some of you will add a comment saying something to the effect that you are using the PAE switch and ended up increasing single server scalability by 60, 70, 80 or even 90%.  All I can say is congratulations and I applaud you  . You are lucky as you have the right set of apps for this to work as well as it has.  But I want you to think about going down a completely different route.  Virtualization...

Keep using Windows 2003 Standard but virtualize it with XenServer. Upgrade the RAM on the physical servers so it can support 2-4+ virtual servers. In the end, you will end up with a system that is more flexible, scalable and easier to manage. 

If you interested in learning more about sever virtualization for XenApp, then take a look at the following:

• TechTalk Recording: Make Server Virtualization work for XenApp (http://www.citrix.com/English/NE/events/event.asp?eventID=1679445)
• White Papers
  • Reference Architecture
  • Implementation Guide
  • Design Considerations

Daniel

Homer Quote of the Blog: "To be loved, you have to be nice to others EVERYDAY! To be hated, you don't have to do squat."

Does anyone care about having high-availability for their XenApp farms?  I would envision many of you would say yes.  But what does HA for XenApp really mean?  On the server hosting side, you essentially have HA because you have load balancing at the application level. So if you lose a XenApp server, not too much of a concern as those users can simply restart their application and get load balanced to another server (of course they lose their previous session information, which can be annoying.)   But what other areas of critical to providing a more available XenApp environment?

I've been thinking about this a lot lately, which is probably because my manager has had a lot of meetings and I tend to space out and watch episodes of The Simpsons on my laptop.  Since my DVD player broke, I started to think about HA for XenApp during these meetings (at least I'm now doing work). I was able to come up with the following thoughts:

1. Smart Monitors:  First, I want to know that something has failed or has gone flakey.  I don't want a bunch of messages telling me everything is ok, I just want to know when something is about to go horribly wrong. For example, the XML Black Hole.  I've seen the black hole cause too many issues, so how do we detect it?  You create a smart monitor that does more than pings. It tries to make requests to the XML service. If the expected data comes back, we are good to go. If the request is never answered or the response is junk, then Homer, we have a problem.  
2. HA for the Critical Components: Now if we can detect a failure, DO SOMETHING ABOUT IT.  As we continue looking at the XML Black Hole, if we see there is an issue, then stop making requests to it. But this requires another XML Brokers to take over the responsibility of the failed one without requiring changes to the environment's configuration.   Sounds a lot like load balancing to me.
3. Business Continuity:  Essentially what I'm saying is that if my XenApp environment at one site fails, I  better have another site already waiting for connections without requiring me to make changes.  Many people have 2 data centers: a primary and a backup. Others have 2+ data centers that are all active.  For those organizations with 2 data centers (primary and backup), how do you fail users over to the backup in the event of a failure?  For those organizations that have 2+ active data centers, how do you tell your users data center is their preferred site?  That is really a trick question (Did I get anyone?).  You shouldn't have to tell your users anything about going to a primary, backup, tertiary site. It should happen automatically.  Users want their applications in the fastest possible means necessary, which could mean that one day it is from data center 1 and on another day it could be data center 2.
   

These three items are all part of NetScaler, and it is easy to setup.  For those of you who know me will notice that I've worked with the integration of NetScaler and XenApp for some time.  Well, the NetScaler product group is actually making my job easier because they are making this solution a lot easier.  I created and maintained a 40+ page document that showed you how to set all of these goodies up. Now that document is about 14 pages (with pictures for each step) because of the new NetScaler for XenApp wizards.  I'm just glad I don't get paid by the word.  Take a look at what I'm talking about. In about 5 minutes you will see me configure and integrate NetScaler with XenApp:

Watch this Video:

Also, take a look at recently released articles  that goes into more detail on this integrated solution: http://support.citrix.com/product/nsad/v8.1/consulting/

• Taking XenApp to the Next Level of Availability - Reference Architecture
• Taking XenApp to the Next Level of Availability - Implementation Gudie

I'm curious what other areas concern you when you are focused on HA for XenApp?   Let me know. Yes, my manager finally ended the meeting, I am outta here.

Daniel

(Homer Quote of the Blog "Kids, you've tried your best and failed miserably.  The lesson is, never try")

Welcome to the third installment of the Dynamic Delivery Center.  This time I will be showing you the Proof of Concept (PoC) we built to validate the DDC is possible.  If you haven't done so already, I encourage you to review the first two blogs so you understand our business and requirements.

• Part I - The Need
• Part II - The Analysis

Now, the PoC.  First, let me show you the architecture diagram we've used.  (Visio Stencils for this diagram are located here).

As you can see, it is fairly straightforward. I'm the type of person who prefers things simple. The whole purpose of the PoC is to see if we can use a web front end to dynamically deliver any number of "Test" environments to the users.  Now, as many of you reading this are techies, let's get to the good stuff...

• External Access: Every user will be remote. Even if you are sitting in the office next to the lab, you are considered a remote user (Ever hear of de-perimiterization?).   All users will connect through an HA Pair of NetScaler 7000s with the SSL-VPN functionality enabled in full VPN mode.  We are doing more than ICA so we need a full VPN connection.
• Web Front End: Users will be able to connect to the Web Frontend when they connect with Access Gateway Enterprise .  The Web Frontend will allow the user to request any number of systems from the lab. 
  
  
• XenServer Resource Pool: Currently, the XenServer Resource Pool contains a set of templates that users can request from the environment. Those templates are reflected in the Web Front End, allowing the user to customize their environment.
• XenServer Template Library: For the PoC, the library only includes Windows 2003 R2 servers, XenApp 4.5 servers, Windows Vista SP1 workstations and Windows XP SP2 workstations.  New virtual machines are created based on the templates, which should only take a few seconds.  The library will grow as new requests come in and new systems are required. The longer the DDC is running, the more complete the library will become.
• NetScaler: The NetScaler devices are setup to allow for either a one-arm or two-arm deployment (hence the reason for the two separate VLANs).  If the user only requires a one-arm setup, they just ignore the second connection.
• WANScaler: The WANScaler devices are setup to allow the user to test any number of backend optimizations across any simulated WAN connection with the Apposite WAN Emulator.  The backend contains another XenServer Resource Pool allowing the user to test WAN optimization against any number of resources including file servers, web servers, media servers and XenApp servers, just to name a few.
  

We have the architecture, but how does it work?  How does the Web Frontend do it all?  In the PoC, we chose not to look into Workflow Studio (Sorry WFS Team) as we want to wait for the next beta release. But the lessons learned in the PoC will help us properly develop our workflows in the design phase.  In the PoC, we used the SDKs extensively to do the following:

• Virtual Machines
  1. A user selects one or more templates on the Web Frontend and selects "Provision Servers".
  2. The Web Frontend code will search for a virtual machine resource in the database that has not been marked as in use. Once an open virtual machine is found, the database will be updated and marked as in use by the user for a period of 3 days.
  3. The Web Frontend will establish a session with the XenServer host using root credentials.  The template the user selected will be cloned.  Once the clones are created, they will be sent a start command.
  4. Once the virtual machines are running, the IP address will be obtained. This information is used to generate an automated email to the requester using the SMTP service running on the Web Frontend server.
  5. The user will use the IP address to make a Remote Desktop connection to the console of the server, which is waiting for the user to enter a name for the virtual machine as part of the SID generation process.

• NetScaler
  1. User selects "Provision NetScaler" from the Web Frontend
  2. The Web Frontend checks the database for an available NetScaler. Once one has been found, it is marked as in use by the particular user for a period of 3 days. In the event that all NetScalers are assigned, the user will receive an automated email.
  3. The Web Frontend will establish a session using XML API calls with the NetScaler using the "nsroot" credentials. The reset process involves using the XML API calls to get into the NetScaler shell to remove the ns.config file.  Simply deleting the NS.Conf will completely reset the NS config. That would be bad because that includes the IP Address.  We don't want to go into the lab and connect a serial port and configure the device.  To solve this challenge, we  copy a base ns.config (which includes the NS IP configuration) over the current one.  We also have the code go through and remove any extra files that the previous user might have created (certificates, configuration files, etc).
  4. The Web Frontend will send code that will clear the NetScaler configuration, while keeping the IP address constant, so it is accessible from the network.
  5. The user will receive an automated email from the Web Frontend using the SMTP service.  The email informs the user on the connection information for the NetScaler.

• WANScaler
  1. User selects "Provision WANScaler" from the Web Frontend
  2. The Web Frontend establishes an HTTP session with the WANScaler web console using the "admin" credentials.
  3. The Web Frontend sends a request to reset the WANScaler config back to factory defaults, while still preserving the IP address. Once the WANScaler has been set back to factory defaults, the WANScaler will be rebooted.
  4. The user will receive an automated email from the Web Frontend using the SMTP service.  The email informs the user on the connection information for the NetScaler.
     

Lessons Learned:

• The biggest thing is that it is possible!!  The tricky part of the project was resetting the NetScaler and WANScaler back to factory defaults without losing the IP address. 
• A more complete set of XenServer templates is required to anticipate any number of requests from the field
  

Next Steps:

• Create a more detailed design that identifies the templates required for the initial release
• Create a detailed set of workflows that are required to see how Workflow Studio can be leveraged to make this environment easier to build and maintain.
• Create a way to hide Simpsons "Surprises" within the lab like logging into the lab and receive a warm welcome from Homer saying "D'oh!"
  

Hope you enjoyed this one.

Daniel

Homer Quote for the Blog "Look, the thing about my family is there's five of us. Marge, Bart, Girl Bart, the one who doesn't talk, and the fat guy. How I loathe him."

In case you haven't heard or seen , I'll be hosting a live TechTalk on Wednesday, July 23rd at 1PM Eastern covering the virtualization of XenApp on XenServer. For those of you who have read my blog, I know there are 5 of you, will know that I've been working on this aspect of server virtualization for some time.  I plan on covering what you should virtualize, how you should do it and how to make dev/test environment better with this solution. So if you want to hear me talk on a great topic, don't forget to register here.

July 23, 2008

1:00 PM Easter

1 hour duration

Daniel

Shipoopi!!

(Homer Simpson Quote of the Blog: "Kids, you tried your best and you failed miserably. The lesson is, never try.")

It has been a few weeks since I first wrote about drinking the Citrix Kool-Aid and trying for ourselves to turn our lab into a dynamic deliver center.  The first part of this project is to identify what is the purpose of our lab.  After looking at things deeper, we are responsible for mainly 4 things:

Technical Readiness Infrastructure: 

The Tech Readiness group is responsible for creating and delivering technical training to our customer facing people. This includes Support, Consulting, SEs, etc.  This group develops hands-on training that walks the student through setting up, configuring and troubleshooting the product.

As you can imagine, during a new product release, we have classes stacked up one after another all around the world. These classes use prebuilt environments in a remote lab in Ft. Lauderdale.  So if a class is occurring in Singapore, London, Sydney, Paris or anywhere else , the students will connect to Ft. Lauderdale to work on the pre-configured lab environment. Because the focus of the classes is to train on the new features, we don't expect our students to run through installations.  This means the environment must be prebuilt ready for configuration.

As you can expect, this is a challenge, which brings us to our first few requirements:

• Requirement 1: Tech Readiness: Must be able to work with the latest products, whether they be hardware or software, remotely.
• Requirement 2: Tech Readiness: Must be able to refresh environment quickly to a base state with items installed, but not configured
• Requirement 3: Tech Readiness: All previous classes configurations must be removed and devices and servers must be put back to base state (including passwords, accounts, optimizations, etc)
• Requirement 4: Tech Readiness: Environment must be able to allow students to work with all features and products including WANScaler optimizations, NetScaler load balancing, XenApp Progressive Display, Smart Auditor, etc.
  

Worldwide Consulting Solutions

The Solution Center and Integrated Solutions team is responsible for developing best practices and recommendations for integrating Citrix products with other Citrix products and 3rd party products.  For example, these two teams have developed items discussing XenServer and XenApp integration and on how to integrate WANScaler and NetScaler with Microsoft SharePoint.  From project to project, the architecture could look quite different, but there is one common aspect to all projects... There is at least one Citrix solution involved. 

The types of projects can vary wildly from validating an application runs on XenApp, defining deployment best practices for a particular web application with NetScaler to performing scalability testing on the latest version of XenDesktop. All of these items come together to bring us a few more requirements:

• Requirement 5: Worldwide Consulting Solutions - Need to be able to bring up a set of Citrix solutions without requiring installation
• Requirement 6: Worldwide Consulting Solutions - Need to have the different project pods separated from other pods as one test might influence the results of another test
  

Field Teams

Working with our customer, many of our field Citrites find themselves in need of a reference system to be able to look up a setting, perform a quick test based on a customer question, or be able to demo a new feature that is easier to show than to explain.  These types of requests are short lived, but require a fast response. Because of the huge number of potential questions a customer could ask, it is impossible to anticipate every conceivable environment needed or when the requests could occur.  This type of situation brings about the following requirements:

• Requirement 7: Field Teams: Need to get access to a base system (regardless of product) in a short amount of time that can be modified. 
• Requirement 8: Field Teams: Each reference system must be isolated as customers will be seeing the systems in a lab that also contains systems of new, unreleased products
• Requirement 9: Field Teams: Need to be able to extend check-out time if work extends beyond original request date
  

Administration

I haven't hit all of the groups that use the lab because this blog would be longer than the movie script to the Simpsons Movie (which I highly recommend by the way), but most of their requirements are contained within the first three groups.  Before I close out, there are still a final set of requirements focused on the administration of the lab. We must make it easy and automated:

• Requirement 10: Administration: Systems should be in a low-power state unless they are being used.
• Requirement 11: Administration: The systems, when allocated, should be powered on automatically.
• Requirement 12: Administration: Systems should be automatically built, cleaned, decommissioned and assigned.
• Requirement 13: Administration: Hardware assignment notifications should occur through email with all pertinent connection information once the systems are online.
  

I know this was quite a long blog, but this is a big project and I didn't want to gloss over anything. 

Up next, a Proof of Concept.

Daniel

(Homer Simpson Quote of the Blog: "Oh, so they have internet on computers now!")

For quite a few years, I've developed a set of Visio stencils that I've used, along with many of my coworkers, to create detailed design architectures for a Citrix solution. The stencils have morphed over the years and I have now consolidated them into a single package.  This one stencil has just about everything you will need to create detailed Citrix solution diagrams.  Included are items for the following products:

• XenApp
• XenDesktop
• XenServer
• NetScaler (includes rack-mountable stencils and NetScaler MPX)
• WANScaler (including the new Branch Repeater)
• Access Gateway
• Application Firewall
• Provisioning Server
• EdgeSight
• Password Manager
• Workflow Studio

I'm always looking for ideas on what's missing so there can be one comprehensive stencil set for Citrix solutions. Drop a comment and let me know what else is needed.

Download Visio Stencils here

The last time I wrote about XenServer and XenApp, I focused on a whole set of items like manageability, availability, flexibility and utilization.  This time, I want to focus directly on utilization as based on the feedback I've received it seems it's the one many people are interested in.

Even before the scalability numbers of XenApp and XenServer came out, I had numerous conversations about virtualizing XenApp.  And now that Citrix is showing the XenServer overhead for virtualizing XenApp, those conversations have increased, but I think some critical points are being lost.  A couple of months ago, Citrix did scalability tests to identify that XenServer has roughly a 7-8% overhead when virtualizing 64-bit XenApp, and roughly 20% when virtualizing 32bit XenApp servers.  I was like WOW, 64bit is great, barely any overhead.  But how many people are actually running a truly 64bit environment?

Most people have the hardware, as it has been sold for years. Most people also have access to the 64bit version of Windows and XenApp. So why aren't we all jumping on the 64bit bandwagon? Because it's the applications. Unfortunately, many applications that XenApp environments run are 32bit, and some are still 16bit!  This conversion to 64bit applications will take time (Does anyone else remember the 16bit versus 32bit migration that happened years ago? It wasn't an overnight thing. It took time. And yet there are still 16bit apps out there.)  So this fact makes it highly unlikely that organizations will be able to convert their XenApp environments into complete 64bit setup. This means many will stay with 32bit only or else have mixed 32/64 bit environments.  So let's focus on the 32bit environments, are they virtualization candidates?

Maybe And most likely Yes.

Take a look at many XenApp deployments and what resource do you typically exhaust first?  RAM.  It is because in Windows 2003, we are limited to 4GB of addressable RAM.  So, when we hit that limit, everything else in the system is wasted (processor, IO and networking).  And I've seen some applications take enormous amounts of RAM. Just the other day I was working on some detailed Visio drawings and Visio took 1GB of RAM.  Yes, I said gigabyte.  (Of course the drawing was about the Simpsons and how Homer stays at the forefront of technology - He even had a blog called "Mr X. - All the Muck That's Fit To Rake").  So, a 20% overhead on 32bit systems? I probably wouldn't notice as my entire server is barely utilized except the RAM.  

RAM is easy to install and one of the cheapest things to add to a server.  Use the same hardware and increase the RAM to at least 8GB.  Now, try to run 2 virtual XenApp servers.  You might not double your user concurrency, but you will get pretty close, which will equate to hardware and power savings.

So take a look at your physical XenApp servers. Is the RAM fully utilized? What about the processor utilization levels?  I bet more likely than not the RAM is fully committed and the processors are running at 10-40% utilization. 

Daniel

(Homer Simpson Quote: "I want to share something with you: The three little sentences that will get you through life. Number 1: Cover for me. Number 2: Oh, good idea, Boss! Number 3: It was like that when I got here.")

Within Citrix,  I'm on the team called Worldwide Consulting Solutions whose overall goal is to provide our customers with the tools and resources to successfully implement Citrix solutions.  Talk about an easy task .  The interesting thing about this goal is our customers. Our customers are potentially any Citrite, Citrix partner or Citrix customer.  Within this team, we have something called the Solution Center.  It is essentially a lab, but with some serious requirements.  The Solution Center lab must do the following:

1. Be capable of supporting integration tests and scalability analysis with Citrix products and 3rd party solutions
2. Be capable of supporting our readiness activities where many of Citrix's field personnel are trained and tested on our latest products and solutions
3. Be capable of providing temporarily systems for solution testing for our consultants and SEs, where they can test a feature, or look into a potential solution for our customers.

This is quite a task.  The challenge with the lab is provisioning these systems out on an as needed basis. From week-to-week the environment changes based on the needs of projects, training and testing initiatives.  Let me explain how this process works now:

1. A user would make a request for hardware based on a SharePoint web page. This page contains numerous fields so the Solution Center team can figure out what hardware to provide for the project. Items include dates, server type, OS, quantity, applications, HW requirements and an overall description.  Once a user submits this request, it is up to the Solution Center team to work out the logistics.
2. The Solution Center team takes the requests and tries to identify available hardware based on a master scheduling SharePoint page. Once hardware is identified, the new systems are built based on the user's specifications with SysPrep and Ghosting techniques. Currently, the group is building a library of XenServer virtual images to simplify this process.  
3. Once the system is built, the user is notified with IP Addresses and connection procedures.  The entire time from request to delivery can be up to 2 business days.   

This group as they have a very difficult job and they do it great, but one would think there would be a better way.  We have started to build XenServer virtual images libraries to more quickly build the requested environment, but there still is a manual process involved that slows the entire system provisioning down. Is it possible to create a self-service system that would not require manual work from the Solution Center staff?  What if we could cut down the time to build these environments from days to minutes while providing 24x7 service?  A self-service system like this would allow the team to weasel out of this time consuming work.  "Weaseling out of things is important to learn. It's what separates us from the animals! Except the weasel." (Homer Simpson).  

During Synergy we heard all about Citrix's vision for a Dynamic Delivery Center on the big screen (It looked like a really big TV).  And again, Homer provides us with a good recommendation, "When will I learn? The answer to life's problems aren't at the bottom of a bottle, they're on TV!"  So, we started drinking the Kool-Aid and we are going to see if the Kool-Aid tastes good.  Can we turn the Solution Center Lab into a Dynamic Delivery Center?  We will all find out together as we have just started to work on this project.

I will continue to blog about our challenges and successes throughout the project, which will contain analysis, PoC, design and implementation.  You will see

• Our requirements are (and they are challenging)
• Our overall architecture design
• How we implemented it
• The final outcome
  

This is going to be a fun project allowing us to get into the details of all of the Citrix solution stack.  So, stay tuned for Part 2: Analysis

Daniel

(Homer Simpson Quote: I want to share something with you: The three little sentences that will get you through life. Number 1: Cover for me. Number 2: Oh, good idea, Boss! Number 3: It was like that when I got here.)

Being a Sr. Architect within Citrix for almost a decade, I've been asked by more Citrix administrators than I could ever count, wanting to know if they should virtualize their XenApp environment.  My typical response, which is common for a consultant, was "It depends."   Unfortunately, this is not an easy yes or no question.  XenApp is a unique beast in the delivery center.  Users don't interact indirectly with a XenApp server like they do other systems (database, web, etc). Instead, users work on the servers directly.  And if the servers have been designed appropriately, they should reach their memory limit or CPU limit.

Let's say, for example, your business is to write screenplays for "The Simpsons" and you have a set of XenApp servers hosting a single application for storyboarding.  This application is critical to the business.  On average, throughout the day, the CPU is 60% utilized and the memory is 80% utilized (4GB on Windows 2003 Server).  What advantage would you gain by virtualizing this system?  The hypervisor WILL take resources.  Chances are slim you would be able to host a second virtual server on this physical system.  In this case, I don't see where server virtualization fits. You could add more memory and additional CPU sockets, but you are spending more money just to try to save money.  Of course there are some XenApp servers that are underutilized.  Why?  Was it an improper design? Or was there a business reason? With underutilized severs, we do have the opportunity to reduce the XenApp hardware footprint somewhat.  But in my opinion, server virtualization is trying to solve a small problem in the XenApp world, consolidation.  With proper hardware design, this can be mitigated. I have seen, based on my experience as a consultant and an administrator, the bigger challenge is management, availability and flexibility. 

When I was an admin, we used to have a scripted build for our MetaFrame 1.0, 1.8 and XP servers.  The scripts were very elegant and worked like a champ (I can say this because I wrote them), but they were a pain to maintain.  Plus you had to take into account hardware changes, application modifications, etc.  I've seen people go to cloning-like solutions, but you still have hardware configuration challenges, which I've seen some people end up with 10, 20 or even 50 different images.  When it was time to patch those systems, the good times rolled (sarcasm). Server virtualization cloning has the same challenges, although hardware changes are mitigated by the hypervisor.  Cloning in the virtualization world allows one to quickly get a system up and running, but does little for maintaining the images.  Just in my own personal lab, I've got roughly 20 virtual images.  And it seems like every time I turn on one virtual machine, there are new updates!!!  We have all heard of DLL-hell, well new we have Patch-hell. 

And we all love the server virtualization solution, even I do, which is why I'm writing this blog entry instead of preparing for my Synergy sessions or watching a good episode of The Simpsons without my boss catching me (Hope he doesn't read these). Everyone is talking about it as the next big thing, but we will continue to have tons of servers that are not virtualzed.  Does that mean 2 solutions, 2 sets of images, 2 sets of tools based on your environment?  When I think of that, I'm thankful I'm not an administrator. But this is where the story gets really interesting:

Provisioning Server integrated with XenServer, what a great concept.  One image for multiple servers.  And what's more, that one image can span virtual and physical servers.    When I need to make an update to the app or OS, I update one image and reboot the servers.  Time to rebuild the farm equals the time it takes to reboot the farm.  I don't care if the server is physical or virtual, they are all the same to me.  As I use this integrated solution more and more, I am impressed with the ease of maintenance.

But let's get back to the original question... If you now ask me if you should use server virtualization integrated with your XenApp environments, my answer has not changed... I will still say It Depends.  But what I say next is to look at the bigger picture.  Why do you want to virtualize? What are you trying to solve? What is wrong with your XenApp environment that you are looking at server virtualization?  And I bet the more we look into it, we will end up with a challenge revolving around management, availability or flexibility.  So I dare you to ask me, but be ready for a longer conversation, which will include some relevant Simpsons quips as well.

If you are interested on architectures, guidelines, implementation guides, then I encourage you to take a look at a set of materialsI've developed focused on the integration of XenServer and XenApp.  If you think I'm totally on, let me know so I can show my boss how awesome I am, but I'm also game for a good discussion with differing viewpoints.