Welcome to the third installment of the Dynamic Delivery Center.  This time I will be showing you the Proof of Concept (PoC) we built to validate the DDC is possible.  If you haven't done so already, I encourage you to review the first two blogs so you understand our business and requirements.

• Part I - The Need
• Part II - The Analysis

Now, the PoC.  First, let me show you the architecture diagram we've used.  (Visio Stencils for this diagram are located here).

As you can see, it is fairly straightforward. I'm the type of person who prefers things simple. The whole purpose of the PoC is to see if we can use a web front end to dynamically deliver any number of "Test" environments to the users.  Now, as many of you reading this are techies, let's get to the good stuff...

• External Access: Every user will be remote. Even if you are sitting in the office next to the lab, you are considered a remote user (Ever hear of de-perimiterization?).   All users will connect through an HA Pair of NetScaler 7000s with the SSL-VPN functionality enabled in full VPN mode.  We are doing more than ICA so we need a full VPN connection.
• Web Front End: Users will be able to connect to the Web Frontend when they connect with Access Gateway Enterprise .  The Web Frontend will allow the user to request any number of systems from the lab. 
  
  
• XenServer Resource Pool: Currently, the XenServer Resource Pool contains a set of templates that users can request from the environment. Those templates are reflected in the Web Front End, allowing the user to customize their environment.
• XenServer Template Library: For the PoC, the library only includes Windows 2003 R2 servers, XenApp 4.5 servers, Windows Vista SP1 workstations and Windows XP SP2 workstations.  New virtual machines are created based on the templates, which should only take a few seconds.  The library will grow as new requests come in and new systems are required. The longer the DDC is running, the more complete the library will become.
• NetScaler: The NetScaler devices are setup to allow for either a one-arm or two-arm deployment (hence the reason for the two separate VLANs).  If the user only requires a one-arm setup, they just ignore the second connection.
• WANScaler: The WANScaler devices are setup to allow the user to test any number of backend optimizations across any simulated WAN connection with the Apposite WAN Emulator.  The backend contains another XenServer Resource Pool allowing the user to test WAN optimization against any number of resources including file servers, web servers, media servers and XenApp servers, just to name a few.
  

We have the architecture, but how does it work?  How does the Web Frontend do it all?  In the PoC, we chose not to look into Workflow Studio (Sorry WFS Team) as we want to wait for the next beta release. But the lessons learned in the PoC will help us properly develop our workflows in the design phase.  In the PoC, we used the SDKs extensively to do the following:

• Virtual Machines
  1. A user selects one or more templates on the Web Frontend and selects "Provision Servers".
  2. The Web Frontend code will search for a virtual machine resource in the database that has not been marked as in use. Once an open virtual machine is found, the database will be updated and marked as in use by the user for a period of 3 days.
  3. The Web Frontend will establish a session with the XenServer host using root credentials.  The template the user selected will be cloned.  Once the clones are created, they will be sent a start command.
  4. Once the virtual machines are running, the IP address will be obtained. This information is used to generate an automated email to the requester using the SMTP service running on the Web Frontend server.
  5. The user will use the IP address to make a Remote Desktop connection to the console of the server, which is waiting for the user to enter a name for the virtual machine as part of the SID generation process.

• NetScaler
  1. User selects "Provision NetScaler" from the Web Frontend
  2. The Web Frontend checks the database for an available NetScaler. Once one has been found, it is marked as in use by the particular user for a period of 3 days. In the event that all NetScalers are assigned, the user will receive an automated email.
  3. The Web Frontend will establish a session using XML API calls with the NetScaler using the "nsroot" credentials. The reset process involves using the XML API calls to get into the NetScaler shell to remove the ns.config file.  Simply deleting the NS.Conf will completely reset the NS config. That would be bad because that includes the IP Address.  We don't want to go into the lab and connect a serial port and configure the device.  To solve this challenge, we  copy a base ns.config (which includes the NS IP configuration) over the current one.  We also have the code go through and remove any extra files that the previous user might have created (certificates, configuration files, etc).
  4. The Web Frontend will send code that will clear the NetScaler configuration, while keeping the IP address constant, so it is accessible from the network.
  5. The user will receive an automated email from the Web Frontend using the SMTP service.  The email informs the user on the connection information for the NetScaler.

• WANScaler
  1. User selects "Provision WANScaler" from the Web Frontend
  2. The Web Frontend establishes an HTTP session with the WANScaler web console using the "admin" credentials.
  3. The Web Frontend sends a request to reset the WANScaler config back to factory defaults, while still preserving the IP address. Once the WANScaler has been set back to factory defaults, the WANScaler will be rebooted.
  4. The user will receive an automated email from the Web Frontend using the SMTP service.  The email informs the user on the connection information for the NetScaler.
     

Lessons Learned:

• The biggest thing is that it is possible!!  The tricky part of the project was resetting the NetScaler and WANScaler back to factory defaults without losing the IP address. 
• A more complete set of XenServer templates is required to anticipate any number of requests from the field
  

Next Steps:

• Create a more detailed design that identifies the templates required for the initial release
• Create a detailed set of workflows that are required to see how Workflow Studio can be leveraged to make this environment easier to build and maintain.
• Create a way to hide Simpsons "Surprises" within the lab like logging into the lab and receive a warm welcome from Homer saying "D'oh!"
  

Hope you enjoyed this one.

Daniel

Homer Quote for the Blog "Look, the thing about my family is there's five of us. Marge, Bart, Girl Bart, the one who doesn't talk, and the fat guy. How I loathe him."

It has been a few weeks since I first wrote about drinking the Citrix Kool-Aid and trying for ourselves to turn our lab into a dynamic deliver center.  The first part of this project is to identify what is the purpose of our lab.  After looking at things deeper, we are responsible for mainly 4 things:

Technical Readiness Infrastructure: 

The Tech Readiness group is responsible for creating and delivering technical training to our customer facing people. This includes Support, Consulting, SEs, etc.  This group develops hands-on training that walks the student through setting up, configuring and troubleshooting the product.

As you can imagine, during a new product release, we have classes stacked up one after another all around the world. These classes use prebuilt environments in a remote lab in Ft. Lauderdale.  So if a class is occurring in Singapore, London, Sydney, Paris or anywhere else , the students will connect to Ft. Lauderdale to work on the pre-configured lab environment. Because the focus of the classes is to train on the new features, we don't expect our students to run through installations.  This means the environment must be prebuilt ready for configuration.

As you can expect, this is a challenge, which brings us to our first few requirements:

• Requirement 1: Tech Readiness: Must be able to work with the latest products, whether they be hardware or software, remotely.
• Requirement 2: Tech Readiness: Must be able to refresh environment quickly to a base state with items installed, but not configured
• Requirement 3: Tech Readiness: All previous classes configurations must be removed and devices and servers must be put back to base state (including passwords, accounts, optimizations, etc)
• Requirement 4: Tech Readiness: Environment must be able to allow students to work with all features and products including WANScaler optimizations, NetScaler load balancing, XenApp Progressive Display, Smart Auditor, etc.
  

Worldwide Consulting Solutions

The Solution Center and Integrated Solutions team is responsible for developing best practices and recommendations for integrating Citrix products with other Citrix products and 3rd party products.  For example, these two teams have developed items discussing XenServer and XenApp integration and on how to integrate WANScaler and NetScaler with Microsoft SharePoint.  From project to project, the architecture could look quite different, but there is one common aspect to all projects... There is at least one Citrix solution involved. 

The types of projects can vary wildly from validating an application runs on XenApp, defining deployment best practices for a particular web application with NetScaler to performing scalability testing on the latest version of XenDesktop. All of these items come together to bring us a few more requirements:

• Requirement 5: Worldwide Consulting Solutions - Need to be able to bring up a set of Citrix solutions without requiring installation
• Requirement 6: Worldwide Consulting Solutions - Need to have the different project pods separated from other pods as one test might influence the results of another test
  

Field Teams

Working with our customer, many of our field Citrites find themselves in need of a reference system to be able to look up a setting, perform a quick test based on a customer question, or be able to demo a new feature that is easier to show than to explain.  These types of requests are short lived, but require a fast response. Because of the huge number of potential questions a customer could ask, it is impossible to anticipate every conceivable environment needed or when the requests could occur.  This type of situation brings about the following requirements:

• Requirement 7: Field Teams: Need to get access to a base system (regardless of product) in a short amount of time that can be modified. 
• Requirement 8: Field Teams: Each reference system must be isolated as customers will be seeing the systems in a lab that also contains systems of new, unreleased products
• Requirement 9: Field Teams: Need to be able to extend check-out time if work extends beyond original request date
  

Administration

I haven't hit all of the groups that use the lab because this blog would be longer than the movie script to the Simpsons Movie (which I highly recommend by the way), but most of their requirements are contained within the first three groups.  Before I close out, there are still a final set of requirements focused on the administration of the lab. We must make it easy and automated:

• Requirement 10: Administration: Systems should be in a low-power state unless they are being used.
• Requirement 11: Administration: The systems, when allocated, should be powered on automatically.
• Requirement 12: Administration: Systems should be automatically built, cleaned, decommissioned and assigned.
• Requirement 13: Administration: Hardware assignment notifications should occur through email with all pertinent connection information once the systems are online.
  

I know this was quite a long blog, but this is a big project and I didn't want to gloss over anything. 

Up next, a Proof of Concept.

Daniel

(Homer Simpson Quote of the Blog: "Oh, so they have internet on computers now!")

The first question that most people ask me after seeing Workflow Studio in action is:

"Wow! That's cool, but what can I do with that?

I would like to use this post to try to explain why I think you should be doing automation in your data center today and then look at some different processes that are good candidates for automation with Workflow Studio.

Why should you automate?
The IT data center of today often includes products from many different vendors and consists of both hardware devices and software. Often, the SDKs that are available for all of these devices and software are quite inconsistent and some of your processes may have been developed by a consultant who is no longer around. Stop and ask yourself this question:

Are all of my processes consistently repeatable by anyone on the team?

If you can't answer 'yes' to that question then you should be looking into IT Process Automation. And if you want an easy way to automate tasks across multiple products/components without the need for complex scripting, then take a look at Workflow Studio.

Here are some broad categories for types of automation you might want to have in your data center:

1. On-Demand Automation - Being able to launch a process that combines several tasks, possibly with some simple user input, in response to something that has happened in your environment (e.g. A call or email comes in that a new user needs to be setup, a new application installed, or permissions modified for a user.)
2. Schedule-Based Automation - Schedule a task to be performed at a set time each day (e.g. Restart your servers every night at midnight.)
3. Event-Driven Automation - Automate a set of tasks based on system events (e.g. When load is increasing bring more servers online and when load decreases take servers offline)

If you download the Tech Preview of Workflow Studio (free download here) you can look at some of the sample templates we include with the product and get a look at some On-Demand Automation out of the box. The Tech Preview doesn't really provide any help with scheduling or event management, but this type of functionality is being added to the product.

Next, let's look at what I think are the 3 key categories of use cases for Workflow Studio and some examples that you could put in action today:

1. Server Resource Management = Dynamic Delivery Center - Transform your data center into a delivery center by creating workflows that tie together your server provisioning process (both physical and virtual) with your web, desktop, and application delivery processes.
2. Power Management = Green Data Center - Being able to power on/off both physical and virtual servers automatically provides the ability to better manage your resources and, ultimately, save $$ by having your power-hungry servers off when they are not needed.
3. User Management = Automated User Provisioning - Every organization has a process for provisioning and de-provisioning user accounts, but often these processes involve multiple manual steps. Automating the process end-to-end ensures that best practices are followed in exactly the same way each time.

I hope this encourages everyone to think about what kinds of tasks you could automate in your environment. Let me know in the comments what kinds of things you would like to automate or you can find my email address in my profile if you want to email me directly.

Within Citrix,  I'm on the team called Worldwide Consulting Solutions whose overall goal is to provide our customers with the tools and resources to successfully implement Citrix solutions.  Talk about an easy task .  The interesting thing about this goal is our customers. Our customers are potentially any Citrite, Citrix partner or Citrix customer.  Within this team, we have something called the Solution Center.  It is essentially a lab, but with some serious requirements.  The Solution Center lab must do the following:

1. Be capable of supporting integration tests and scalability analysis with Citrix products and 3rd party solutions
2. Be capable of supporting our readiness activities where many of Citrix's field personnel are trained and tested on our latest products and solutions
3. Be capable of providing temporarily systems for solution testing for our consultants and SEs, where they can test a feature, or look into a potential solution for our customers.

This is quite a task.  The challenge with the lab is provisioning these systems out on an as needed basis. From week-to-week the environment changes based on the needs of projects, training and testing initiatives.  Let me explain how this process works now:

1. A user would make a request for hardware based on a SharePoint web page. This page contains numerous fields so the Solution Center team can figure out what hardware to provide for the project. Items include dates, server type, OS, quantity, applications, HW requirements and an overall description.  Once a user submits this request, it is up to the Solution Center team to work out the logistics.
2. The Solution Center team takes the requests and tries to identify available hardware based on a master scheduling SharePoint page. Once hardware is identified, the new systems are built based on the user's specifications with SysPrep and Ghosting techniques. Currently, the group is building a library of XenServer virtual images to simplify this process.  
3. Once the system is built, the user is notified with IP Addresses and connection procedures.  The entire time from request to delivery can be up to 2 business days.   

This group as they have a very difficult job and they do it great, but one would think there would be a better way.  We have started to build XenServer virtual images libraries to more quickly build the requested environment, but there still is a manual process involved that slows the entire system provisioning down. Is it possible to create a self-service system that would not require manual work from the Solution Center staff?  What if we could cut down the time to build these environments from days to minutes while providing 24x7 service?  A self-service system like this would allow the team to weasel out of this time consuming work.  "Weaseling out of things is important to learn. It's what separates us from the animals! Except the weasel." (Homer Simpson).  

During Synergy we heard all about Citrix's vision for a Dynamic Delivery Center on the big screen (It looked like a really big TV).  And again, Homer provides us with a good recommendation, "When will I learn? The answer to life's problems aren't at the bottom of a bottle, they're on TV!"  So, we started drinking the Kool-Aid and we are going to see if the Kool-Aid tastes good.  Can we turn the Solution Center Lab into a Dynamic Delivery Center?  We will all find out together as we have just started to work on this project.

I will continue to blog about our challenges and successes throughout the project, which will contain analysis, PoC, design and implementation.  You will see

• Our requirements are (and they are challenging)
• Our overall architecture design
• How we implemented it
• The final outcome
  

This is going to be a fun project allowing us to get into the details of all of the Citrix solution stack.  So, stay tuned for Part 2: Analysis

Daniel

(Homer Simpson Quote: I want to share something with you: The three little sentences that will get you through life. Number 1: Cover for me. Number 2: Oh, good idea, Boss! Number 3: It was like that when I got here.)