Common Application Streaming question:

How much data lands in the execution cache compared to what is captured at profiling?

My quick answer to this question is 25 to 30% of the captured content will land on the execution machine at runtime.  A more correct answer is ... "it depends".   People like numbers and they need numbers, so we give them numbers.  Statistics though can be misleading.  Are they good numbers? Sure, absolutely! Read more to understand some of the complications.

To get it started, this is the wrong question.

What people really want to know is how much disk space is used to deliver this application via Application Streaming as compared to delivering it via local install?  Answer: 35.6%.  Again, read on for more details because the real cost is a bit lower than this number and gets back to that 30% space, or the real cost is quite a bit higher than this number depending on whether you're going to go offline or not.

Application selection

The selection of the application here greatly changes the outcome of the question and deliverying for OFFLINE really changes the calculation.  First, consider online execution.  If the application has only a few files, all of which are accessed at runtime, then the isolation system will populate 100% of what was profiled.  By contrast, if the application has thousands of files, most of which it never accesses, then the answer approaches zero. 

The key point is that files are brought in when they are ACCESSED.  If they are not accessed, the isolation system LIES to the application to tell it that the stuff is present when it really isn't.  When it isn't, it doesn't use any disk space and it doesn't occupy any network time being copied in from central strore. 

It is amazing to watch how much stuff applications install that they NEVER, EVER reference.

NUMBERS

The measurements that follow are taken from my primary notebook machine, running non-release Windows 7 RC1, with Citrix Application Streaming, non-released development "tips" build from about 2 days ago that flunked build test, but which seems to be working pretty good for me.  I'm connected to the "showcase" farm inside Citrix, which is where we put the stuff that isn't released yet to "use what you sell".  Hundreds of people using this by the way.  I'm running a non-release and pretty current PNAgent which I haven't changed since before Synergy conference.  This is about as "stable" an environment as I get.  Other than these items, my environment is pretty typical and should be fine for statistics.
Why Microsoft Office 2007?  Answer: It's a really big application and everyone understands it.  Is it an example of a typical application?  I'm not sure, but it is sure an excellent testcase for proving out an isolation system.

Framing the problem
There's more to it than what gets populated into the execution cache; offline/deploy and online are also part of the equation.  If the execution is "stream to server" then there's no need to deploy a copy of the execution image on the execution machine.  Disk space usage will be optimal, the low number.

For "stream to client", offline will be common, so disk usage will be higher as you have to store a COPY of everything that is on the Application Hub that supports your execution.  This storage though is a compressed version of what was captured at profiling, so it is reduced in size and you only get a copy the execution target that is right for your machine, so the size used for deploy will often be less than the disk space used on the server that stores the execution content.

Application Hub:
Our showcase farm is consuming 27.2 GB of disk space to store MS Office 2007, on the server.  Much of this is wasted space because no house cleaning has occured.  There are for example 5 version of the execution target stored which support Vista/Windows 7 and there only needs to be 1.  My admin could save 80% of this disk space and nobody would notice.  By spending the disk though, he retains the ability to perform rollback. 

LOCAL MACHINE
Deploy: 1,161,958,306 bytes. 

Notice this is a WHOLE bunch smaller than on the App Hub.  I'm pretty sure our admins flush out the old stuff to help keep this number small, but even if they don't, I format and reinstall machines often enough that this doesn't come up.  The deploy location though is COMPRESSED.  We install everything and a bunch more when profiling for the showcase farm.  The expanded number for the size of the packaged MS Office is 2,125,243,930 bytes.

RadeCache populated execution space = 321,401,139 bytes.

Drum roll for the math

Online: 321,401,139 /  1,161,958,306 = .2766 = 28% OR

Online:  321,401,139 / 2,125,243,930 = 15% (That's a nice statistic!)

Offline: (321,401,139 + 1,161,958,306) / 1,161,958,306 = 128%
Did you just say that offline streamed uses MORE than locally installed?  Yes.  We made a conscious decision for this actually.  The extra disk usage allows online and offline streaming to be largely the "same" from the view of the isolation engine.  Technically, this is called "deployed" or "not deployed".

Compare to locally installed.

I had to struggle, but I found a Citrix PM who is still has MS Office 2007 locally installed where I could get some numbers.  His machine has 904MB, 904,155,136 bytes consumed.  Notice that this is smaller than the 1.1GB that is in the CAB file that our showcase farm has.  This is expected.  Our build has some other stuff added beyond the base MS Office and also was profiled with the of MS Office install selections, selecting "everything". Among the nuggets that are "extra" in the profile are a full copy of Mozilla Firefox.  I'm not sure what it's doing in there, but adding these things does make the profile grow.

Using these "common user" numbers for locally installed and comparing to the showcase MS Office 2007 profile, we can calculate a second statistics.

Online vs common user: 321,401,139 / 904,155,136 = 35.6%

Offline vs common user: (321,401,139 + 1,161,958,306) /  904,155,136 = 164%

Fascinating numbers.  What good do they do me?  I'm not totally sure.  If you operate on the theory that streamed delivery will use about the same disk space as installed delivery, you're in the right ballpark.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

User32.dll is a magic item.  As a programmer, if you want to take the machine over, then user32.dll is your best friend.  User32 is a system DLL that gets loaded into all programs, system and user, that do anything with the GUI.  User32 has a nice side benefit that it also loads other DLLs, by name.  The list of DLLs to load is stored in the registry in a string item, AppInit_Dlls.  Yes, this space is only writable by privileged proceses, but if you can get yourself on the AppInit_Dlls list, you're golden!  This is so handy that it is a common method that viruses use to attach themselves to all the processes on a system and ... is how application isolation systems like Citrix Application Streaming do their work.

Notice above, I said ALMOST all processes link to user32.dll.

There are many processes which do not load user32.dll and if they don't, then things that load as part of AppInit_Dlls will not get loaded.  If you're in the application isolation business, this is not good because it means that you can't isolate that application. 

A common question - Does Citrix Application Streaming depend on the application loading user32.dll in order for the isolation system to hook the app's execution?

Answer: No. 

The more elaborate answer is that AIE on Presentation Server 4.0 did depend on the application loading of user32.dll, but Application Streaming does not have this limitation. 

Propeller talk on user32.dll. 

If you want to know more about hooking processes and user32.dll, here are some good and entertaining references. 

• User32.dll is really important
• AVG Virus scanner incorrectly recommends erasing user32.dll (Awesome!)
• AppInit_DLLs should be renamed Deadlock_Or_Crash_Randomly_DLLs
• How virus loads
• Microsoft Malware Protection Center blog (good reading)

Probably best for a separate post, but the second item here is really interesting.  If you think you have a virus and do a google search for "user32.dll virus" you'll get 574,000 hits!  Sometimes, it seems like the "fix" for virus is worse than the virus itself.  If you delete user32.dll, you're up a creek with no paddle!  You can hope that the Windows XP system file protection will put it back, but it's still a scary proposition. 

Consider that if you are evil, and you're inserted into system code, then the obvious next step is to hide from anti-virus.  This must be an interesting battle. 

Enjoy,

Joe Nord

This is"part 2" in a propeller-head series about the internal workings of Citrix Application Streaming.  This post covers how stuff in the registry can be "erased" during profiling without really effecting the profiling machine.  The same concepts apply at runtime where the application can erase things from the machine and have this effect only be apparent for the current user rather than the whole machine.

Part 1 covered the same topic for files.

Consider: Machine has registry space at HKLM\Software\DeleteMe (yes, notice the easy to recognize naming).  In this example, there are also a few registry items located in that space, abtly named, item 1 and item 2.

The installation program observes that this space exists and as part of its installation activity, erases it.  The job of the isolation system is to let the installer BELIEVE it erased the space, while not really erasing it, so that the streaming client can present this space as "not there", when it really is.

For the installer, I have used CMD.exe and here's the activity, captured as text.
c:\>reg query hklm\software\deleteme

HKEY_LOCAL_MACHINE\software\deleteme
    Item 1    REG_DWORD    0x1
    Item 2    REG_DWORD    0x2

c:\>reg delete hklm\software\deleteme
Permanently delete the registry key HKEY_LOCAL_MACHINE\software\deleteme (Yes/No
)? y
The operation completed successfully.

c:\>reg query hklm\software\deleteme
ERROR: The system was unable to find the specified registry key or value.

From outside of isolation, go look at the "real" registry.  You'll see:
What happened? 

Answer: The true registry was not "hurt" by the profiling activity.  The registry space that was deleted wasn't really deleted.  The captured profile though believes it was deleted and when moved to the execution machine, the Streaming Client will do similar isolation stuff to make this space appear "gone" even if that space really exists.  The "how" follows.

Package the app up and peek into the captured registry contents and you'll see the magic that makes this happen.  Recall from the file discussion, NTFS Alternate Data Streams are attached to the deleted files to describe them as erased.  The isolation system does similar activity for deleted registry.

Notice that the deleted registry space "exists" as part of the captured registry space.  It exists, but there is also this extra stuff that the isolation system references to conclude that this regsitry space is "erased" and should be masked from vision to the application.  In the case of the registry as well as for files, the erased markers are attached to the erased stuff so that the isolation system doesn't have to look far away to index things that are erased.  When a registry key is opened, the isolation system looks to the side to see if the marker is present and if it is, it "hides" this registry key from the view of the application. 

Magic?

I called this magic above.  In reality, no magic.  If the application happened to use and depend on a registry item named CitrixAIEDeletedStatus, this whole thing would not work.  Fortunately, this is a rather unique name and this makes it okay to add to registry space, where the application won't be impacted.  As a last step, the registry items that are the markers "don't exist" from the perspective of the isolated application, so this prevents confusing an application by giving it extra registry items that it did not define.

What is the other Citrix thing

Good question.  The CitrixAIEPlaceHolder item exists so that the isolation system can transport registry KEYS that have no contents.  Compare in concept to XCOPY /S and XCOPY /S /E.  Some application create registry keys (similar to file system directories) and the mere presence of the key has meaning to the application even if that key has no contents.  Since the isolation system stores and later repopulates the registry space in a manner similar to .reg files save and restore, keys with no contents get lost.   By providing an item, any item, the empty registry key is preserved from profiling system to execution system and the application "works" because it sees what it expects to see.

Disclaimers

Lots of bit head stuff here on how the isolation system works.  Don't become dependent on it.  We change these things from release to release and it is likely that the method of representing deleted registry or empty registry can and will change some day in the future.

But - if you're looking at your machine and are asking yourself: What is all the Citrix stuff?  Now you have the answer.

Joe Nord

Deleted files are an interesting topic in isolation systems.  How do you represent a file as "gone", when it is still present?  During profiling of an installation program, the isolation system has to isolate additions to the file system; this part is easy and pretty easy to understand using the "layers of glass" metaphor.  You look down from above and the highest version of the file is the version that you see.  What is more interesting to consider is that the isolation system also has to represent DELETED files; and deleted registry content.  These require placing a marker at higher levels that say to mask vision to the content that really does exist.  Deleting files is much more interesting than adding files!

Consider a view from above, looking down through the isolation system goggles:
 
I used the IIC version of this graphic because it was handy, but consider the above with only 2 layers as this is the environment during profiling.  The PHYSICAL or "REAL" disk/registry is off limits for writes.  When the installer during profiling, or when the program at runtime writes to the isolate space, the changes are reflected as writes to the higher layers of isolation (the first "writable" space, which is always on the top).  This works great for adding content because the presence of the content at the higher layer "masks" vision to the layers below.

For deleting things, this gets interesting.  Consider a program at installation which erases content from the physical machine.  Looking down from above, the installer needs to BELIEVE that the erase it commanded, really happened.  Here, the isolation system has to represent "deleted" files and registry content without really deleting the stuff from the true disk.  Failure to do this would require a VM snapshot thing and resetting of the environment before/after profiling and we didn't want that.  Instead, you can profile over and over all day long without hurting the true physical image - at least for the spaces that are isolated, which is pretty much everything during profiling.

A deleted file is represented as a real file, at a higher layer of isolation.  The real file is changed to have 0 size, but still has the same filename as the lower layer file.  The filesize reduction is just to conserve disk space, but in concept the deleted file is the same file as the lower layer; promotion from physical to isolated occurs commonly for files "changed".  To make the file "deleted", the isolation system tags the file with a NTFS Alternate Data Stream, "CITRIX_DELETED_FILE_MARKER".   If the file exists on the layer of glass, but has zero size and the required tag, then the file "disappears" from the perspective of isolated processes.  The file still exists and is even visible outside of isolation; though it still has no contents.

References:

Forked file systems (Mac)

Scary reference - ADS are evil  IMO, They are not evil, they just aren't widely understood.

Microsoft KB describing how to use ADS

Sysinternals tool for displaying Streams

The fact that ADS are rarely used by "normal" Windows applications makes them a PERFECT candidate for usage in the isolation system.  The isolation system can attach a stream to the file and feel pretty confident that this won't hurt the applications ability to manipulate that file, yet the information associated with that file hangs out and even follows the file from place to place when it is copied.  Note: this isn't exactly true - keep reading.

Profiling

During profiling, the isolation system sets the marker on files that the installation program erases.  In concept, this means that the marker is present on the execution machine at runtime because the captured layer of glass is transported to the execution machine.  In reality, this is true only in concept.  In reality, we use CAB files to hold the layer of glass and CAB files do not maintain streams.  Hum.  Problem.  What did we do?  Answer: When the application installer completes processing, the profiler goes through and "deletes the deleteds"; removing the files with zero size and the stream attached.  This means that the deleted file will not be present in the captured image, but it also means that the marker will not be present at runtime.  Problem?

First, why is this a risk?  If the file existed on the profiling machine (at true disk location), and if the installer erased it; then on the execution machine, if the file exists on the true disk, then at application runtime, the erased file will re-appear.  As the theory goes, the application at runtime will get unhappy.
Is that a problem?

At the time, I wore the FSFD dev hat and took note that darn near zero application installers actually erase things from isolated spaces during profiling.  Those that do, likely will also tollerate that the file comes back.  We also took note that even if the file did re-appear, we could always erase it again using a pre-launch script.  Bottom line: Application Streaming has shipped this way from the beginning and I have never seen a failure attributed to this behavior.

But wait, there are more problems

At runtime, the application may decide to erase things.  In general, the only spaces isolated are the \windows directory and below and the \program files spaces or more precisely, these are the locations where an ill-behaved program may want to erase things; things that could effect the execution of the program.  

It could also be that the application erases some of its configuration data from the directory to which it was installed (a bad application). In either of these cases, a deleted file marker will be generated and will land in the per-user layer of isolation.  This will mask vision to the file on the true disk.  Since this is part of the per-user space, all should be happy because the per-user space will follow the user from machine to machine as part of roaming profiles or even better, Citrix User Profile Manager!  The Win32 CopyFile API DOES maintain Streams!

Problem: Roaming profiles do not maintain ADS Streams, so the deleted markers will disappear on movement from machine to machine.  4 years in the field, has this been a problem?  Nope.  

I have asked the Citrix UPM team to maintain Streams on copy of the user profile content to/from central store which is a bit of a curiosity because it is different behavior than Roaming Profiles.   I'm not sure if it occurs yet and not even sure if they will actually implement the change.  Why?  Fundamentally, the ADS streams haven't been missed yet, they will likely be missed less in the future; except for all them MAC users.  Stick with me.

What file systems support ADS Streams?

Excellent question!  The answer is "NTFS".  The Mac file system REQUIRES streams, but we rarely get the mac file system as the foundation of a Windows machine.  FAT32 definitely doesn't support streams and I'm not sure about CDFS.

Since deleted file markers are needed for isolation processing, the Application Streaming file system code restricts it's filtering to only NTFS disk volumes.  In some ways, this is a plus.  You don't want to isolate USB drives and these tend to be FAT32, so they don't get isolated.  But, the FSFD knows the USB drive is "removable" and it assumes all removable media is for "data", so it doesn't mess with it.  In this way, the NTFS restriction actually gets in the way a bit.  Note: you can convince the FSFD to look at the USB drive as I outlined here.

What about more elaborate file systems?

Some customers have enterprise storage which is network based, but appears to the execution machine as local.  These disk volumes COULD be NTFS, but often they have a custom file system.  Do these support NTFS style ADS streams?  Answer: It depends.  Let's say they don't.  How to solve...  Answer: Convert the representation of "deleted files" from ADS Stream to something a bit more common.  Has this happened? Nope.  Is it really necessary?  I'm not sure.  Experience of the last few years says it is rare enough that the failure case probably won't come up.  With this wide post though, something tells me someone is going to say: "I've seen the failure case".

In a follow up post, I'll describe how the registry "deleted" markers are implemented.

Enjoy.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

How do you add FTAs to profiled applications that the installer forgot to install?  Answer: There are a few steps.

This has been on my list to Blog about for a while, just haven't gotten around to writing about it, mostly because  I hadn't had a chance to verify the solution would actually work.  Conveniently, Karl Muller of I-Access took care of the trial for me, and it worked, so here's the solution.  Thanks Karl.

Review

Backup a couple steps... "FTA what!"

When an application installs, it writes entries to the machine to establish associations between file extensions (multiple) and programs (single).  Each file type definition also includes a reference to the ICON that goes with the extension. This is what Windows Explorer uses to show you icons that looks like MS Word for file's that end in .docx.  

The Streaming Profiler runs the installation program under isolation and blocks the installers attempt to set FTAs on the true machine.  As part of post processing, the profiler examines all the FTA entries that the installation program TRIED to create and for each of these, creates corresponding FTA entries in the streaming profile XML data - you can see them in the .profile file, if copy to .xml and view with web browser.  Easier, just look at the properties of applications captured during profiling and ask to see the FTAs.  Yes, that's much easier.

The Access Management Console (XenApp publishing console) reads the profile information and gets a list of all the captured FTAs.  It makes these AVAILABLE for association with applications that you publish.  The default is that no FTA will be associated with the published application, so as an admin, you have to go to direct action to establish the FTA by clicking on boxes.

BUT - There's a catch, you can only establish FTA reference for the FTAs that were captured by the streaming profiler.

Say you know that .DOC9 (.DOC and the number nine), is SUPPOSED to be associated with MS Word, but you note that the installation program seems to have left that one out, or that the streaming profiler seems to have missed this one.  Let's focus on the first one.  You have an extra FTA that is SUPPOSED to be associated that seems to be missing from the list.  How do you add an extra FTA?

Option 1: Streaming Profiler, application properties, FTAs, Add

Option 2: In the Access Management Console, application properties, FTAs, Add 

Both options above would be GREAT, if only they worked.  They are both READ-ONLY.  You can see the FTAs, but you can't add a new one.  
The solution...

FTA must be captured during profiling

When profiling, tell the profiler that you want to launch an installer, it will prompt you for the path and name of the installation program, tell it "CMD".  When launch the installer, you will get a command prompt running that is running inside of isolation.  Two commands are needed at the command prompt to establish an FTA.

• assoc .doc9=Word.Document.12
• ftype Word.Document.12

If the ftype is already in place, you're done.  If not,

• ftype Word.Document.12="C:\Program Files\Microsoft Office\Office12\WINWORD.exe" /n /dde
• exit

Now, let the streaming profiler do it's thing to see what the "installer" did and it will discover a new FTA for the .doc9.  Save the profile and return to the Access Management Console.

Publishing

In the AMC, publish a new application, point it at the profile and it will see the FTA that you added.  EASY!

There's a catch

There is always a catch.  The catch here is that the FTA will exist only for new application that you publish.  Applications that you have already published have already had the FTA data sucked out of the streaming profile and placed into the XenApp publishing infrastructure.  The AMC will not "pick up" or automatically update the applications to reflect changes to the profile, so the FTAs won't exist.  If you need that FTA added to an application that is already published, it is necessary to delete the application and republish it. 

The other solution

Not recommended, but you can accomplish the same thing by hand editing the XML data in the streaming profile and saving.  This isn't the RECOMMENDED answer, but just between all of us, it will work.

Joe Nord 

Citrix TV folks have made me look really good!  Here's a link to a 9 minute video where I go through the "layers of glass" in application isolation and the fundamentals of how Application Streaming works.  The video outlines the XenApp 5.0 new capabilities for HTTP based streaming and provides guidance on wide area network architecture including Branch Repeater usage to eliminate the Application Hubs at each branch office.

I have previously blogged about the layers of glass, here, and with inter isolation communication added, here.

This video is better.

Joe Nord

Today I learned that Citrix posts its XenApp build debug symbols to a publically accessible symbols server.  Cool!  This is mostly to make life easy for the Citrix support folks, but what ever the reason, this means that the symbols are in a place that you can see them!

If you are familiar with WinDBG or Debugging Tools for Windows, then you know that the Microsoft Symbol server is a welcomed item in understanding the state of any machine you are inspecting.  Microsoft posts all the symbols for every component of every released version of Windows to msdl.microsoft.com/download/symbols. Set up the _NT_SYMBOL_PATH environment variable and the debugger does the rest to fetch Windows symbols from the internet.  Yeah, old news. 

I recently learned that Citrix also posts the XenApp symbols!  And to this I say, down right neighborly of us! 

Here's the link that says how to set it up, CTX118622.

What are symbols?

When building a program, the compiler does its thing and then the linker does its thing to actually create the executable content.  When putting the program together, one of the things that the linker does is produce a table of program symbols and their location in memory (symbols).  Programmers use these to figure out what's going on where they can then see nice "strings" rather than hex numbers. I say programmers, but it is really the debugger that does the work. 

There are two choices in symbols, the "FULL" and the "STRIPPED".  Simply put, the "full" ones are better, much like source code access.  The stripped ones though are still very helpful and it is these that are on the Microsoft download server ... and the Citrix download page. 

Neat thing of today, I've been writing code at Citrix for a long time and never known that the symbols were available to the outside.  NEATO.  This sure beats debugging stack traces with only hex locations of functions.

If you're not debugging systems on a regular basis, you may still find this useful for analyzing crash dumps.  One of the first things you do after a failure is windbg -z memory.dmp and then !analyze -v.  Give the debugger symbols and you can get a more precise location of a failure.  Good stuff.

Joe Nord

The "layers of cake" presents great opportunities for Citrix Delivery Center storage optimization.  Rather than store separate images for each machine, you can have ONE image for maintaining the operating system, ONE location which holds the application images, and ONE location for the User Profile.  This is valuable for XenApp, but it is especially important in XenDesktop where the number of machine images multiplies to be a really big number. 

This post describes the "problems" of getting applications into this layered world and discusses multiple solutions; including several that are not ideal.  It concludes with short term and long term recommendations.

Here's a slide that I presented at the Citrix Synergy conference in Las Vegas last week.
 
Even has candles!  Provisioning Services provides the bottom layer.  Application Streaming provides the middle layer and Citrix Profile Manager provides the per-user settings as the top layer of this multi-layer cake.  Getting this to move from "vision" to "reality" though involves system design and piecing this into your larger data center configuration is a complex operation.  The benefit though is increased flexibility and improved management of your systems as you are managing only images and assigning these as workloads to machines. 

Why does Application Streaming provide the middle layer
In this discussion, the "Streaming" aspects of Application STREAMING are really overrated.  The important fact is that the base operating system image focuses on being the operating system and the application layer focuses on providing applications.  Application Streaming is well suited to this need primarily because of the just in time application delivery - delivery based on which USERS will actually visit the machine.  To implement the separation in the layers of cake, the lowest layer cannot have knowledge of the application set and this means that the applications have to be RUNTIME provided to the execution machine.  The Citrix XenApp publishing does exactly this, both for Stream to Server (XenApp) and for Stream to Desktop (XenDesktop).  In both cases, the machine is "vanila".  A user logs in, application icons are placed onto the machine via the Plugin for Hosted Apps (aka PNAgent) and eventually the user clicks on something to cause the application to run.  With "streamed" applications, the applications are presented to the user "just in time" for their use.

Where it gets complicated
WOW, this is awesome.  I can have my cake and eat it too!  The details though are subject suitable for significant debate of best solution.  In particular the delivery of applications to the middle (application) layer. 

Provisioning Sevices will provide the base operating system image to the bottom layer and this layer will be absent most applications.  Sure, some foundation applications that are used by all users may be in this layer, but the advantage in the layers of cake is separating the applications from the operating system and this motivates us to want the apps delivered via Application Streaming regardless of benefits of isolation. 

Provisioning Services implements ONE to MANY by having a single disk image that it distributes to MANY "machines".  These machines may be real machines, or they may be virtual machines.  Either way, they get a MAC address and this defines them as a "machine".  Since this is a disk image and used by many, PVS must implement "write-back" cache so that machines using the shared image can update that image at runtime.  Important changes to the single image will be performed in "single" mode of PVS, but runtime changes to the disk must be maintained so that PVS can runtime support the temporary changes that the host operating system and applications will make the the "disk".   For each machine that PVS is running, it maintains a write-back cache.

The PVS write-back cache is "temporary".  When the machine shuts off, it is discarded.  This is what we "want", but it also "must be" because PVS is a block based implementation of a disk and it really has no idea what is stored in what block on the disk.  It doesn't even know what the "files" are.  With numerous machines "writing" conflicting changes to the disk at the same time, the only way out is to write cache these and then discard the cache when the machine shuts off, or in the XenDesktop case, when the user logs off.

Why is this a problem

If the Application Streaming application content (RadeCache) is runtime populated, then users will experience a first time application launch penalty each time the machine is booted.  In the XenDesktop case, this will be experienced on each logon.  The RadeCache must be populated and while it holds only a portion of what the application "installed", it is still enough data that the population of this space is a sore-point for first time use of applications.  In server case, or stream to a physical desktop, the first time launch is a one time cost.  The second time launch is much more important.  BUT, in the layers of cake the first time launch penalties become a every logon penalty.  The first experience on each logon is that the user behavio will be "slow" and then this "slow" performance will be repeated each time the user logs off and logs back in.  NOT GOOD.

It gets worse.  The PVS system write-back cache is about to get hit with LOTS of stuff.   As an example, consider MS Office 2007 on the notebook from whcih I'm writing this post.  The populated stuff to support MS office is about 400MB of disk space.  This will populate into the PVS write back cache and that's alot of disk space if you multiply it by 1000 machines.  More to consider, the Application streaming System does a network read to get the data from the Application Hub, it then writes it to the "local" disk, which in this case, isn't local.  The PVS write-back system then has to send that data across the network, back to the PVS server, who has to maintain the data on a per-machine basis; and potentially later, send that data BACK across the network to the execution machine to support execution.  I'm not sure of the numbers, but if you load it heavily enough, this will start to be a burden on almost any network.

How to solve

First a bit of background on the streaming system storage of application content.  The disk storage is the key here.  Everything that represents the application gets added to the streaming execution cache, which defaults to disk location, \program files\citrix\RadeCache.  There is a GUID beneath here to hold the execution content for each "profile" or execution "target" which represents the installed application that the streaming system is keeping track of.  Technically, a versioned GUID, GUID_version where version is used to allow "hot" update of applications.

You COULD populate the RadeCache into your base PVS disk, but this rather defeats the separation between Operating System and Application images.  This solution is "out".

The better solution says that the RadeCache should not be stored on the same disk volume that is the bottom layer.  Instead, the RadeCache should be moved to another disk volume and this disk volume should be common and shared across all the using systems (XenApp servers or XenDesktop clients).  Now, the maintenance of the operating system image can be ONE; and done via Provisioning Services and the maintenance of the applications can be ONE; and done via the Application Streaming profiler + the shared RadeCache execution space.

Getting more complicated
Much of what follows are multiple approaches to solving this problem.  The "correct" answer will depend on your network configuration and the tools that you have available to assist with representing a shared network resource as a local system disk volume.

PHYSICAL MACHINES

The easy solution.  In XenApp for example, even if using PVS as boot source, you likely still have a physical disk volume on the execution machine.  If yes, configure that space as write-back cache for PVS and the complications are improved.  The extra network hops will go away.   I'm not positive, but it is likely that this write back cache still resets on each boot, so there will still be first-time launch penalties.  With that, we should instead focus on machines that have no physical disk as the solution here will work for both configurations. 

We need a central source that can be mounted into all execution machines and treated as streaming runtime execution storage.  We want the "RadeCache" to be "fully populated" so that the streaming system NEVER needs to do a runtime cache fill.  In this way, the "streaming" aspects of Application Streaming are handed off to other technologies that will make the RadeCache appear as present even when they are remoted.  A side benefit of this is that App Streaming file based cache fills then get replaced with a page based system.

Application Hub

In a way, we already have a central space that holds the application content, it is the Application Hub.   Generally, all the profiles are stored beneath some "top" location and each profile contains the execution content to support that application on what ever the execution system may be.  One complication is that the "execution stuff" is stored inside a .CAB file.  While this is handy for compression and handy for portability to offline execution, it is not handy for access at runtime, pretending that the cab contents are actually present on the referencing machine.  It WOULD SURE BE HELPFUL if the CAB file were replaced with a subdirectory that has the same name, but is an uncabbed representation of the execution content.   In this way, the streaming system could redirect the local machine RadeCache\GUID_v to the Application Hub, profilename\GUID_v directory.  Do this and the problem is SOLVED.  Only problem, it doesn't work. 

Redirecting the execution content to network source

The "LINKD" command in Vista command prompts provides the foundation.    You need to use the /D option to create a directory junction.  Cool part here is that you can redirect the directory space to another volume and that volume CAN BE ON A NETWORK!  My experiments say that this works on Windows Vista, but that it does not work on Windows XP.  Given most XenDesktop configurations are Windows XP, this is a complication.  Note that the foundation reparse points that make this magic possible having been present in NTFS since Windows 2000.  The ability to convince the OS that this is present appears to only work on Vista and above.  I wrote some programs to exercise this back in the early Tarpon days; none bore fruit, but then again, Windows 2000 was a required operating system at the time, so more options may be available.

But wait, I can mount a volume into an empty directory on Windows XP.  Yes, yes, you can, but you can only mount the ROOT of that volume.  On Vista, you can mount subdirectory to subdirectory -> Much more useful!

The foundation commands are: As an admin:

CD \Program files\Citrix\RadeCache

mkdir \\apphub\share\profiles\profilename\GUID_v  (and then uncab the cab contents into that dir).

MKLINK /D GUID_v \\apphub\share\profiles\profilename\GUID_v

DIR GUID_v  ( Local directory, then you see everything on the network server )

In theory, DONE!  The streaming system will redirect stuff into the RadeCache, the I/O manager and NTFS will convert those into reparse points where they will ultimatley go back down the stack, this time as a network file system operation and all will be happy.

But, runtime, the Application Streaming system gets unhappy.  I have tested this only with the non-released streaming client that I'm using on all my machines, including the machine I'm using to write this post.  There is some chance that this will actually work on the currently released stuff.  The stuff "you don't have yet" does registry MOUNTS rather than loading of registry information.  This is far more efficient at runtime, but it is also dependent on the operating system supporting the mount and when asked to mount a registry hive from a non-local disk location, the system responds, very simply, "NO".  DUDE!  Nothing is local!  You think C: is better than that redirected space, trust me, it's all bad!  Please push on and give it a shot!  Nope.  Doesn't happen.

Using a second disk volume
Provisioning Server focuses on providing the operating system image; read this as "one volume" per machine.  What if you had a second "local" disk volume?  Yes, that would work!  You could tell the Application Streaming system to move the RadeCache to the second disk volume and all would be happy!  The streaming system already comes with a utility to move the cache, it is ClientCache.exe and is included and installed with the streaming client.  Here's a link that describes how to use it.  So far so good.  The streaming system though will insist that this "new" location be on a "local" disk. 

The second disk volume would contain only a fully populated view of a conceptual RadeCache of a machine that has ALL applications fully streamed.  Requires the admin to create this second disk image and requires that this image be maintained when applications are updated, but all of these things are containable and might even make sense as we all get used to maintaining the layers of cake.

How do you MOUNT a second disk volume?

Nicely, this disk volume does NOT have to exist at machine boot.  It only needs to exist BEFORE the user has a chance to launch any "streamed" applications.  If we assume that they do that in their startup folder, then this means that we need the second disk in place as a function of logon.  It could be in a system logon script.

The mounted disk volume must be LOCAL.  The streaming system insists on it.  It isn't that the streaming system is against storing execution content on remote systems, it just insists on it - which is interestingly very similar to the registry mount rocks I threw earlier in this post. 

The streaming system file system filter driver observes all file system volume mounts.  If it's "local" and if its "NTFS", it gets involved in the I/O stack and can use this space as a place to store the execution content.  

The App Streaming system spends alot of time lying to applications about what is present.  Our trick is to LIE to the LYING system to fool it into believing a disk volume is "local" when everything is really remote.

How to lie?

Some customers have elegant disk systems that allow them to mount extra "drives".   If so, then the ideal such system would allow mount of space into N machines where there is ONE backing store.  In utopia, this space is read/write.  Only the Streaming Caching service will write to it and any writes done for any user on any given machine would then benefit all users on ALL machines.  ELEGANT!

I belive such systems exist, but I have not yet seen it successfully implemented.  One complication is that the streaming system insists that the disk volume be formatted NTFS.  If the file sytem isn't NTFS, we get out of the way.  In this case, we don't want to get out of the way and given numerous machines accessing a single shared source, the odds of this having NTFS as the file system are ... remote.  It will though have most everything we need.  If you have one of these, shoot me a line and we'll work with you to get the streaming filter willing to put the streaming cache into that space. 

VHD, ISO

XenServer, VMWare, Hyper-V

The machine virtualization technologies also provide tools to mount disk volumes.  If we can mount a single disk volume, local disk, into multiple machines, ideal.  If this supports WRITES, better.  At logoff, would need to merge this common RadeCache back to a master store to become the base image for additional users.  All of this would work - but it is all very involved.

Ideal world

In an easier world, the Application Streaming system itself would permit redirecting the RadeCache execution space to a central network location, on the Application Hub.  That does not exist.  The fact that I'm talking about it though should tell you that I want to solve this problem.  I've been avoiding talking about it because I am not yet satisified with the answer.  This post though attempts to discuss your NEAR term solutions.

What should I do?

Near term, populate the RadeCache into the base PVS image.  I know this isn't ideal, but it is a workable solution and solves the performance aspects.  Longer term, investigate mounting .ISO and VHD and look into abilities to leverage storage arrays to provide "local" disks that are really remote.   Longer term, understand that this is a common need and solving it, is broadly important. 

Joe Nord

Product Architect- Application Streaming and User Profile Manager

Citrix System, Fort Lauderdale, FL

I recommend: Never PreDeploy to XenApp Servers; never PreDeploy to XenDesktop clients.  Always PreDeploy to notebooks.

Background: Application Streaming supports server side execution, client side execution and client side execution that can go offline.  These are controlled with publishing; should the application be available for offline or not, and via client side utilities (RadeDeploy) to optionally predeploy the execution content to the execution machine.  Notice that PreDeploy is REQUIRED and automatic for offline, but that Deploy is optional for online. 

What does "online" mean?

Online execution means that the streaming client has everything it needs to run the application available via the network.  This means that it can "see" the Web Interface and can "see" the Application Hub.  By contrast, "offline" means that execution can still happen when the Web Interface and Application Hub are not available.

In programmer speak, this is a 2*2 matrix with 4 possible states; one of which is invalid, offline without Deploy.  I argue that Deploy + online, while valid, should not be used if the execution machine is inside the data center.  If the network is "solid", don't Deploy.  The corollary, if the machine is "sketchy", always publish for offline, with automatic deploy.  This would get the number of states down from 3 to 2, which makes programmers happy.

What is Deploy

The guts of the streaming client are unaware of whether execution is server side or client side, online or offline.  The execution engine brings execution content into the RadeCache (\Program files\Citrix\RadeCache) and fetches that execution content from a CAB file on a file server or web server (App Hub).  In the online case, the cab file location is REALLY on a server.  In the offline case, the streaming engine is pointed to the Deploy location on the local machine and does "local streaming".  I call this streaming from one side of the hard disk to another; it was my idea back at the beginning and I must say that 4 years later, I'm pretty happy with myself.  The caching activities in the streaming service and the kernel mode stuff in the file system driver are completely oblivious to "offline/online" and the streaming happens with the same code path in all cases; happy.

Deploy is implemented by copying the execution content from the Application Hub and placing a copy of the Application Hub contents onto the execution machine; in \program files\citrix\deploy.  Looking back, I wish we had put the RadeCache and Deploy directories one level deeper, but this is where they exist. 

How to deploy

There are two ways to deploy.  In the automatic case, the admin publishes an application and, to some users (you), publishes that application as available for offline.  When you refresh applications in PNAgent (plugin for hosted apps), PNAgent gets a list of all of the applications that are available offline. 

The administrator has a choice when publishing.  In the Access Management Console, the admin can say that the Pre-Deploy should happen immediately, or they can state that the Deploy should happen on the FIRST use of the application.  I have not yet found a single admin who has gone with the Deploy on application refresh.  Network storms at 8:05am are not how they want to start their day.  This means that applications are not available for offline until the application has been run online, at least once.  

Eventually, the user (you) clicks on an icon to run the application.  The application is run as if "online" and in the background, the server side content is xcopied down to the Deploy location on the execution machine.  This automatic action does not occur if execution is on a XenApp server. 

The first use of the application continues in "online" configuration and it isn't until the next fresh launch of applications from that profile that the offline content is used.  Looking back, it would have been better to have completely copied the offline content to the execution machine before ever running the application.  The user feedback could then be ... "Copying your offline content, please go get a cup of coffee while I transfer 2GB to your machine".  Instead, the user is faced with online execution, competing  copying execution content from the file server into the RadeCache AND the streaming system copying the whole CAB file from the file server into the Deploy location and this presents a less than ideal first time experience, if the application is large.  Yes, the offline deploy is done on an "idle thread", but single user, 2GHz + dual-CPU machines spend lots of time idle, so it competes.  Eventually, you get the execution content local and further streaming from that profile is local to the execution machine AND available for offline. 

How to PreDeploy to XenApp server

First, don't!  Second, you could use RadeDeploy.exe to command the deploy to happen, and if you do, it will occur.  This would normally be done by software management system.  The streaming client running on that server will at runtime look for the deploy content and if it is there, it will use it. 

WHY NOT Deploy to servers!

It isn't about the first execution.  Yes, the first execution will be faster if you are PreDeployed.  The application update though will be SLOWER.  You first deploy.  This happens once.  Later, you will update the application numerous times.  The update in "online" scenario is more efficient than the update in "offline".  I should note that I really mean that the update is more efficient without PreDeploy.  Online/Offline is not the trigger, the trigger is Deploy/NotDeploy - then again, one shouldn't deploy without offline, so they really are the same state.

Consider a large application, say 1GB in size.  If you PreDeploy it, you will have everything available for cache fills based on copies across the local disk; efficient.  If instead, you don't PreDeploy it, some small percentage of the application will be runtime copied into the execution cache.  

When it comes time to update the applications MOST of the execution content will be unchanged.  The Streaming Client will "KNOW" what files are the same, and which are different and it will "KNOW" this without a deep look at the files inside the CAB.  In the online case, all of the "same" content can be immediatley promoted from version 1 to version 2 with no network involvement and this happens whether deployed or not.  In the "deployed" case,  the streaming system though needs to bring down EVERYTHING and this is very inefficient. 

In Streaming Client 1.1, this resulted in a full copy of the CAB file from the network server to the Deploy location on the client.  Notice that it is a single file (the cab); the client copied the whole thing and this is not friendly to the network and worse, it is very bad if the user is remotely connected over a VPN; where Deploy SHOULD be used.  

In Streaming Client 1.2 (XenApp 5.0), the streaming client was enhanced to bring down only the "deltas" from version "old" to version "new".  This is much more efficient network wise and brings Deploy case on-par with the online case, but it is still less efficient.  Ultimately, the whole cab file is needed for both "old" and "new".  The streaming system knows what has to exist, but the CAB file format does not allow update in place.  To save network usage, the streaming client trades client side CPU and Disk activity to save network.  This means that the Deploy update consists of a full un-cab of the CAB file, apply the updates, and full re-cab.  For a 1GB application, this can take minutes!  10s of minutes?   What is my machine doing!!  AArgh!

In a XenApp hosted environment, there is no need to pay the un-cab/re-cab penalty; the most efficient update is directly inside the execution cache and this has existed since the first release of Application Streaming.  All of the above rocks thrown, are we going to make the Deploy/Offline case more efficient?  Certainly!

What should I do to optimize first time launch on the XenApp server?

Turn on "-e" RadeRun switch.  Here is a KB article with more details.  Execute one application from each profile, ONCE.  Finally, be sure to then turn off -e.  Yes, I say turn it off because if you don't you will un-pack everything from the CAB on EVERY launch of any application for every user - which will be dog slow.   

You can achieve the same thing by running RadeRun.exe directly from the command line, specifying the -e switch.  This best done by automated systems, when the users aren't around.

With the single run with "-e", everything will be extracted from the Application Hub and placed into the streaming execution cache.  The streaming system will still do it's "just in time" programming, but it will pretty much never conclude that a cache fill is needed and the result is that execution will not suffer a first time launch penalty.

Enjoy, 

Joe Nord

Citrix Systems, Product Architect for Application Streaming and User Profile Manager.

The game in Application Streaming is "profile once", "run many".  The Streaming Profiler captures stuff at profiling time and stores it someplace that it can be referenced at execution time.  This post describes the format of the "stuff".  "Describes it" is a loose statement of documenting the format because it isn't exactly "hidden" from the get go.  Having a written description though can be useful when looking at the content stored on your Application Hub. 

The stuff stored on the file server (Application Hub) is two portions:

1) A profile contains a list of applications, by name along with a list of icons, by image. Combine that together with the location of the Application Hub and the name of the profile and you have EVERYTHING you need to publish an application.  The "profile layer" contains only the publishing data.  Notice that the Citrix Access Management Console which publishes applications in XenApp needs to look only at the "profile layer", this is on-purpose.  You can publish a single icon and later implement multiple execution methods to execute that application, for example, 32 and 64 bit versions of the application.  Publish once!  Less icons, less publishing.

2) One or more execution targets.  The targets contain only the stuff needed to RUN the application and a profile may contain multiple execution targets, though fewer is preferred. For convenience, everything associated with the target is contained inside a CAB container file, but the execution stuff is extracted to non-compressed space where it is accessed for execution.  This run time extraction occurs only once for the life of a target and the extracted content is shared across all users on a XenApp server side execution.  This means that only one user pays the "first launch penalty" and even that can be pre-performed.  The targets are named with GUID_version.cab and that naming exercise is outlined in excruciating detail, here.

Directory structure

Each profile is stored in its own directory; generally beneath a common "profiles" or "packages" directory on the Application Hub.  Each profile gets its own directory to simplify copying and maintenance of the profile content.  This also makes it easier to restrict access rights so that only the users who are published, can see the application profile.  Active Directory groups are very useful for this common publishing/access management.  Keeping them all in a same general location also makes it possible to have relative references between the profiles and this is exploited in Inter-Isolation Communication, so that links can be established without having to code non-portable paths to other profiles, handy.

Files that exist at the PROFILE layer

profilename.profile

XML formatted text file which is the "TOP" file of everything about the profile and all of its execution targets.  The "properties" of a profile that you can edit in the streaming profiler are all stored inside this XML file.  You COULD edit them via manual XML edit, but I would not recommend that as a practice.  The profile includes information about the applications that can be published and pointers to the icon data described later in this post. For entertainment, copy the profilename.profile to profilename.XML and then load into Internet Explorer or Firefox for syntax assisted viewing.  You can see "secret stuff" beyond what the profiler GUI shows you, such as the version data of the streaming profiler that was used to create the profile or to most recently modify a target.

icons.bin

Binary file holding bits which represent all of the icons of every application installed into every target of the profile.  The Applications at the profile layer are a "set union" of all of the applications of all of the Targets.  Likewise, the icons at the profile layer are the "set union" of all of the icons of all of the applications of all of the targets.  Notice that Targets do not store icons as icons are not needed for execution.  The format of the icon data is binary.  There is no API in Windows to "convert this HICON to binary", or to extract all the icons from this file and store them where I can get to them later in a screen resolution that is right on the machine that I will be running on then rather than the machine that I am running on now.  The streaming profiler/client pull this off and doing that along with upgrading color depth to 32-bit means that icons file format is an area ripe for change.  We have changed the format of the icon data 3 times that I can recall.  To get to the bits, use Profiler SDK (bottom of post).

hashes.txt

SHA1 cryptographic hash of every file at the profile layer. sha1 = rich man's CRC.   More details on these in target section.  At concept, this file is the "key" to Digital Signature.  The profile XML data can be digitally signed and inside that file is a hash of the hashes.  The hashes has a hash of all of the files of the profile/target and all of this flows downhill to assure the streaming client that the content it is pulling off of the Application Hub is the SAME content that the signing administrator put there.  Neat stuff! 

GUID_version.CAB

One or more CAB files which hold the execution content, the Target.  On the development team, we all rather wanted a container file.  A single file makes pre-deployment an atomic test. Do I have it?  The answer is yes, or no, but never "maybe".  This simplifies pre-deployment, but looking back it wasn't critical to have a container.  An important point: the CAB file is NOT compressed using the default Windows utilities. Those would cause the whole CAB file to change on a single file update and would make the action of pulling a single file out of the cab file slow.

Scripts (sub directory)

Subdirectory holding all "scripts" defined at the profile level.  Scripts are not text, they are binary.  Anything can be a script including textual bat files, executable content such as .EXEs and even "data". Though that last one should be marked as "disabled" so that the streaming client doesn't attempt to "execute it".

TARGET FILES (Stuff inside the CAB)

When browsing profile content using Windows Explorer, after opening the CAB file, sort by PATH.  All of the important stuff will be at the top (root of the CAB).

fontdata.dat

Font's captured during profiling that are accessed during execution

InstallRoot.tab

This file appears to be textual as it has a .TAB extension, but it is binary.   The .TAB file contains the "registry" content of the installed/profiled application captured during profiling.  In early development of Application Streaming, we actually used .REG files for this intermediate storage.  The content of the .TAB file is the same as a .REG file, except the hard coded paths common to .REG files are replaced with "relative" path storage and the registry content is converted to binary for efficient access.  I have previously described how to extract the registry content.

hashes.txt

The calculated hashes were originally to be used only to support digital signature of the entire profile (which is optional).  They work out to be really handy for other purposes.  Having these from version 1 to version 2 of a target upgrade lets the streaming client know which files, among potentially thousands, are the same from version to version, without even looking at the server side version of the files. This is so time efficient that the hashes are calculated and used even if the profile will not be digitally signed.  I say efficient because the streaming client can "know" that a server side version of a file is the "same as it was" based on a 4 byte in memory compare!  Good stuff for the bit heads in the crowd.

The TARGET level hashes file also stores the file date, time, size, attributes and everything else associated with directory enumeration.   The contents of this file are what is used to respond to directory enumeration at execution, for the files that "aren't there".

Yes, sha1 hashes can be a beast to calculate during profiling.  A high percentage of the time that passes during "finalizing your package" is spent calculating these hashes.  The other large portion is spent creating the CAB file.   The theory is that time spent in profiling is less important than time spent by the users, so the hashes are calculated, always. 

preextract.txt

The streaming client consults this file and will extract the given files before launching the to be run application.  IMHO, this file is bogus and should not exist, but back in the day I lost that battle.  I am pleased though that a current inspection of a "fresh" profile has this file created, but empty.  This means it will still be compatible with "older" streaming clients, but gets back to having no purpose.

SNF.txt

In the early days, this file held the short filenames that go with files that also have long filenames.  This is to support applications that use only short filenames at runtime and absolutely insist that the profiling time (install time) and runtime short filename be the same name.  The most demanding offender here is the MSI installer service itself, which is interesting to ponder as you would think that the MSI installer could handle long filenames. 

In the modern days, the contents of SFN.txt have been moved into the end of the per-file entry on hashes.txt.  This means that SFN.txt is a legacy that exists today only to support "older" streaming clients (backward compatibility).

SandboxData.XML

Another XML formatted file.  This one holds the "rules" which govern isolation. All of the "isolate", "ignore", "redirect" and "strictly isoalte" rules are defined in this file.   You can see the "default" rules by looking at the installation directory for the streaming client on your system.   The Streaming Profiler reads the installed version of this file to define the default rules for all created profiles.  Then, any runtime defined rules get added along with any rules that the administrator defines using the GUI of the streaming profiler.  The Targets rules are stored "with the Target" so that it's rules are "steady" even if a new version of the profiler is installed.

Device/drive letter

Represents the installation file system content for the captured installation.   It is a grand mystery why the AIE/Rade FSFD stores "Device" in the name of the captured file content.  In a perfect world, this would be replaced with just "drive letter" or even better, "boot drive". 

Summary

More details than you probably ever wanted to know.  But, there it is.  Please don't become too dependent on the specific format; we do not promise that it won't change.  If you want to change things in the profile or even look at it programatically, the RIGHT method is to use the streaming profiler SDK to get to the bits.  Really!  Use the SDK!  It insulates you from the format.  Knowing what the native bits are though, can be useful.

Joe Nord

Citrix Systems

This is the 3rd post in a series on Inter-Isolation Communication.  I previously shown the layers of glass with IIC and how the streaming profiler can mount sub-profiles as a part of running an installer for a new profile.  This post describes how ANY profile can be "sub" to another profile and how "profile trees" result that are potentially far deeper than one level.

Consider that before inter-isolation communication, administrators wanted to allow communication between sandboxes.  They want to have two profiles that they know are good; but want the applications in those sandboxes to be able to talk at runtime.  With inter-isolation communication, any profile can be part of an IIC profile.  The existing profile can even be made with a profiler that had no idea what IIC was.  The IIC supporting profiler and client first shipped with XenApp 5.0 (Streaming Client 1.2), but the profiler/client at this level can define IIC links to profiles created at earlier levels.

It is easy to consider that an IIC profile links 2 or more sub-profiles.  What is not so obvious is that IIC profiles may themselves be the item that is linked.  That is, the IIC profile may be the "linkee" rather than the "linker".

Consider this graphic
 
It is much to consider, but "TOP" is a dependent profile that links to 3 other profiles.  As TOP has its own isolation target, the profiler will mount everything underneath as a part of running the profiling activity to support creation of the Target at top.  The profiler supports setting the relative height of isoaltion spaces within a profile, so the order of A, B and C can be adjusted.  The layers of B1 and B2 though are defined as part of profile B and must be adjusted at that level.

Configuration of the profiles

A.profile may have come from an ealier version of the streaming system, where it did not know about IIC. 

B.profile is "links only" and this means that it was created with the IIC supporting streaming profiler and since it is an associative profile, it is merely "pointers" to the dependent profiles, which are B1 and B2.

C.profile is very much like TOP.  It mounts the C1 profile and then C has a produced execution target of its own which is the result of running installation activities while editing C. 

Eventually, TOP exists and all of this stuff is stored on an Application Hub, somewhere on your network.

NOTE: All profile links are by name only and are required to be on the same application hub/file server.  This is on-purpose so that all links are "relative" and the profile content can be moved from one server to another without having to edit any of the profile information.

PUBLISHING

When publishing applications, the Access Management Console will open TOP.profile and will present a set of applications for publishing which is the SET UNION of all of the sub-profiles.  It is possible that you want to publish all of the applications, but possibly more likely that you want to publish a subset.  Either way, as the administrator, you have the full set available for selection and it doesn't matter which target provides the initial executable.  

That pretty much puts it all together.  This stuff has been out for a while, its complex - but powerful.  

Joe Nord

Citrix Systems

The streaming profiler defines IIC profiles as either "associated" or "dependent".  These are fancy words which translate into, is it "just links", or does it have an installation image of its own?

For background, refer to this recent post on the layers of glass for Inter-Isolation Communication.  This shows how the isolation system creates an N-Layer isolation environment based on a collection of separate profiles.  What was not mentioned is that this also occurs during PROFILING.

Consider that you have a big profile and want to attach a small application to it.  Let's call the big one MS Office and the small one, YouSendIt.  No, I haven't tested this, I just needed a small office plugin that is requires installation "on top of" an existing MS office installation. 

The grand question: how do you separately profile MS Office and YouSendIt if you can't install the second app without the first already being installed?

Answer:  You create a DEPENDENT profile and you MOUNT the big profile before installing the app.  This happens at this screen in the streaming profiler.  Observe the red-circle...

 
In the "unchecked" form, the profiler is creating a dependent profile and will give you an opportunity to "run an installer", which ultimatley means that this profile gets its own layer in the isolation stack.  If the circled box is checked, then the profile is an "associated" profile meaning that it is only LINKS. 

First, consider "just links"
Notice that a "just links" example has no installation image for the associate profile.  The profile merely defines that the streaming client should IIC connect other existing profiles at runtime and make available for publishing, all of the applications of both A and B.

Back to dependent profile example

During profiling, the streaming profiler "mounts" all of the sub-profiles (A and B as below) that are defined as part of the new profile being created.  What the installer sees is a merged view of the sub-profiles and an installation target being created on the top. This highest target is "writable" at profiling and there is no per-user space during profiling. 

When "installing", YouSendIt will modify stuff on the machine including potentially also in the MS Office installation space.  In all of these, the isolation system prevents those writes from hurting the sub-profiles and stores the written-to stuff into the highest level of the isolation stack.  When done, the streaming profiler collects the "changed" stuff and stores this as the isolation target that is part of this new profile.

The end answer is a profile, called "MS Office with YouSendIt", which is a "link" to the big MS Office profile plus a true installation image (target) for the installation of YouSendIt.  At runtime, this is what the streaming client sets up to run the application.

The profiler can also adjust the relative position of the linked profiles (in blue above) via the move up and move down buttons on the sub-profile selection panel. 

Joe Nord

I have previously written about inter-isolation communication and how it can be used to minimize the application maintenance for isolated applications.  That is, to retain the advantages of isolation while getting maintenane of isolated applications back on-par with maintenance of locally installed applications.  I have also written about the layers of glass which make the isolation possible.  The previous discussion of the layers of glass ignored IIC.  This post combines the layers of glass with IIC to show how IIC happens and how this allows separate maintenance of the various installation images that are run under isolation.

Refresher on Inter Isolation Communication

If you have multiple big applications, all of which also need the use of a small application, running under isolation says that you have to install the small application into each of the "big" applications.  If you later decide to update "small" application, then you have to update small application in ALL of the using profiles. 

Compare that to a local machine installation where one installation of the small application effects all of the big applications.  That is handy, but it is also something that isolation systems try to avoid.  In a "completely isolated" world (Streaming Clients before XenApp 5.0/Streaming Client 1.2), the only choice was to install the small application into each of the big profiles.  You could put the small application outside of isolation and have it visible to all the big apps, but stick with me on the concept.  You want the small application (lets call it Adobe Reader) usable to each of the big applications, but you don't want anything installed on the local machine. 

Without IIC, you have to install Adobe Reader into each of the "big" profiles and this creates a situation where isolation produces MORE maintenance than does installing the applications locally.  Sure the local install case has opportunity for the apps to crater each other, but you still only have to install it once.  Generally, it is the "big apps" that are colliding with each other and this maintenance of the small applications becomes an isolation headache.

IIC gets isolated execution and locally installed back on the same "maintenance footprint".  With IIC, you have ONE place to maitain the small application and updates applied to the single small application image automatically fold into the big applications at runtime.  PRETTY COOL STUFF!

How does it work

When having this as a goal, we considered a complex scheme to open COM and other communication holes in isolation spaces so that applications in different sandboxes can communicate across the sandboxes; honest to goodness INTER-isolation communication.  This becomes pretty complex when you consider how to define holes that should leak through and how to define where that should not occur.  You would also also wonder what you missed.

We cheated

Instead of defining complex communication between isolation spaces, we wrapped the SEPARATE isolation spaces into a single isolation space.   Now, you have separate MAINTENANCE of the "big" and "small" profiles, but they are runtime combined into a single image.  Notice that this eliminates all the complex stuff of defining holes between isolation spaces, because in reality, all the apps are running in the same isolation space; but they are separately maintined.

This decision requires an assumption that big-app and small-app already get along.  A safe assumption as millions of locally installed instalations already confirm that for the most part, these TWO applications get along.  We still want them separate from other applications and separate from the local machine, but as far as these applications go, we already know that they get along.

In many ways, this is moving the isolation bar to the left.  If right means isolate everything and left means isolate nothing, then the early isolation system was all the way on the right.  IIC moves it a bit back to the left, but in a measured manner that produces a good outcome at a reasonable cost.

Layers of glass

The classic layers of glass view shows 3 layers.  Here's a diagram along with graphics that show how the bottom two layers are actually shared across all the users on the machine. 
 IIC takes this to new depths!

There are really N layers of glass.  The parts in dark gray above which are marked Installation/Execution image are really deeper than 1 layer.   Each of the profiles that is part of an IIC profile provides 1 layer of the "installation root".  Here's a graphic.
 
Position matters!

I usuall title this "DLL Hell is back!".  When viewed from above, the application sees the world through the mask of the layers between it and the real machine.  Each layer can add or DELETE registry and file system content and the layer at the top (the per-user layer) wins!  Beneath it, the top installation image wins in preference to the lower layers.  To avoid this, the Streaming Profiler allows the administrator to define the order of the layers of glass.  This is subtle, but occasionally it is important.  You'll see "move up" and "move down" buttons in the profiler panel that lets you select existing profiles to be made part of this new profile.  The sub-profile at the top is the one that is highest in the layers of glass.

Generally speaking, the order of the installation images in an IIC profile doesn't matter because any applications that the admin is putting together likely already work cleanly together.  The profiler UI is really there to accomodate the exception case.  The admin could just as easily cure the collision by deleting the offending content from all but one of the layers.  The one remaining will "shine through" all of the higher levels which do not have that same content.  Its still good to make it adjustable. 

Making it adjustable also lets the admin put the "biggest" at the top.  Sometimes this isn't possible, but if it is, putting the biggest at the top should provide more efficient execution as the isolation system will find things earlier.  It starts at the top and works it's way down and if it finds things sooner, it runs faster.

There is no theoretical limit to the number of layers.  Anything beyond 5 starts to seem busted in concept, but the isolation system has no inherrent limits to the number of layers.

Joe Nord

Citrix Systems

Application Streaming executes applications whose names and icons are listed in a PROFILE.  The streaming system uses the contents of an execution TARGET to run the application.   I have previously written about this separation of publishing data from execution data and how it lets a single profile support multiple execution environments.  For example, only one set of icons are published even if the execution platforms will be for 32 and 64 bit systems; which have different execution images (targets).

As the Targets mature, their version number increments.  Users who have the application up "now" keep using the old version and users who freshly launch the application get the latest version published.  Lots of details on this in this post, RADE => Rapid Application DElivery. This is what allows the application to be updated HOT. 

What is often glossed over is that it is a one way movement! 

If your users are happily running on version "3" and your update the profile (Target) to get it to version "4", your users will be moved up to version 4 the next time they run the application.  

What if version 4 is "BAD"?

Officially, there is no way to "go back".  The original developers of Application Streaming failed to implement "rollback".  I was there, so first rule - everything is my fault.  Just for the record though, I jumped up and down, screamed and held my breath to force the issue - and still lost.  The idea was that even if you roll the targets back, there are other items to consider such as profile level changes that affect the targets; can you roll everything back?

Target Rollback is not implemented in the profiler GUI, but it can be manually achieved.

The profile and all of its targets are stored in a directory on the file/web server which is also known as the Application Hub.  The top level file is filename.profile which is an XML formatted file, which is the keys to the kingdom for all of the other files for the profile and all of the files of each of the execution targets.  The targets are stored inside of a CAB file, for easy transport and to provide an atomic test of "deployed" or "not deployed".  The naming of the targets is GUID_v.cab and the definition of which GUID goes for which Target is present in the profile XML data, which is stored inside the .profile.

This question has come up before.  Here's a forum post that talks about it.

I summarize it to these steps

• Copy filename.profile to filename.xml
• Favoriate XML editor, edit filename.xml (I use Visual SlickEdit).
• Tools menu, beautify, enter, enter (this is why edited as XML - syntax assist editing).
• Search for "<Ver>" 
• Change the integer that is there to be the smaller number that you're after
• Save
• copy filename.xml filename.profile
• Rollback is complete!
• No need to erase the "newer" CAB file - it may still be supporting users who are using the "BAD" version.

Things are mostly as they were and your life and users are now happy!

Immediately, new executions of applications from that profile will use the "older" Target. 

NOW - What won't work?

The editing that you did - which caused it to not work - is gone.  That TARGET will be overwritten on the next save of that target inside the streaming profiler.  So far, nothing will occur that seems bad.  But, if you managed to edit the target fast enough, you could replace the already in use GUID_version.cab and this could heavily confuse your user community.  For purposes of rollback, I assume that the user community are already standing in your office and will understand when you tell them close the app and launch it again.  There could be some "left overs" in the users Cache which would be different than the next time they run that same version of the same target.  Hey, it could happen - but it is really unlikely.

More stuff that can go wrong...

IF you edited the profile and stored SCRIPTS at the PROFILE level, then these scripts are GLOBAL to all of the Targets in the profile.  IF the Target is set to "inherit profile level scripts", then the new scripts will apply even if you rollback the target.  Good news here, the scripts at profile level are stored in a directory on the network server (Application Hub) where you can replace them - don't tell anyone I said to do this.  You will also then have to force the scripts to be re-downloaded onto the execution machines after the update.  More discussion on that here.

What really won't work?  DIGITAL SIGNATURES!

If your profile is "digitally signed", then by changing the profile data, you have invalidated the signature and the streaming client will do its job and declare the end of the universe and otherwise refuse to run the application. The streaming profiler will (or at least probably will) allow you to load the invalid profile, fix it and save it back to get signatures in place again.

For people going "digital what", the application streaming system uses SHA1 hashes of every file in the profile, every file in every target and an overall digital signature of the profile XML data to authenticate that the streaming content is from an approved source, e.g. you.  This is optional, so if you're not doing it, no worries. 

It is also possible that Pre-Deploy could get confused - thinking it has the latest version when it doesn't.  The good news is that once you update it again, the later version will be "new" and will be brought to the execution machine.  If you're only using stream to server, this doesn't really apply to you because deploy isn't normally used unless you're taking the application offline.

Summary

The fast majority of the time, target rollback will work just fine.  The most common reason for rollback is that an application update was applied - and it made things worse, not better.  So, providing a means to rollback the application image achieves what the admin wants; albeit with a few things to keep an eye on.  It's not in the profiler GUI, maybe it will be some day.  Until then, the needed function can be achieved.

Enjoy.

Joe Nord

About once a week, I am asked "how do you start streamed applications directly"; that is, without using the standard publishing infrastructure.  This post describes how to do it, where it works and where it doesn't.

The quick version:

• Can you do it?  Yes. 
• Does it work? Sortof.

The second part above is the part that has kept me from promoting this as a concept.  That said, I'm not the first to bring up the idea, or at least to publicly talk about it.  Jeroen van de Kamp included information similar to this during his BriForum pitch, Chicago, June 2008.  I was there.  Being the App Streaming "Citrix-guy", some folks in the audience thought I would be bothered by the idea of Jeroen discussing how to bypass the Citrix publishing system.  I wasn't bothered at all - the puzzled look on my face was amazement that his presented method actually worked!  Read the rest of this and you'll understand why I was puzzled. Short version, the demo should have gone down in flames!  But it succeeded.

First, some history on how applications are launched.  Lots of components: PNAgent, Web Interface, Internet Explorer/Firefox plugins and the streaming client.  I have written about this application launch activity before.  The important part is that everything comes down to a little program, raderun.exe, who throws the launch request over the wall to the Application Streaming service that does the work.

RadeRun is to streaming what WFCRun32 is for hosted applications.

wfcrun32 takes an .ICA file as a parameter, establishes a communication link to a XenApp server where the application is executed.  RadeRun takes a .RAD file as a parameter, establishes a link to the streaming service, where the application is executed.  

To get the parameters for raderun, run it with no parameters and a help screen will be displayed.  This isn't usually shown to users or admins, so the form of the help is a bit rugged, but the needed information is displayed in a message box along with an alarm sound that triggers and scares me every time I run it, but I have disgressed.

How to launch applications without publishing

Jeroen hadn't invented anything mysterious; he had resurrected the old!  The actual parameters and method he used to kick off the application was slightly different than the method I'll show later in this post, but the concept is the same.  Immediately at the demo, I was intrigued because I thought his alternate parameters had overcome a limitation that I didn't know how to get past.  Having the benefit of the ultimate documentation (source code), when I came back to the office I took the demo apart line by line and ... still concluded that it shouldn't work.

Some history...

When Application Streaming was first being developed, RadeRun was all there was.  There was no publihsing infrastructure and none of the important components in XenApp knew anything about "streaming".  Everything was developed in parallel.  The streaming team needed to launch applications to test the isolation and caching systems, but all they had was RadeRun.  At this point in time, raderun with parameters works great!

Time goes on, programmers working the Access Management Console, IMA, Web Interface, PNAgent, Internet Explorer and Firefox plugins all teach their code about application streaming and all of a sudden, RadeRun isn't the only way anymore.  Worse than "not the only way", when the product goes out the door, the standalone executable method doesn't work anymore.  We know it doesn't and can all remember when it did, but no longer matters because the correct methods are in place. 

The loss of function happened as the infrastructure grew.  Why?  One of the things added was the central management of applications running stream to client.  The Access Management Console can DISPLAY information on what applications are in active use, by any given user and a variety of other information.  This happens only for stream to client for online execution.  For offline, you take the app with you, so the "its running now" information isn't useful for much because there is no way to send it home.  Here, you get "Joe has this app available for offline use".   For server side, none of this applies, the hosted session has its own monitoring and the streaming client skips all of this.   This works out to be important for the view of this post.  Server side, the limitations I'm talking about do not apply. 

This makes the "does it work" question posted at the front of this post have an answer of "yes, server side".

Back to the online stream to client discussion.  When the launch activity starts, the XenApp host infrastructure generates a launch ticket for the streaming client.  This is provided to the client as part of the launch processing.  Let me reword, that, the launch ticket is provided to RadeRun as a parameter.  The streaming cleint does its stuff and eventually tells the central infrastructure that the launch for this app has proceeded good and is about to succeed.   In the normal case, the publishing infrastructure responds, "great, good to hear it".

If the launch ticket is "unknown, invalid or blank", the publishing system responds "who?".  And the application launch unravels.  Now, you would EXPECT that the streaming client gets this unexpected response and then ... PUSHES ON.  Instead, it displays an error message close to this: "The Web Interface returned an unspecified error".  Then, it aborts the launch.  Technically, the launch is aborted in parallel to the message box, so by the time you see the error, its too late to ask the debugger what happened.  Bummer.

BUT IT WORKS
Here's the rub.  I saw Jeroen's demo and it worked.  I have myself done it hundreds of times because when testing the streaming client, I often like to skip all the publishing steps - it really saves alot of time! 

But, I know that the code says it shouldn't work -- but it does.  My experience is that it works about 90% of the time and works most-often when the streaming client has already run a successful application, recently.  If it doesn't work 100% of the time, it isn't a good solution, so I've avoided bringing it up.

But I still get inquiries

Customers want it to work this way so they can engage all kinds of really valid scenarios. 

When does it fail?  First thing in the morning, it tends to fail.  Launch email program or any other really published application successfully, then non-published launches start working and you can't make it fail again.  I have no explaination for the success and I have studied it.  The fact that it is sporadic makes it harder to track down and correct, but it would appear that the client is caching a recent success case and using that to say "good enough", push on and have it work.  Bla bla bla - not good enough.

What to conclude: I want it to work and you can accept this as a heads up that it might start working better in a future release.  No promises though - notice that since BriForum in June, there have been TWO streaming client releases and we haven't changed the behavior yet. 

Enough on the "history", here's example batch file on how to do it.  For this example, I profiled TextPad and stored the profile to the C:\PACK directory on the local machine hard disk.
Batch file to start the application

start "" "C:\Program Files\Citrix\Streaming Client\RadeRun.exe" /package:"c:\pack\textpad\textpad.profile" /app:"Textpad"

The supporting .RAD file

[RADE]
PackageLocation=c:\pack\textpad\textpad.profile
InitialApp=Textpad

Run the bat file, and textpad comes up running inside of isolation; no publishing required.

I hope this is useful.  Sometimes I don't know if it helps to know things to this level of detail - or to know the unexpected behaviors in the technology that can cause headaches.  Hopefully at least makes for good reading.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

Doing a bit of system maintenance, I noticed some interesting stuff today about my %TEMP% folder. 

The folder and its decedents are 13GB in size (that's with a G and 9 zeros) and has 9,483 files!  This is almost equal to the "free space" on my drive.  Besides needing a bigger drive, this isn't a good use of space on the machine.  Hum...

How'd it get there?  I'm not totally sure, but a big piece of it is debugging the streaming profiler.  After getting half way through profiling of an application - killing the profiler.  I do this alot.  The profiler tries to be a good citizen and clean up when it is done, but if you kill it, it never gets the chance.  I'll also add that the profiler doesn't delete on the way "in" as it ASSUMES that their might be other instances of the profiler running at the same time, so it doesn't delete their stuff.

Back to me.  13GB wasted in temporary file system space!

XenApp hosted people have it easy.  There's a configuration option to obliterate the TEMP folders on logoff.  I'm a domain based user using a machine that is part of the domain.  I don't get the automatic delete on logoff and because I am not admin of the Citrix company domain, I also don't have ability to set the setting to obliterate everything.  In an ideal world, I would have this Citrix Profile management stuff on my domain and all of this would be magically taken care of for me.  We will get there, but for now I still have 13GB wasted space!

What to do?  Just deleting it won't be good enough - it will come back!

I went browsing the internet and found this nice article on how to clean house.  I can't make deleting the TEMP space part of my system's logoff, but I can command my machine to run batch files of my choosing at logoff.  Great!  If only I had a most-excellent set of batch files to help with the house cleaning.

A bit of coding and now I DO have these and I'm so happy with myself now that I now share the batch files with you.  Installation instructions are inside the batch file comments.

First bat file: logoff-script.bat@echo off
REM This script runs at every logoff for every user.
REM Enable by group policy (local policy)
REM
REM Start/Run GPEDIT.MSC
REM    User Configuration - Windows Settings
REM    Scripts - Logoff
REM    Add - Add this script.
setlocal
set OUTFILE=c:\logoff-script.log

REM Use a worker bat file to faciliate redirection of output.
call logoff-worker.bat | tee -a "%OUTFILE%"
endlocal

Second bat file: logoff-worker.bat@echo off
REM Called by logoff-script to faciliate redirecting output to log file
echo.
echo logoff-script running
date /t
time /t
echo Erasing TEMP directory

echo BEFORE
du -q "%TEMP%"

rd /s /q "%TEMP%" 1>nul: 2>&1
if not exist "%TEMP%\." mkdir "%TEMP%"

echo AFTER
du -q "%TEMP%"

Things to notice.  First, I logged the output to the root of drive C:.  Mere user's don't have privilege to write to this space and on Vista, you'll probably get a UAC dialog.  Easy to fix by writing to My Documents or similar.

In the main bat, there is a call to "tee -a".  There are a million tee filters on the Internet.

In the worker, there's a call to "du.exe" to log the before/after statistics.  This is the du.exe utility from Mark Russinovich (SysInternals).  Download from Microsoft Technet.  If you don't have the DU utility and still want to log output, dir /s "%TEMP%" 2>&1 | tail -2.   Right, better to use the tool.

What about the registry?  If the file system has garbage left over from interrupted profiling, then the registry should also be poluted.  Went looking, nothing there.  Luck?  Probably not, but I uninstalled/reinstalled the profiler and client yesterday, so it is possible that the installer took care of this for me.  The place to look is HKLM\Software\Citrix\AIE, yes AIE!.  More information on the temporary spaces of the streaming profiler can be found here.

I'm not asking for a "mine is better than yours" tool comparison, but if you have a method that just plain makes this batch file stuff unnecessary, do post a comment to let me and others know.

Enjoy, 

Joe Nord

I compliment Microsoft on UAC.  This is a painful problem, but somebody has to encourage application vendors to fix their stuff and the UAC dialogs seem to get that point across.  Complaining to the users forces the application vendors to fix the apps, but still allows the applications to run. 

Here though, I am torn.  I make a living convincing poorly written applications to run in a XenApp world and if all the applications are fixed to make them runnable on normal user privilege, there won't be as much magic to work.  Friends tell me to not get discouraged; application vendors will continue to produce poorly behaved software for eternity; please no comments about Citrix...

The common question

Can Application Streaming help convince this poorly behaving application to run successfully on XenApp hosted?  Answer: You betcha! 

Applications can desire privilege for many reasons; some of those reasons are valid, but given the flury of UAC dialogs we saw when Vista shipped, many of them are invalid.  These invalid ones are great candidates for running under isolation because applications that want privilege, can be run under Application Streaming and then will successfully run in a terminal services world, without privilege and not complain about it. 

Often, the incomptibiltiy is a small problem of the application wanting to write to protected spaces at runtime, such as \Program files or HKLM in the registry.   Windows programming 101 - you can't do that, still many applications do, especially those with a long herritage on Windows 9x and then ported to NT.  The UAC dialogs are helping and we're getting on a improved track to applications that are designed to work with users running on user privilege accounts and by inferance, becoming multi-user XenApp hosted execution friendly.  This is a great boon for application execution on TS/XenApp; more applications will "just work" even when locally installed.  Still, a large number of applications will continue to misbehave and this creates an endless need for running applications under isolation.

Give me an example please

What happens at runtime is that the applications writes to protected spaces end up writing to per-user spaces instead.  The isolation system layers all this back together to make the application THINK it wrote to the global space and the application pushes on blisfully unaware of the deception and you as the administrator push on gleefully happy about your ability to put ANYTHING into a hosted XenApp environment.

With multiple users, each user gets their own per-user version of the "global" space and the application - for each user - sees a world that is global, yet per-user in its structure.  This same thing happens for named objects (like pipes), the registry and file systems so that multiple misbehaving applications can all get along. 

Here's a graphic to get it across.
There's no magic; only smoke and mirrors.  When you understand the smoke and mirrors, this isolation stuff becomes pretty plain to understand.   Application Streaming is a tool you can use to make your misbehaving applications behave.  Its been around for years now, it works and can solve some of your headaches. So, I say ... Isolate and enjoy.

One last thing, in case anyone wanted to know, the above graphic is actually repeated for EACH sandbox on the server. Each 3 layer view is a sandbox supporting application execution for a given user.  Parts of the isolation space are "shared" between the users because there is no need to have multiple versions of the application content present, that would just waste space.  Take a heavily loaded XenApp server with 50 users; each running a variety of applications, all streamed.  150 different sandboxes would not be uncommon.   The isolation system has to keep track of all of that and not blow up even if Citrix has people on staff who do nothing other than try to break it.  Good fun and they don't call it the "stress lab" for nothing.

Joe Nord

Product Architect for Application Streaming

Citrix Systems 

The XenApp Application Streaming system uses a File System Filter Driver (FSFD) to implement file system isolation.   I have previously described the isolation layers and how they present a different image of the system to each sandboxed application.  This post describes where in the system that the file system filtering occurs and points out places where things can get "interesting" when having a system with multiple filters, all competing to handle the file system operations.

GEEK SPEEK is turned on for this post 

File system 101 - in two paragraphs...

Before considering filtering, first consider the world without filtering. What happens when my application tries to open "letter to mom.doc".  The answer is that your application issues a command to Windows to ask it to open the file.  This almost always starts with the CreateFile() Win32 API.  It travels from "high level" Windows to "details level" Windows where it lands in the Windows NT I/O Manager.  The I/O manager looks around its registered "File Systems" and finds one who is willing to take on the work of satisfying this file operation and "sends it down".  Eventually, the file system will respond "done" and the I/O manager returns to the application.

The file system (think NTFS or FAT32), would like to move things to and from disk, but it doesn't really know how.  Instead, the file system does it's stuff to convert the high level concept of a file to "blocks" of same sized binary data that it will ask other device drivers to actually move to/from permanent storage (call it a DISK DRIVE).  The DISK device driver doesn't actually know what it is storing, but it absolutely assures the file system that if it is asked to store 512 or 256 bytes of data at location 123456, when the file system later asks for the data at 123456, it will get back the same data that it put there last week.  It is the file systems responsibility to convert high level concepts like a "file" into low level concepts like "blocks of data" that can be stored.

Most of the above describes OPERATING SYSTEM components.  To implement "isolation", you have to interfere with what the operation system really wants to accomplish and that's where filter drivers come in.

Here's a graphic.
Ponder for a moment the opening of any file.  It travels from the I/O Manager, through a number of file system filters, into the file system and eventually on to the disk device driver.   To implement isolation, we are one of the file system filter drivers and that pus us "above" the file system.  Notice that the filter drivers "below" the file system are called are not file system filters, they are "block" filters.  For application isolation systems, we focus on file system filtering.

You will hear the term "IRP".  Not to get too deep, but an IRP is the I/O managers definition of an I/O operation - it stands for "Input/Output Request Packet".  The I/O manager sends IRPs down and filter drivers and file systems do stuff to satisfy the commanded operation.  To do application isolation, the filter driver "changes" the IRP along the way so that operations intended for one location actually effect another.   I have commented before that there is no "magic" in application isolation.  Smoke and mirrors however do exist and for isolated file operations, the smoke and mirrors redirection is done in the file system filter driver.

Where the world gets interesting

Everybody wants to be on top!  The "highest" filter is the one that sees the I/O operation first.   In principle, if you're higher up the stack, you can satisfy the I/O operation without the lower level filter drivers ever being informed that there was an IRP on the way and nothing those other guys do will interfere with you, well at least not until it recurses, but stick with me on the concept - If you get to see it first, you "win" compared to the other filters. 

Who should win?

Put your programmer hat on...   The answer is that it isn't necessarily you.   The right answer can become so complicated that it becomes hard to contemplate.  Here is a link to the MSDN documentation. http://msdn.microsoft.com/en-us/library/ms790738.aspx

Priceless stuff.  What it comes down to is that when installing, a filter driver can ask the system to load it at a particular "level" or "altitude" in the filter stack.  The registry item that controls this is called "Group".

Here's a snapshot from my notebook for the Citrix FSFD, CTXSBX.sys at the XenApp 5.0 / App Streaming 1.2 level.

 
Notice that it has requested to be a "FSFilter Content Screener".  This puts it, kind of in the middle of filters.  Anti-virus is "higher" and some other filter types are lower.  Read the MSDN docs for the whole list, but it is a very complicated puzzle.  Anyone who has stayed with this article all the way to here probably already knows that it's complicated....

As the programmer who owned the device driver for the development of Application Streaming, I had to work out the "right" position in the filter stack.  Occasionally, folks inquire to relative positions and we have to look up information like this to know where one companies filters should live in relation to others.  There is a bit of "art" in the selection.  It is such an interesting thing that Microsoft holds "plug fests" where various filter driver writers can show up and test inter-operation with other vendors, in particular anti-virus.

A final note

Every few years, the model for device drivers changes.  The CTXSBX.sys device driver is a "legacy filter driver".  It had to be to be compatible with Windows 2000, which was one of its design requirements.  Since then, Microsoft have built the "mini filter" model which is the normal thing for writing filters today.   In addition to being cooler because they are "mini", mini filters also have a more refined system for specifying load order position.  I'll leave that for a MSDN inquiry, but it ultimately comes down to the same question: who's above me, who's below me.

This was probably more techno bit-head stuff than anyone wanted to read, but there it is...

Joe Nord

Citrix Systems, Fort Lauderdale, FL

Many people know that an early codename for Application Streaming was "Project Tarpon".  What they often don't know is that the original codename was "RADE", which stood for "Rapid Application DElivery".  The design goal was to provide the administrator an ability to rapidly update applications, to an entire farm of XenApp servers without having to kick users off of the server, or even kick them off the application.  The foundations of RADE are still present in Application Streaming though elegant concepts have been added including "offline" and "streaming".   The centralized publishing here is also key because a streamed application never has to be "installed" onto a server.  You just tell the Access Management Console, which servers should get the streamed application and you're done.

The reason this is important is that the stream to server case for rapid application update remains one of the most important use-cases for Application Streaming.  This post describes the "RadeCache" directories and attempts to provide a clear description of what "stuff" goes in what place and how this all leads to a world where user settings can follow a user from machine to machine as "payload" of a user profile system such as Roaming Profiles or Citrix User Profile Manager.

History

Citrix AIE technology in Presentation Server 4.0 provided the foundation of isolated application execution, but it did not provide a mechanism for updating an "in use" application.  While the 3 layers of isolation still existed, the ability to replace the execution image at runtime was still impossible because there may still be users on that server, using the application binaries that you are trying to replace.

With RADE (aka Application Streaming), the execution image for an application is versioned.  Rather than a single directory holding the execution image, each version of the application image gets its own directory.  By updating the application, the administrator tells the isolation system that for new launches, use the new version of the application.  All presently running instances of the application continue to use the already running image, but new launches get the new version of the applications.  Awesome stuff and it is very RAPID!   Notice that the administrator updated the application in one place, on the Application Hub.  It then automatically gets distributed to all of the servers in the farm!  Update once, deliver the application to all the servers!

Here's a picture of the layers of isolation showing the versioning.  Noting that execution images (Targets) are identified by a GUID, pay attention to the underbar and integer that follow the GUID.
 
Notice above that the execution space (the part that holds the image of the "installed application") is versioned.  The first version of a Target gets "_1", and from there, the right hand part bumps by 1 forever.  When an update is applied to the profile, the new image is immediately available.  Every time the streaming client launches an application, it checks the Application Hub to see if the administrator has updated the application.   Usually, the answer is no, but occasionally, the answer is yes and if "yes" happens, the streaming system builds a new RadeCache directory to support the new version of the application.   Notice that this behavior is 100% exactly the same whether the operation is stream to desktop or stream to server.  The streaming client is the same and does the same actions in both cases.
Just with the above, the application can be updated rapidly!  

Here's a picture of the RadeCache, middle layer of isolation for the computer I'm using to write this post.

Yes, there are a bunch of GUIDs in there.  I'm not even sure what they all are though I do know that the top one is MS Office 2007.   Seems the internal farm is up to the 6th version of profiling MS Office and my administrator has happily done some network magic to erase the preceeding 5 versions.  Nice of them.  That the streaming client should manage this itself is a topic I will save for a future post. 

To observe in the above, the GUID's have a number after them.  That number is the versioning.

Per-user cache

About once a month, I get asked what happens to per-user application settings on a Target upgrade.  Answer: They are maintained.  A corolary to the same question is "how do the per-user application settings get referenced for use on a next execution machine?"  Answer: The streaming content is "payload" for the user profile solution.

Here's a picture of the per-user space (files) for the machine I'm typing this on.

Notice these also have a GUID per-Target, but that they do not have versioning.  In the early days, we actually considered versioning this space as well, but ultimately concluded that handling old user data in new versions of applications is something that applications just have to be able to handle, with or without isolation.  Decided not to version it and a few years in the field says that we made the correct choice.

Since the per-user RadeCache is located somewhere beneath %USERPROFILE%, well is supposed to be.  See here for details.  Stick with me on the concept.  Since the per-user RadeCache is in user profile space, the user profile solution will copy it from execution machine (XenApp Server included) to a future execution machine where the streaming system will find it for use in running the application.  The end result is that the user settings stick with them. 

There are some limitations such as replacing a profile with a fresh profile generating fresh GUIDs and this makes the per-user space empty on first execution.   For this, the administrator should "upgrade" profiles rather than replacing them.  Upgraded execution targets retain their GUID, while new execution targets get fresh GUIDs. 

Joe Nord

Product Architect - Application Streaming and User Profile Manager

Citrix Systems, Fort Lauderdale, FL

In Citrix Application Streaming, the "Application Hub" is the place where streaming profiles are stored.  The streaming profiler writes content to the Application Hub and the streaming client pulls content from the Application Hub at runtime.  Here's a picture of App Streaming high level infrastructure.  Focus on the box titled "File Server" circled in red, this is the "Application Hub".  It could also be a web server.

Calling the place where streaming profiles are stored the "Application Hub" is great, it describes the concept and makes it clear that this is the place where streaming content is stored, in the hub!  Marketing folks LOVE IT.  Programmer folks are underwealmed, its a file server.

What gets obscured by the fancy title is that there is ZERO Citrix code running on that server.  This was a design goal from the beginning and we have worked hard to keep that pronciple in place - no Citrix code on the server!   This done both to reduce the number of places we install stuff, but also to follow the model of "keeping it simple".  Customers already have servers, don't ask them to put yet another protocol on their network.

The Application Hub stores your applications, but it is YOUR hub.  Any vanilla file server will do or web server.  It doesn't matter if its SMB, CIFS, Samba, Novell or HTTP, HTTPS, Apache or IIS.  The streaming client will access the content either via UNC based file opens or via a HTTP/HTTPS "get".  That it.  No extra magic.

Now - you can use your Application Hub (server) to control access to content.  Profiles are on-purpose stored in directories of their own.  Notice that you have a top level directory that holds all your profiles and that you have subdirectories for applications (profiles) below that space.  The streaming profiler absolutely insists that this structure be maintained when profiles are stored.  Why?  So you can DACL protect the entire profile with rights assigned to a single directory.  Its low budget, but it works and administrators are already good at controlling things like this.  Relying on the network/web infrastructure that is already in place makes it easy for the Application Hub to implement things like controlled access, again with the Citrix dev team not having to write any code.  This makes my life happy and I hope this description of the Application Hub takes away the mystery.  Remember though, it is the very grand and glorious "Application Hub" that provides the content for Application Streaming!

Joe Nord

Product Architect - Application Streaming and User Profile Manager

Citrix Systems, Fort Lauderdale, FL