I get this question less often recently than I used to but it still shows up.  

• Can Application Streaming be ported to Apple Mac or Linux? 

The question is usually based on the idea of wanting to run XenApp published streamed applications in an isolation system on the foreign operating system.  That is, to bring streamed Windows applications to the other system. 

You can insert your favorite operating system on the above list, but the answer remains the same, no.

APPLICATION ISOLATION is about changing things and lying to applications so that they think they are doing one thing when they are really doing another.  Fundamentally though, the executed application is still a "native" application for the operating system.  The executed Windows based application is still a Windows based application and it will not run unless something exists below to satisfy the Windows APIs.  The application won't even load unless the Windows loader brings it into memory.

Can you use App Streaming on Mac?  SURE! 

Insert your favorite MACHINE virtualization system such as Parallels, install Windows into the virtual machine, install the streaming client (aka: offline plug-in) and then run all the applications streamed that you want.  This works fine! 

Is it "streaming" to the Mac?  No! 

I see people around Citrix doing this all the time.  They run streamed MS Outlook 2007 and happily check their email and do many things of their job, all day long with lots of apps.  Many of them spend most of their day inside the Windows environment of the Mac machine.

In this usage, I call the MAC the ...

• THE WORLD'S LARGEST WINDOWS LOADER!

For the non programmers in the room, the "loader" is the component of the operating system that is responsible for bringing the operating system to life.  The quick version goes something like this:

The machine powers up and and a whole bunch of things happen, but eventually the hardware kicks off the machine loader from ROM in "real mode" at address CS:IP FFFF:0000, this kicks starts the BIOS.  The BIOS h has the job of finding a 512 byte sector of disk, loading it into memory and "jumping" to it.  From the BIOS perspective, at this point the machine is "booted".  The 512 byte initial loader, brings in a bigger loader, which brings in a bit more, which brings in a primitive part of the operating system, which brings in some "boot" device drivers such as "disk" boot load device drivers, which brings in more of the operating system, which loads more device drivers, like NTFS, enables paging and does a bunch more stuff until you eventually get a machine, running and ready to do useful work.  You can make a career out of any of these activities.

In my mac example without machine isolation, the Mac must boot first and once it's done, it loads the virtual machine thingie which "powers on" the x86 box, which does a bunch of things, which then runs from "ROM", which is really "RAM" and jumpts to a "real mode" address FFFF:0000 and then boots the Windows machine.

This continues on until the Windows box is ready to do work => ergo, the Mac is the worlds largest Windows loader.  While boot sequences are fun, I am way off topic.  

Can you run App Streaming based apps on a non-Windows platform?

Answer the question with a question:

• Can you run WINDOWS based applications on a non-Windows platform?  Answer no.

Sometimes this answer receives a follow up: Have you considered adding this capability?

Now, a white-board is needed.  We use a white-board because nobody has chalk-boards anymore.  Frankly, I prefer the old style because they could be readily and reliably erased, but I'm digressing away from the topic.

How much slower does a streamed app run compared to a locally installed app?

Answer: They are the same!  CPU wise, it's the same.  A process is a process is a process and program code is program code.  The isolated app runs NATIVE on the machine.  It is loaded by Windows and the app uses Windows to do things that apps do with Windows. 

Eventually, the program may call a Windows API, such as RegOpenKeyEx or CreateFile.  When this happens, the program execution takes a brief side journey through the isolation system where the parameters to the API are "adjusted" to make the application run inside of an "isolated environment".  This is how the layers of glass are implemented.

The application is still an application and it is still dependent on the Windows machine for running the application.  Things do get a bit more complicated because even DOS apps running on the Windows machine can be isolated (link), but fundamentally, Application Isolation "adjusts" the execution of applications that are running native on the Windows machine.

Finally, the question can be answered: You can't run "isolated" Windows apps on a non-Windows machine, so there is no point is worrying about running App Streaming under MAC or Linux or others.

What about App Streaming to Windows XP Embedded?

Sure, that will work and this has been done. 

What about App Streaming to Linux under Wine?

Sounds like an interesting activity.  I'm quite sure it won't work, but there could be other neat things.

Enjoy!

Joe Nord

Citrix Product Architect - Application Streaming and User Profile Manager

As the Citrix Architect of Application Streaming AND Architect of Citrix Profile Manager, you might infer that I'm interested in leveraging one technology to help the other. 

Background on roaming profiles and Citrix Profile Manager

First, background on Windows "roaming profiles" and similar.  Consider that when a user logs onto a machine, the logon activity must "roam" or "copy" the network stored version of the user's profile onto the execution machine.  In the general sense, everything on disk beneath %USERPROFILE% or C:\Users\usename, will be copied onto the execution machine at logon and then copied back to central store at logoff. 

During logon, this is a "large" consumer of logon time where it consumes perhaps the largest portion of the overall logon clock.  With roaming profiles, this full copy happens every time, but with efficient systems such as Citrix Profile Manager, the "copy" is actually a "sync", so the copy happens really fast and the copy back is limited to only the files that changed.   While this also speeds logoff time, let's stick with the value of logon time because ... nobody cares how long it takes to logoff.

Where all of this stuff gets more interesting is when you consider a user logging on to XenApp hosted session or logging onto a hosted XenDesktop session where a common disk image is used for the base operating system.  Notice that in each of these hosted cases, the user's profile on the execution machine is initially "empty" and it will be initially "empty" on every logon.  This means that the glorious logon sync that the Citrix Profile Manager does at logon will actually be a "full copy" and here, it starts to behave with the same inefficiency as the base operating system profile solution because it will be a full copy at EVERY logon.  We like to do better than this.

For a more detailed introduction to Citrix Profile Manager, consult this Sepago white paper.  Recall that Citrix Profile Manager is based upon the Sepago Profile technology that Citrix acquired some time back.

Use "streaming" to solve profile population
Logical move: Instead of copying stuff onto the machine at logon, use isolation technology to LIE to the system to tell it everything is copied local when it is really still on the central store.  Eventually, when the system or an application references stuff in the user profile, go fetch it and make it present.  This is "just in time" population and it has the promise to greatly improve logon time in a hosted environment.

For JUST IN TIME population, the bet goes, some large portion of the user profile will never be referenced, so you save big on the logon speed and you save big on the runtime because much of what exists in the user profile will NEVER be copied to/from the central store.  This means that using a just in time profile solution will save LOTS of time for logon, and this is a great benefit!

Great - How much quicker?

The answer: LOTS QUICKER!

Yes, but do you have a number?

I'd like quote: Just in time Profile Manager speeds XenApp logon by 100%  

My gut says that the number is closer to 40% - 50%, but I don't have any hard evidence and thus the premise of this blog post...

Getting a "number" is harder because the answer is that "it depends".  Marketing people and customers prefer hard integers.  The integer number is hard to dream up because the answer depends on the size of the user's profile and the efficiency of network activity to/from the central profile store to the execution physical machine or virtual machine.  The BIGGER the profile, the more efficient.  If the profile is zero size, then JIT doesn't do anything and if the profile size if infinite the the JIT logon benefit is also without limit.

So, the answer for the logon value of just in time is is somewhere between a 100% benefit and 0%.  This doesn't help.

Let's go with an example:  The profile on my primary computer is 11GB, yes Gigabytes.   I could be a rare case.  This is pretty close to "infinite" so I will save plenty in an average logon.  

It turns out that 10 GB of my 11 GB profile is a TrueCrypt encrypted hard disk container.  I'm sure glad I'm not copying that down from a central store on each logon!  In a hosted VDI, I would be.  Technically, I'd store stuff differently, but in concept I'd be copying this down.  In a hosted XenApp execution with just in time, I would never copy down this file so Joe's benefit of just in time will be either 0% or 100% and nothing in the middle.  This still isn't helping me come up with a number.

For my normal machine, I am not connected to profile manager or roaming solution or even to a domain so my system may not be the perfect example.   As XenDesktop becomes more and more prevelant though, the strange things that users do to populate their user profile will make examples of users doing stupid things like placing 10GB files into the user profile more and more common.

If you are using the same profile for the primary hosted desktop as well as numerous XenApp server based app executions, you experience the victory!  Only ONE of them will be accessing that really big file.

In my case, the primary machine will access the really big file, but all the "vacation request" and similar applications that I run will run on another computer, where the really big file will never be referenced.  Using just in time population of the user profile, the majority of my logons and I'll say that ALL of my quick in/out sessions will have a HUGE benefit to not copying down that 10GB file!  This will make my logon time benefit near 100% on these other sesions and near 0% on the machine where I do access that single file that is 90% of my user profile!  

It is much better to quote percentages on something like this, so the time saved will be some percentage of the overall logon time and the LARGER the user profile, the HIGHER the savings!  Okay, we're getting closer.

Right - what's the number to quote?

Let's start with a formula:

• TimeSaved = TotalTimeWithouJIT - TotalTimeWithJIT;
• PercentFaster = (TimeSaved / TotalTimeWithoutJIT) * 100%;
  

How to calculate "TotalTime"?  This number will be the sum of the entire logon, nobody cares how much more efficient the roaming profile copy is, they want to know how many SECONDS this will save on logon time and how much of a percentage faster the logon time is. 

This requires breaking down the logon time of a "NORMAL" logon.  What is a "normal" logon?

Need to have: Computers that are representative of a "normal IT shop".  Need networks that are also representative of "normal world" and network servers and end user machiens that are "normal".  Must simulate some kind of load on these machines or just take it as a given that the load during the test will be similar to all the other stuff going on with the test network at the time of the measurement.

The key ingredients are:

1. Size of the user profile.
2. Speed of the network.
3. Overall logon time 
4. Logon time used to copy the full user profile

Given the above, we tigger the measurement to figure out how much time is profile population and poof!  Take the total logon time, subtract out the portion spent copying the user profile without JIT and ... We have a number!

What's that number again?

What is the SIZE of an "average" user profile?  What is the average file size?  How many files are "normal". 

Do normal users have giant files inside their user profile?  Yes, they do!  If you have have you ever copied a .MPG file or .MP3 onto your desktop, then you're as guilty as I am.  The PROFILE WILL GROW and will be large.

How large?

We need to exclude some files.  What about the files that will NEVER copy onto the execution machine even ignoring just in time.   Some stuff like "My Documents" will not be roamed, but will instead be accessed straight off the network via folder redirection.  This is "standard procedure" for setting up profile environments and here, "just in time" doesn't have any effect.

Let's get to statistcs.

Start with the initial 11GB and take out that 10GB file that is an anomaly and I'm left with 390MB.  The missing 610 MB is round off error.

Administrators usually redirect "My Documents".  Take out Joe's "My Documents" = 208,055,865 bytes and I'm left with 182,450,081 bytes.

Okay, I wonder what I have inside my USERPROFILE that could possibly constitute 182MB?   Dig deeper.  I have 24 MB of pictures!  While I am sure that they are lovely - I am also sure that I haven't looked at them in months.  If I were "server side" my admin would probably redirect "My Pictures" too.  Now I'm down to 158MB.

Keep looking....  BING BING BING BING BING!!  We have a winner.  I have 149MB of "Downloads".

First - before anyone starts, "Downloads" have ZERO relation to the 24 MB of pictures!

Something is wrong here because after you subtract all this out and I'm down to 9MB of stuff that wouldn't normally be "redirected" and I KNOW that NTUSER.DAT on my machine is 8.9 MB.  This leaves me with 100KB of stuff that is candidate for JIT value.  There's a number breakdown here someplace, but let's keep it going.

Pretty soon it's obvious that I don't have ANYTHING in the user profile that matters.  I store it all in that huge the container file and in "other places" on the hard disk.  In a hosted case, these "others places" would find their way into the user profile, so all my utilities would be a plus for the profile.  Go looking...

What are "other places".

Utilities.  I have lots of them and store them off the root.  In a hosted desktop model, they will be in the user profile.  Add in 137 MB.  I have 77 MB of sound .wav files left over from my days of writing audio device drivers.  These would almost never be accessed, but they would live in my user profile.  Batch files.  They are kept separate from executable utilities, so add in another 9 MB and utilities and 33 MB of Windows SYMBOL files for debugging stuff.  137 + 9 + 77 + 33 = 256 MB of additional stuff for the user profile.

I love it when numbers come out to a power of 2!

One number:  "Average" user profile size is 256MB!

Yes, I left the 10GB file out of this mix.  That quantity of storage just has to kind of go away from the calculation.   I hear numbers of 20-30 seconds of XenApp logon time being required for copying down user profile content?  If we can make this number be "zero", then there can be real value in just in time profile solutions.

Add in some stuff that would be moved from my container file onto the user profile and I propose that the real size could easily double. 

Joe's proposal: The Average size of user profile is 512MB!

If any of this math makes sense, then I have an example number set that can be used to construct a measurement.  Is 256MB the right number?  Is 512MB the right number?  How about 1GB?

Real world statistics are the elusive number.  If you happen to have a couple hundred profiles representing a years worth of regular hosted desktop usage and wouldn't mind sharing, please send me an email or comment below.  

THANKS.

Joe Nord
Product Architect of Application Streaming, Profile Manager and a few side projects
Citrix Systems - Fort Lauderdale, FL

I have heard some rumors of the production level App Streaming service (radesvc.exe) dying at runtime.  In the reported failure, the administrator has configured the service for automatic restart to work past the issue and I have suggested that this is only masking the problem, don't do that!  The streaming service, like most NT services, should never die and I'd much rather cure the root cause than work around the issue. 

The realities of "real users" and "production use" sometimes necessitate doing things that aren't ideal in a theoretical sense so this advice cannot always be followed, which brings us to this post where I will bring vision to the perils and values of configuring the streaming service for automatic restart.

Put your FSFD programmer hat on

You wear this hat when you're writing kernel mode code.  You write the file system filter code for the App Streaming isolation system and this code has two primary purposes; file system filtering and process monitor for sandbox management.

As a FSFD writer, you are never allowed to die or the entire machine will turn blue.  Today's post is not about kernel mode things dying, its about application level things dying.

Put your NT Service programmer hat on

You wear this hat, you think you're powerful because you run with "higher privilege"; higher than mere apps.  You may even be considered part of the "system", but from the perspective of the kernel code, you're a mere app too and as a class, all of you are untrustworthy. When a service dies, the machine does not turn blue, but it is still bad!

What does the service do?
Among other things, it is responsible for launching all isolation sandboxes and placing applications into the sandbox for execution.  Here's a chart that brings some color to this description.  What isn't drawn in the below is that the service talks to the FSFD to define sandboxes and launch applications into sandboxes.

What does the File System Filter Driver do

The FSFD hangs out and implements file system redirection - the layers of glass for the file system.  It is also responsible for managing which applications are in the isolation spaces; yes, that's plural on purpose. On a given machine, especially on a XenApp server, the FSFD can easily be tracking 500 isolation spaces.  Consider that there is state data for each of these.  It isn't large, but it exists and the code that keeps track of this actually does it in a balanced binary tree, which seems like overkill until you get large number of isolation spaces.

In the service, you also have state data for each sandbox.  Here though the state data is allocated per-thread.  Put differently, each sandbox gets a thread and this thread and only this thread is used for communication with the kernel mode code.  In this way, a few things are achieved.

1. The streaming service doesn't have to have complicated logic to manage its sandbox state
2. The kernel code can gate who it's willing to talk to based on the thread of the creator
3. When the FSFD has work for the service to do, the service "always" wakes up in the right state.

For computer science stuff, these are all positive actions. 

The negative actions

The service isn't supposed to die without a graceful shutdown and it should only close gracefully if it isn't managing any sandboxes.  In practice, "non scheduled" terminate happens all the time during development and recent reports show, it can also happen during production. 

The FSFD tolerates service death.  Why?  Primarily it does this because it doesn't have any other choice. 

If the service dies, the kernel code, being all powerful isn't surprised by this action - it "observes" that the service has died, but there isn't a whole bunch it can do about it.

Consider an example

You have isolated applications up.  Let's say you have 10 of them running, from 5 different profiles.  This means that you have 10 applications running in 5 different sandboxes.

The service dies...

The applications are still running, but they have lost their support network.

Let's say that the application now issues a DIRECTORY ENUMERATION on stuff in the isolated space.  Normally, the FSFD gathers information from the service to satisfy this request.  This is how the FSFD "LIES" to the application to tell it that things are present that aren't really present.   In this case though, the service is "gone", so what does the FSFD do?  Answer: It does the best it can and "falls back" to AIE style N layer directory merge.  The directory enumeration is satisfied, but the files that are there via a lie will not be included in the directory enumeration results?  What effect does this have on the application?  Don't know - depends on the app, but in general the results are bad.  

If the application issues a file open, you'll satisfy it based on the things you can answer without the help of the streaming service.  This means that if the file is really present in the cache, the file open will succeed and if it isn't, it won't, or execution will drop down to a lower layer in the layers of glass to answer the file operation.

Will this work for the application?  Maybe.  Ideally, you'd like to terminate the applications, but terminating applications when users have stuff running and haven't saved their work is considered bad form.

New sandboxes are launched

Recall that new sandboxes cannot be created without the help of the streaming service, so here it is a given that the service has been restarted.  When the service loads, it contacts the FSFD to register itself.  The kernel code says "nice to have you back" - but there isn't a dag gone thing it can do to help the orphaned sandboxes from the previous run of the service.  All the "app level" state data is "gone" and there's no way to put it back together again.

New launches though can be handled.  When created, the FSFD notes who the service is and will communicate with this "new" instance of the streaming service to manage the "new" sandboxes.

During development this is cool!

When developing the code, if you are the NT Service writer, this is really really cool because you can write code, debug it, terminate the debugger (which unloads the service), change the code, compile it again, run it (which loads the service) and the FSFD will just plain deal with all of this.  Very fast for development; no reboots needed and you can even do all this stuff from a visual development environment like MS Visual Studio.

During PRODUCTION this is not as cool!

Being willing to take on new sandboxes means that auto-restarting the service can seem like a good idea.  The thing this overlooks is that the orphaned sandboxes are, well they don't have their support network and without the streaming service, directory enumeration and file opens are not going to occur correctly unless the streaming cache is completely full.

Put your ADMINISTRATOR hat on

What should you do?  Answer: Treat death of the streaming service with caring detail.  It should be investigated and fixed.  The Citrix support team will love this - Joe said we should report service death rather than restarting the service.  My response, the service should not DIE unless you kill it!  I'm pretty sure service team already has the report, so I'm really writing for the next person and hopefully by the time you read this, we'll already have it fixed....

How to work around.

Above said, if you get in this situation, run one app from each profile with "-e" populated RadeRunSwitches.  This will fully populate the streaming cache and will minimize cases where the application will fail a file open or directory enumeration.  Next - Turn "-e" off as it will command a full extract on EVERY App Launch and you don't want that.  Next step - get the service fixed.  In the mean time, you can auto-restart the service to get new sandboxes created, but just be sure you aren't using the auto-restart to hide a problem that really needs to be investigated.

Before people ask, I already have feelers out to the people that have seen the service die.  Hate to have this happen with production code, but the correct answer is to research the problem and make the fix.  Hopefully readers of this post will appreciate the open nature to acknowledge a bug that isn't widely seen.

Joe Nord
Citrix Systems Product Architect - Application Streaming.

Administrators are used to the idea, that running applications under Application Streaming will permit poorly written applications to run in a multi-user terminal services environment.   For example, if the application wants to write to the \Windows directory, no problem; the application will believe that it wrote there and later if it reads the same stuff, it will see what it put there and generally, the application will work. What is less known is that that Application Streaming and XenApp publishing can be used to reduce the rights of the application at execution so that it has a reduced chance of hurting the machine.

Privilege vs. Isolation

Isolation and "privilege" are different things. Running the application "isolated" does not mean that the application can't do powerful things.   An administrator privilege ISOLATED application CAN still perform privileged operations such as adding new users to the machine, marking them as administrators and adding them to the remote desktop group where the evil doer can then remotely login, as a non-isolated administrator and easily do evil things. 

Not a problem for XenApp hosted execution

To be clear, none of this is important for XenApp hosted execution.  Here, the user is already a user and stripping power from the user to get them to user power is a "nop" because they were a "user" to start with.  This discussion of "privilege" reduction is more of a Windows XP client side, or hosted desktop statement where "admin" power users are the norm.   On Windows XP, unless you're very good at locking down the machine the end user will be running as an "Administrator" and this is not desired.  How can you make this happen as little as possible?  How can you get MOST of the applications to run with the least privilege possible?

Brain damaged applications

Some applications even CHECK to see if they are admins and refuse to run if they are not.   Awesome!  If you can't figure out how to code it, demand admin rights machine wide!    You can easily hit a situation where 90% of your desktop applications will run fine without admin rights, yet you have no choice but to make the user a full blown administrator because some small subset of the applications demand admin rights; or perhaps, even really need them.

What about the "normal" applications that don't need admin rights, or at least don't need admin rights when run under isolation?  It would sure help if we could at least make the "all powerful" user be a "lowly user" for the purposes of the majority of application execution, even if the user is really an administrator.  You can, and XenApp makes this easy.  First, some history.

DropMyRigthts

Go back in time and take a look at this 2006 technet article from Microsoft on Least User Access and a description of the DropMyRights utility by Michael Howard.   Excellent stuff and here is a related set of blogs from Aaron Margolis of Microsoft who seems to have a passion for running apps as a user!   The output of this early work was a command line utility called DropMyRights which would duplicate the user's logon token, strip the powerful rights - and then use the modified token to launch the application.  Good stuff.  As an example, here is the .BAT file I used to use to launch MS Outlook.

• dropmyrights "%PROGFILES%\Microsoft Office\OFFICE11\OUTLOOK.EXE"

The idea of running apps on forced user privilege on Windows XP was not unique to App Streaming, but we did wrap pretty GUI around it and wrapped application publishing around it to make it easy to use - and then we didn't tell anyone it was there.  To be fair, most of the usage was server side, so it wasn't as important, but hosted desktops are changing this.

The XenApp publishing system makes this dropping of user rights accessible via easy to use GUI.

Access Management Console

Here's the AMC screen that controls this setting.  Notice that this "stripping of rights" is controlled in the AMC - not in the streaming profiler.  Could it be controlled in the profiler?   Sure.  Both of these tools are nice GUIs which could accomplish the same goal, so yes, it could be controlled in the profiler, but it isn't.  One could even make a really good argument that it is in the wrong place and SHOULD be in the profiler because this is where the admin is that knows more about the application.  I would agree, but it wouldn't matter, it's still in the publishing console whether or not this seems like the right place.  

 
When I wrote the draft for this post, I did it in a place without internet access, so I couldn't easily check the default.  I wrote that SURELY! the default is that we strip the rights before launching the app.  Surely, Shirley, what ever you call it, the default is the other way; by default, the launch leaves the user's token alone and launches the app using what ever power the user has according to logon.  If you CHECK the box, then the Access Management Console tells the Citrix IMA to tell the Citrix Web Interface to tell PNAgent to tell the Streaming Client that it should strip power from the user for the purposes of running this stream to client application.  Where the application will permit it, You should set the checkbox.

XenApp server side, it won't change anything;XenApp Client side, it will ensure that the application is launched using a user token that has "lower power".  Lower power is better...

Here are some other writings on Application Streaming related to this:

•  Enhancing the Security of Application Streamingfor Desktops

Enjoy!

Joe Nord

Citrix Systems Product Architect - Application Streaming

One of the first screens you will see in the Streaming Profiler wizard is a screen about "Enable User Updates" or in the earlier profilers, this was called "Enhanced security" or "Relaxed security".   Wow!  Mysterious terms!  The first thing we do in the profiler is hit the admin with a question that they don't know the answer to.  Hum.

Steps:

1. Describe the panels
2. Describe what the settings do
3. Examples of how this effects application execution
4. Guidance on how to configure the setting

Here's the panel in the streaming profiler version 5.2 (XenApp 5 Feature Pack 2):  Hot off the presses, released GA to the web download last night.

Here's the same panel in the previous streaming profiler (1.3)

What does this setting do?

Under the profiler, it doesn't do a whole lot.  It just sets a BOOLEAN that accompanies the streaming profile.  You can see via nice visual form in this streaming profiler, but if you dig down, you'll find that all this does is set a boolean in the profile XML data; at the PROFILE layer.   Changing this setting actually does more work, but I'll get to that in a minute.

Going back to the Layers of Glass, there are conceptually 3 layers of isolation.  Here's an abbreviated version.

At runtime, the applications in the isolation sandbox see a multi-layer merge of the true machine at the bottom, masked by the installation image and at the top, a per-user layer.  The per-user layer is seen "first", followed by the lower layers of isolation and finally the true disk or true registry of the machine.

The normal action is that the machine starts out pretty much clean, the streaming profiler captures the installation activity of an "installer" that writes stuff to the file system and registry.  These are packaged up to become the "blue" layer above, the installation image.  

At end user execution, the installation image is laid down on top of the execution machine and as far as the isolated applications are concerned, they are installed.  It's all a lie - they aren't really installed.

The top layer is initially "clear" or "blank".  As the programs run, they may store documents and similar, but these would generally not be in isolated space, so they don't really show up in this picture.  The application though may WRITE things to "off-limits" locations which would be caught by the isolation system and end up with storage of stuff to the per-user layer of isolation.  These land in the top layer of the isolation stack which is set up as one per-user.  This is what allows ill-behaved application to run happily under isolation on a multi-user machine when they won't happily run without isolation.  As an example, consider an application that stores settings to the program installation directory in a .INI file.  Under isolation, this will be captured and land in per user space and the application becomes runnable in a XenApp Terminal Services world where otherwise it would not work successfully.

Back to this post

If the application updates itself at runtime, the update will land in the per-user layer of isolation and this is bad.  Standard procedure when profiling application installations is to TURN OFF all automatic updates.  The application should not update itself - this should only be done in the profiling scenario where the administrator commands the update.  Recall that the isolation space is ONE and the per-user space are MANY, so we only want application content to be updated in a single place.

What does "Enable User Updates" do?

If the user downloads application updates such as .DLL updates or .EXE updates, should this be permitted?

The general answer is "NO!".  Some administrators may have a scenario where this is desired.  The common ones are users wishing to install their own plugins for isolated web browsers or install their own addons for things like Microsoft Office.

How does it work?

Put your file system filter driver writer hat on.  For isolated applications, EVERY TIME the application opens a file or tries to open a file, you get first look.  If the file open is for executable content, should this be permitted?  If "enable user updates" is "off", then file opens for RUNNING executable content from the user layer will be denied.  

The neat part here is that the isolation system distinguishes this behavior based on WHO the caller is.

If the caller is vanilla application wanting to read or write content, no problem - do what you want.  If the caller is the Windows LOADER, then this setting comes into play.  If the LOADER is trying to load executable content from the per-user layer of isolation, then the isolation system can be told to FAIL that operation, and this is what this setting controls.  Pretty neato.

One headache

The setting while stored as a profile level single property (a boolean) is implemented in the isolation system as an attribute of EACH of the isolation rules for EACH execution target of the profile.  If you set the profile level property, the streaming profiler must modify the isolation rules (many) for each Target of the profile.  This means that if you have a profile with 4 execution targets and you're editing one of them - and you set the profile level property, behind the scenes, the profiler brings the other 3 execution targets into "edit state" to make the change and will eventually write all 4 targets back to the application hub.  Going to edit state to change the rules requires unzip of the can file from the network server onto the profiler machine.  If the profile/targets are large, this can be a very painful operation to accomplish a single boolean set, but this is how it is.  If you make this change, be aware of the large behind the scenes work that the profiler is doing.  Grummble yell a bit and then it will be done.

Fun with streaming - Great entertainment in isolation circles

Turn on the -x RadeRunSwitch so you can an isolated command prompt when you launch your next favorite streamed application.  This assumes you have user updates disabled, which is the default.

cd c:\windows\system32
c:\Windows\System32>notepad.exe
< it runs >

c:\Windows\System32>type notepad.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>copy notepad.exe n.exe
        1 file(s) copied.
< file copy was successful - n.exe is at the per-user layer of isolation >

c:\Windows\System32>type n.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>n.exe
The system cannot find the file c:\Windows\System32\n.exe.

FIREWORKS HERE!

The isolation system LIED to the Windows Loader - returning ERROR_FILE_NOT_FOUND (2) rather than completing the loaders request to run this file from user layer of isolation.  This is what this setting does!

But wait, there's more!

c:\Windows\System32>copy n.exe notepad.exe
        1 file(s) copied.

c:\Windows\System32>notepad.exe
< it runs!! >

Why does notepad.exe succeed in the final case?  Easy, there are two notepad.exes.  At the per-user layer, there's a notepad.exe which was written on the file copy from n.exe.  We don't care what this file is, but it is executable content and it exists at the per-user layer of isolation and therefore it doesn't exist for purposes of running programs.

Since the "Enable user updates" setting is set to disable user updates, executable content at the per-user layer of isolation does not exist from the perspective of the Windows loader.  BUT - at the physical layer, there does exist a file with that name and this can satisfy the file open, without violating the isolation rules.  There could also be a file with that name at the application installation image layer.  In this example there wasn't, but there could be.  The isolation system starts at the top and goes down until it finds a hit.  If "Enhanced security" is enabled, then the per-user layer is "off-limits" for execution of executable content.

The grand result

The application "update" applied by the user may have been applied as far as the user or application is concerned, but in reality, it was not applied.  The version of the application that is running is the version that the administrator profiled.  Pretty cool stuff.

Why did we rename the setting?

Putting "security" in the title implies that this will somehow prevent users from doing things to run content that they download and this is not what it does.  If the program updates itself, then this setting will block that content from being executed.  The setting can also block user installed additions to the program (plugins), depending on the location to which they were installed - was it included as an isolation rule during profiling?

Take a web browser for example, if the user downloads executable updates to the browser, this will be captured and the user installed stuff won't run, but if the user downloads evil.exe and places it on their desktop, and then double clicks it - this will be outside of isolation so the layers here do not apply.   This is also true if the user downloads evil stuff to locations outside of isolation and launches it from the isolated application.  It will still run isolated, but it will run!  Describing this activity as "disable user updates" is more accurate than the previous words, so we've made the change.  I also hope that it removes confusion in the streaming profiler wizard.  "Enable user updates" is pretty easy to understand.

How should you create your profiles

1) Enable user updates should generally be "off".  Plugins are a rare need and where there is a real need for users to add plugins, start asking yourself if you can add those plugins at profiling to the common layer.  OR, if the use of user installed executable content is large, should this application be locally installed rather than isolated?

2) Always tell the application to NEVER update itself at runtime.

 A look to the future

Streaming dev team are discussing removing this option from a future release.  That is, "Enable user updates" will always be OFF.  I'm not sure of all the ramifications of this yet.  The question really is "how many admins are profiling their applications with user installed updates permitted"?  I hope the number is "few".

Joe Nord

Product Architect - Application Streaming

Citrix Systems

To use agent-less streaming or installed agent streaming; that is the question.

I have received inquiries recently - lots of inquiries on the same question actually - that all imply that "user mode" streaming and isolation is "better" than evil kernel driver streaming and isolation, because it doesn't have any kernel components.  This sounds exciting on first glance, but when you dig down the problem is more complex.  There are advantages to installed isolation systems and there are advantages to all user mode.  This blog shows some examples for both categories. 

To get it started, consider an isolation system's ability to load and isolate a Windows NT Service.  How can you isolate the execution of an NT Service if the isolation system itself is "user privilege".   At user rights, the isolation system lacks the privilege to start a service that isn't already installed, so it can't run and isolate services unless the service is already installed outside of isolation.  

Scalability is also a big concern.  It doesn't matter for a small system, but if you load the machine up with lots of users and lots of applications, having a separate copy of the installation system for each application becomes problematic.  

On the plus side for agent-less, it is really nice to NEVER need admin rights on the execution machine and this can help engage work from home or kiosk environments.    Another plus for agentless, the "attack surface" is smaller if there are fewer privileged components; this is the point that usually starts the conversation that "agentless is better".   Keep reading...

First, some definitions:

Agent-less streaming

Nothing is "installed" into the machine.  The isolation engine can run the profiled software without ANY components themselves being installed.  "Admin" rights are never required and the system usually embeds the isolation system and profiled application into a single .EXE image that is "executed" to run the isolation system and the isolated application.

Examples of agent-less isolation systems include ThinApp, InstallFree, XenoCode.

Installed agent streaming

With an "installed agent", parts of the isolation system are "privileged", they require an install time execution which usually includes the installation of at least one kernel mode device driver and generally also includes the installation of a NT Service that supervises the sandboxes that are active on the machine. 

Examples of installed agent isolation systems include Citrix Application Streaming and Microsoft App-V.  Even this is a simplification as the Citrix system uses kernel mode for process monitoring and file system filtering and USER MODE for registry filtering, which makes it kind of a hybrid.  App-V uses kernel mode for both.   The real key is that an installation step was required.  If you require an "install", you are a member of the installed agent camp!

Which is better?

The answer is simple: Mine! 

The real answer is that you have to ask what "better" means.  I mean, better at what?  Each of the systems have their advantages.

Consider installation

Not needing to install an agent is handy for some environments especially where the execution machine is not company managed.  USB Thumb Drives were the original player here; you can take you application anywhere!  By contrast, if you're really running in a company environment, either stream to client, running on a XenApp Server or even on a XenDesktop hosted world, the installation of an agent is not a gate - the admin controls the base image and can install any agents that they want.   For more on this, see "security" below.

Consider scalability

If you are running a heavily loaded XenApp Server.  Let's say 64-bit machines, with 64-bit Operating System, a handful of high end CPUs and enough RAM memory to run 100 concurrent users, with each user running 5 distinct isolated applications.   How many isolation spaces is that?  100 * 5 = 500 isolation spaces.  If the isolation system is 100MB, that's 500 * 100MB = 50GB of allocated virtual memory just to load the isolation engines.   I made these numbers up, but stick with me on the concept.

How many separate copies of the isolation system do you want to load?  Answer: One would be nice.  

With kernel mode or NT Service isolation systems, you'll have ONE single code space for all of the isolation sandboxes, a bit of instance memory for each sandbox and other than that, you're off and going.

This too is a over simplification.  For starters, the Citrix App Streaaming case is a hybrid.  I haven't checked the memory footprint lately, but the kernel pices are ONE and the registry and named object pieces are MANY as these are implemented inside the isolation space.  To get back on subject for this post, think of it as ONE installation system because all "installed agent" systems are in the same camp.

Does agentless mean that there is really a separate isolation engine for each application?  yes and no.   We tend to generalize this to say that the isolation engine is carried along with the to be run application.

If the agentless isolation system uses DLL references to get to the isolation engine, then the agentless system's memory load will be page table shared across the images.  This a function of the Windows NT PE Loader and memory manager.  With DLL load of the runtime, many pages of the 500 instances of the isolation system will be shared.   I don't know if the clientless systems use DLLs for the agent runtime, but if they don't, they could and this will share the load.  

Citrix App Streaming uses DLLs for the registry filtering, so the shared model here still applies (ONE).  

How to update the agent

If the isolation agent needs to be updated, how many application images do you have to update?  This is a plus and a minus for both again.  With installed agents, you update one and all the profiles/sequences benefit.

With agentless, you have to touch each of the isolation images.  Or, back to the DLL approach, you could achieve the same ONE update in agentless if you're using DLL based runtime.   

If go with the "one app, one executable to distribute" model, then the isolation system is not shared and the memory usage on a XenApp hosted model will totally suck.  Prototypes will be great, but actual performance under load will be a heavy hit to single server scalibiltiy.  When it comes time to update the isolation agent, you'll have to touch all of the profiles to get things updated; or you can skip this as a plus toward not having to maintain anything once you profile it.

Consider security

There is a perception that if kernel mode components are involved, then it's less secure.  This is mostly a statement of "attack surface".  There are many privileged components in the machine and they are all candidates for attack.  The real headache with privileged components is that they have to be "installed", but the security aspects still apply and are real.  

If you're installed and you have power, then you can do powerful things.  The corollary is that if you are not installed and you don't have power, then you CANNOT do powerful things and this is as much a plus as a minus.

Consider that many agent-less streaming systems receive "yes" check-boxes in comparison matrices when discussing isolation of NT Services.  

Either you're USER MODE or you're not user mode, you can't claim to be both.  

A step back: NT Services are installed applications, with no GUI and no direct user interaction.  Services run on either a powerful system defined account or a named user account of install time specified rights. "LOCAL_SYSTEM" and "LOCAL_SERVICE" are the common installation configurations and these equal "powerful".

In many cases, the reason that the programmer went through the pain to write a service is that they needed to do something with privilege and that was impossible from the user privilege application space.  This is why they are done in the service rather than in the application itself.

If the isolation system is agentless (user mode), and if that isolation system can load and isolate NT Services - and have that service work - then a boundary has been passed and the isolation system is no longer user mode.  Given that the agent itself would not be privileged, how can the applications that it runs be privileged?

Answer: The agentless streaming system requires the applicaiton services to be installed outside of isolation, and by installed, I mean REALLY INSTALLED.   This pretty much deletes the check-box for isolation of services.

Some NT Services likely CAN tollerate user mode execution under isolation, but for the general discussion, the answer will be that this breaks down and the service requires installation to the local machine - breaking the isolation boundary between the application and the local machine.
An "installed agent" isolation system CAN have the power to start services itself, and in this environment CAN run the service isolated.  This is only possible because the agent itself is INSTALLED and has power.  If the isolation agent is not installed, then the user cannot start isolated services and there's a pretty big gap in the claimed support of isolated services.  The services have to be outside of isolation and installed - they aren't isolated and the access to that service is not governed by the published application set.

Consider application launch performance 

Now that Application Streaming 5.2 client is out (real-soon), I can describe the greatness of 2nd time application launch.  If a sandbox is running that will support this app, there is no longer ANY need to create an isolation space for a second, third or fourth application launch.   Creating an isolation space/sandbox/bubble is an expensive operation.  The 5.2 client's ability to skip this expensive operation will provide great benefits in launch speed.  I describe this briefly here, but need to write more.   

Can agentless do things to equal this application launch performance?  Can it toss execution over the wall to already running isolation spaces?  Probably, but as an big executable based execution, by the time they run to make this decision, they will already have the isolation system mapped into memory and it's big and I'm betting, slow.  Bottom line, I'm looking foward to a new round of statistics with the 5.2 client - we should kick some major butt!  "Agent based" here has advantages, but this alone will not sway a discussion.

Consider central management
Both Citrix Application Streaming and Microsoft App-V are heavily predicated on the concept of communication with a back-end infrastructure and administrator driven management of the applications available to users and even preventing the execution of non-approved applications. 

The applications are published to users and in App-V case, users or machines and both systems communicate information back to the central authority to decribe the use of applications.   App-V can even block the launch of applications based on license metering.  All of this is enabled because of communication with back end, from an installed agent.  

Can that communication be done from an agentless system?  Probably.  Is this done?  I'm not sure.  If it is done, is that something you really want happening from a user privilege component?

What applications are available to the user and how can I trigger the update of application content?  What applications have been actively used across my whole organization and which ones are published that the users really don't care about?  What applications should I focus my support on and which ones should I deprovision without telling anyone?  Do I have enough application licenses?  

All of these questions can be answered with back end information.  Agent-based makes this easier.   Having a NT Service hanging out to collect this information and centrally report back statistics is an opportunity for central management, control and monitoring of applications.

Then again - agent-less can get most of this too. 

For example, Citrix EdgeSight monitors application usage on a machine and reports this back to make truly beautiful reports that tell the admin what has been happening on their machines.  It doesn't matter if the applications are isolated or not isolated, the EdgeSight monitoring system still sees them and can report on usage.   This happens for Citrix Application Streaming and can just as easily occur for clientless based isolation system.   

Conclusion 

Is agent-less better or is agent based better?  The answer really depends on how the whole system will be architected and what control the administrator has on where the agent will be installed.   Both have advantages.

Joe Nord

Product Architect of Citrix Application Streaming (An agent based isolation system)

Citrix Systems - Fort Lauderdale, FL

Have you ever looked into the App Streaming execution cache and said, "good grief!  What are all those GUIDs!"?  GUIDs are great, for the programmer.  For the administrator, they can be a bit more mysterious. 

I recently wrote a utility to decode the GUID data of the Application Streaming execution cache, displaying the decoded information to the console, but neater than TTY, it also updates the Windows Explorer display of that space so the "secret" information is exposed in permanent fashion and via the GUI, which is good stuff.   The utility is released to the Citrix Developer Network, in source code form so you can tweak it if you'd like. 

Download from here, look for the "Rade Cache Decode" link at the bottom of the page.

For background, I have written about the GUIDs before and how they are key to the streaming system maintaining the separation between "profiles" that you can publish and execution "targets" that the streaming system uses to actually use to run stuff.   There is more on this in this post.

Public credit

The idea for this tool actually came from very smart people at Sepago, Sascha Juch and Helge Klein, who I work with on Citrix Profile Management (the artist formerly known as Sepago Profile).   They also clarified for me that despite MSDN documentation claiming "system" attribute is necessary for this explorer magic to work, marking the directories read-only gets the job done.  Tested and true!  Thank you guys.

Update: Dieter Schmitz at Sepago is the gentleman who came up with the idea.  Thank you Dieter.

What does RadeCacheDecode do?

Application Streaming can store one or more execution Targets for each App Streaming profile. This level of indirection allows a single set of "Applications" and icons to be published even if more than one execution image is required.  This is common for a profile that supports both 32-bit and 64-bit operating systems. While the application is profiled twice to create two targets, there is only ONE profile to publish and only one set of "applications" from an Access Management Console perspective.  

A necessary consequence of this is that the execution targets have names that don't necessarily have any resemblance to the name of the profile from which they came.   Targets can be associated with execution systems for a variety of configurable options and these can change from version to version of the profile. 

That is, just because a specific target was right for Windows XP German yesterday doesn't mean that it won't also be right for English AND German tomorrow.   Even ignoring "one to many" advantages of execution targets to profiles, the streaming system STILL would use some abstract term to keep track of the execution targets. 

If you're editing a profile in the Streaming Profiler and you SAVE, all the identifiers of the execution units remain unchanged.  BUT - If you Save-As, you'll get new identifiers.   "By spec", the thing that was "saved as" is a New profile and will mature separately from the original - even if the before and after profiles have the same name.  Putting it a bit differently: Two profiles with the same name are not necessarily the same profile, the unique identifiers in the profile define the uniqueness of the execution units and in theory, or the profile itself.  The studious reader will see below where at least parts of this break down.The short version: Thanks to the use of GUIDs, the streaming cache manager will never confuse two execution targets, even if they come from different profiles that have the same name.

*What about the administrator?*GUIDs are there just to make the RadeCache space hard to digest.  After enough time looking at the Citrix internal showcase farm, you get used to the idea that the execution Target that starts with "1B3" is the MS Office 2007 profile execution target that is used for Windows Vista and Windows 7.  Type 1b3, hit tab and you get the right one.  Not ideal.  Here's what it looks like in the Windows explorer.

Okay, we know that 1B3* is the MS Office 2007 execution target for Windows Vista and Windows 7.  What are the other 5 execution targets? 

What you need is something to look up each GUID and to TELL YOU which profile it came from.  Yes, it would be BETTER if the streaming client itself had left you a clue, but it doesn't.  Notice that when the streaming client created these directories it KNEW what profile it came from and there can be at most ONE profile in the entire globe that has that GUID.  Did it leave you a hint, a small hint?  NOPE!.

Ahh, the fun life of being a XenApp adminstrator. 

HELP IS ON THE WAY!

The location of the application hub is generally "known" to the admin, but to know which GUID is associated with which profile, takes some work.  Once the running application terminates, not even the streaming client knows where the GUID came from.  It's just a directory that exists in the right place, but the link that says that the "1b3" target is somehow related to MS Office is gone, there's nothing there.  

Making it easier

For offline execution, the streaming system stores information that provides the secret decoder information required to convert GUIDs to profiles.  The Deploy space holds a copy of the Application Hub content and stores it in a place that has a fixed location (registry configurable and readable from the utility).  The RadeCacheDecode utility looks at the Deploy space, sees what execution targets exist for which profiles and uses this information to "decode" the GUID entries in the RadeCache.  This should make more sense by example.

Command line execution (readonly)

If RadeCacheDecode is run with the -r (readonly) switch, the utility processes the subdirectories of the RadeCache space, and for each tries to figure out what Deployed profile goes with that execution target.

Example output:RadeCacheDecode -r092b3450-e543-4541-837a-c374cc4e73cc_1 (GoView1067)

1b345768-25fd-45d4-b689-f3984f9221ee_5 (Office20071067)

274bd686-a305-45d9-a479-a127338586b1_1 (project20071067)

4d971270-cbac-4366-8c4a-b11f6f867a58_1 (AdditionalOffice1067)

76dc5736-0c5d-4600-a7e9-a3a41c3b5c51_6 (-not-deployed-)

a9984581-0e41-44b6-861f-b6d748dacb93_1 (Visio20071067)

Okay, we have now decoded the 4 of the 5 remaining execution Targets on my notebook.

Making it really useful!

Consider the technique that the Windows Explorer uses change each user's "Documents" directory into "My Documents".  This same activity can be used to permanently improve the readability of the RadeCache space. 

When thinking this was a good idea, I tried it out by manually creating files and while I was initially intrigued, my first conclusion was that it had negatives that outweighed the positives.  If you read the MSDN docs on the desktop.ini, it leads you to believe that the directories have to be marked "System" for this Desktop.ini stuff to work.  It isn't so.  Marking the directory readonly will do the job and with that, we have a solution!  Why is "system" bad?  Answer: it makes directories disappear from normal directory searches 

"readonly" works, so we have a solution.

This time, run the utility without the -r (readonly) switch and it will WRITE to the RadeCache space to mark all of it's subdirectories with "decoded" versions of the GUIDs.  The directories are marked readonly so the desktop.ini stuff comes into play with Windows Explorer and a file named "desktop.ini" is added to each execution Target's directory to create a "friendly name" to go with the mysterious true GUID_ver name.

The end result is that the next time the Windows Explorer browses to this space, you will see a nice version of the GUIDs - with the name of the profile placed in parens after the GUID. Sometimes it is necessary to force a refresh in Windows explorer to get it to reload the friendly names.

Here's a picture of the output

We're making good progress.  It is much easier to see these profile hinting entries than just plain GUIDs.

How does it work?

The entry above that says "not deployed" is the hint.  Since the Deploy space is at a known location, the decoder utility KNOWS where to look up the execution target GUIDs.  While the streaming client doesn't know what guid goes with what once the applications terminate, the DEPLOY space knows!  The Application Hub also knows, but unfortunately, the decoder utility doesn't know where THE (think plural) Application Hubs are located.  The Deploy space is fixed and given large offline streaming usage, this becomes pretty easy to inspect.

The utility loops through the subdirectory space of RadeCache.  For all directories that look like a GUID_v entry, it considers them an execution target and then goes looking for GUID_v content in the Deploy space.  When it finds a hit, the name of the containing directory in the Deploy space tells the decoder utility the name of the profile from which the GUID_v came.

Awesome!  Works good.

What's wrong with this picture

The studious reader will notice that the Deploy space is named based on the NAME of the profile that is deployed.  The Decode utility leverages this fact to figure out what profile a given GUID came from.

This naming in the Deploy space violates the premise that two profiles with the same name, but with different GUIDs are different profiles!  Dig into the internals of a .profile file (it is XML) and you will see that the profile also has a GUID.  GUIDs aren't just for guaranteeing uniqueness of the execution targets, they also guarantee the uniqueness of the profile layer.

Good news: the caching system is correctly using GUIDs and will "never" get confused.

The Deploy space SHOULD be done based on the GUID of the profile and all the sub-directories below the Deploy space SHOULD be GUIDs (with no version).  They SHOULD be, but they AREN'T and this means that the premise of two profiles with the same name, but different GUIDs are different is somewhat not precisely correct when considering offline content. 

In reality, this doesn't come up because admins tend to stick all of their profiles onto a single Application Hub and this means that the name of the profile is a unique key.  Still, know that this isn't right and know that I've been chasing a "fix" for this for over 2 years - so far, with no fruit.  The App Hub side is fine based on name.  The Deploy space should be GUID based.  Okay, I'm off my soapbox.

Enhancements

Good enhancements to RadeCacheDecode would expand the profile search to go beyond the Deploy space and also consult the network Application Hub.   The utility should also use the COM based SDK whose webpage it is posted in.  That is, the utility looks in subdirectories of the Deploy space to figure out which profile holds a file with the right name.  This breaks the separation of the profile as a programatic thing from looking behind the curtain.  I could have coded it that way, but it just seemed like a lot of work to open up a bunch of XML files to peek for GUIDs, when all I wanted to know was presence (is it there, or not?).

The "ultimate" enhancement to RadeCacheDecode is to change the streaming service itself so that when it creates RadeCache directories, it creates the desktop.ini file directly.  As of Streming Client version 5.2 (XenApp 5 Feature Pack 2), this does not happen - but now we know how...

To achieve almost the same thing, you could run RadeCacheDecode on a timer via some automated means.  It needs to be run with power so it can see and write to the RadeCache space, but if you do this, your desktop.ini entries should always be pretty much correct.  I coded the utility to WRITE only if the existing stuff is not "correct", so you can run it repeatably and the desktop.ini files will only be written occasionally.  This was probably overkill.

If you find it useful, let me know...

Joe Nord

Citrix Systems Product Architect - Application Streaming

The first releases of Citrix Application Streaming had isolation on the brain.  If you don't know what else to do, ISOLATE!  Don't let the app hit anything that might be important.  If you don't know if it's important or not, then it's important!  Protect it! 

This over protective behavior is being relaxed, the first step was in the XenApp XenApp 5.0 (Delaware) release where streaming client 1.2 was released which changed file system isolation to auto-ignore non-boot disk drives and soon, in the XenApp 5 Feature Pack 2 release where the version 5.2 streaming profiler and client change the default file system rule from "isolate" to "ignore".

Isolation rules review

When a user stores documents to disk under Application Streaming, the isolation system inspects the file operation to decide whether that operation should be isolated, ignored or redirected.  Details on each of these can be found here in the Citrix documentation library and here in my previous blogs.  All rules are named from the perspective of the streaming system, so "ignore" means don't change it - let it go to the file system without modification.

What isn't obvious, is what the isolation system does with disk volumes beyond the boot volume.  This behavior changed in streaming profiler version 1.2 (XenApp Delaware) so that the over paranoid behavior of the original Tarpon technology would instead leave users data alone.  Notice that I'm assuming users put data on drive D: while the OS is on drive C:.  This change has been beneficial to most customers and really helps engage the desktop, though I have seen a few that needed this to not occur and this oversight will soon be corrected to be backwardly consistent - for profiles created at the 1.1 level. 

To be clear, space beneath the user profile is and always has been "ignored", which means the normal places for users to store documents have no isolation collisions.  It's the non-normal places that require attention.

In App Streaming 5.2 which isn't out yet, the isolation will be further relaxed.

In addition to ignoring non-boot disk volumes, the isolation system will change it's default rule from "isolate" to "ignore".  Notice that this will happen only for NEW profiles.  If you have existing profiles, their behavior is already defined.  

Example user of Citrix showcase internal farm

Consider an example user, we will call him "Nabeel", because - well, that is his real name.  Nabeel is a Citrix executive and he travels a lot.  When he travels, he does lots of presentations and ... uses Citrix Application Streaming to run MS Office 2007.

In the example case, he was on a 3 week trip to Asia where he visited lots of Citrix sites and lots of customers.  With the magic of Application Streaming, he was able to use MS PowerPoint "offline" on the airplane to refine his presentations and also update the PPTs throughout the journey.  All of this worked super! 

Presentations updated all over the world, presented all over the world.  Everything was well received, and then he came home....

When he returned to Fort Lauderdale, he used Windows explorer to zip up all the files from the 3 weeks of presentations and e-mailed them out to the people he had met.  The receiving people noted that the presentations were mere skeletons.  CONTENT FREE if you will.  Everything worked on for many weeks was "as it was" 5 weeks earlier before he put Application Streaming onto the notebook.  OOOPS!

My phone rings...

What happened?  Answer: the files were all stored to the \Citrix2007 directory and since this is not a space that is "known safe", the isolation system isolated it.  From the view of the powerpoint application under isolation, it saw the correct and current version of the document.  But look at the real disk and the files were their original selves.   BAD.

Step 1: Where did the good files go?

That one is easy:

Consider the layers of glass.  The answer is that the file is in the top layer of the layers of isolation.  Find it in  %APPDATA%\Citrix\RadeCache\GUID\Device\C\Citrix2007

Step 2: How to make this not happen again...

Harder.  Insufficient answer is to add an ignore rule for Citrix2007 to the profile.  No good.

Not a problem for stream to server

Before going deeper, it should be said that this problem never comes up for stream to server.  In a server environment, "users" do not have the power to create directories off of the ROOT.  Users can store documents to %USERPROFILE% and that's about it.  Server side, this problem doesn't exist.

Stream to desktop

Client side, "users" tend to be "administrators" and this creates new problems.  "Users" like storing stuff to folders off of the root.  This makes it really easy to know what you need to backup.  

The isolation system not knowing how to handle the \Citrix2007 directory isolated the operations and this is far too over protective.

App Streaming 5.2 (Yellowtail)

In the upcoming release, the default file system rule for new profiles is changed from "isolate" to "ignore".  The default rule set then includes \Windows, \Program files and similar "important" directories as places that should be isolated at runtime.  I'll add that the profiling time rules and the execution time rules have to vary a bit to make this occur.  The streaming system takes care of this automatically.  Profiling time remains paranoid by default, isolating most everything.  Runtime becomes much more relaxed - making the isolated application execution more consistent in behavior with locally installed, while maintaining "protection" of spaces that the application should not be allowed to write per the layers of glass.

File system permissions still apply

If the user tries to write to a space that they aren't allowed, the file system permissions remain in place.  A user will only be able to write to or read from the real \Citrix2007 if they have DACLs in place that make this permissible.  This is a file system statement, not a isolation system.  In the "ignore" case, what happens is that the isolation system sends the I/O operation down to the file system without change and the file system will then decide if the user is allowed to access those files. 

This change greatly improves the offline streaming experience.

Joe Nord - Citrix Systems Product Architect - Application Streaming

Learn more about Citrix XenApp 5 Feature Pack 2

• Official Press Release - http://citrix.com/English/NE/news/news.asp?newsID=1857726
• XenApp 5 Feature Pack 2 Web Site - http://citrix.com/xenapp/featurepack2
• XenApp 5 Feature Pack 2 Executive Video - http://citrix.com/xenapp/fp2/video
• XenApp 5 Feature Pack 2 Webinar - http://citrix.com/xenapp/fp2/techtalk
• XenApp feature matrix by platform, version and edition - http://citrix.com/xenapp/comparativematrix
• XenApp Expert Series videos for this release - http://citrix.com/xenapp/fp2/expertseries
• XenApp 5 Feature Pack 2 Blogs- http://community.citrix.com/blogs/tag/xa5fp2
• Download XenApp technology previews - http://citrix.com/xenapp/techpreviews
• XenApp Product Page - http://citrix.com/xenapp/

Follow XenApp on | | |

I received an interesting inquiry recently.  Given my post on Never Deploy to XenApp Servers, how do you stop deploy from happening for hosted desktop?   Excellent question!

Here's the situation.  Happy XenApp Customer (they are all happy) who uses Application Streaming to deliver applications to their XenApp servers as well as stream to client, primarily for offline access.  The applications are published to the user as offline enabled and this works swell to trigger the deploy operation that is required to execute the application when disconnected from the corporate network and also works swell to not trigger Deploy in the XenApp hosted case.

Quick refresher on Deploy

The Deploy space is a COPY of all the needed stuff from the Application Hub, but it is local to the execution machine.  The Deploy space makes it possible for the streaming client to support application usage even when disconnected from the network.  The Deploy space is generally in \Program files\Citrix\Deploy and the execution cache space is generally in \Program files\Citrix\RadeCache. 

At runtime, the streaming client runtime fills the execution space as needed by copying content from the Application Hub - or if Deployed, from the Deploy space.  Other than one being remote and the other being local, the activity of the client is the same whether "offline" or "online".  Technically, "Deploy" enables streaming from one side of the hard disk to the other and this keeps life pretty simple from a programming standpoint.   I thought it up back in the Tarpon development and I'm still pretty happy with myself.

Deploy is automatically not done on XenApp Servers

It may seem like a good idea to Deploy to the XenApp Servers, but I have already squashed that myth and apparently did a good job there because the corollary, "Never Deploy to XenDesktop" has also been heard.

Back to the customer.  They have XenApp running good and are engaging XenDesktop.  Application Streaming is delivering the applications to the XenApp servers as well as to the end user machines (notebooks). 

Here's the thing that's not always obvious.  The streaming client KNOWS that Deploying to XenApp Servers is a bad idea.  If you publish an application to a bunch of users and offline enable the applications AND if you publish the applications to hosted execution, a single user can use both without triggering a Deploy on the XenApp server.  

If the user is running the application server side, the streaming client is still involved but it knows that it is running ON a XenApp Server, so it SKIPS the Deploy.  This is the right thing to do.  If the admin doesn't like that, they can still command a Deploy to occur on the server by directly calling the deploy utility, RadeDeploy.exe, but again, you shouldn't do that.

The problem

The customer is now engaging XenDesktop and have connected the hosted desktops to the App Streaming infrastructure from the existing XenApp configuration.  Here, the streaming client is running on the hosted desktop and ... when performing its app launch concludes that this is a client side execution and background copies down all the bits for offline execution. This is bad.

Here's a good write up on the percentage of space that is used for online vs. offline execution.

The point: Lots of disk writes start happening to copy the Deploy content down from the Application Hub to place them in the Deploy space where they will exist and get copied again into the RadeCache.  All of this is bad to the layers of cake.  We want to minimize disk WRITES.  Disk writes are backed up either by Xen/Other Virtual Machine manager or by Provisioning Services.  Either way, large writes by large numbers or users are bad.  Beyond being slow, these writes will occupy space in the per-user write back cache; which is discarded on logoff, so this whole thing will repeat on the next logon.  

The solution

The Deploy should not occur!  But it does.  In a "correct" world, the streaming client would be XenDestop aware as it is XenApp aware.  If running "hosted", don't Deploy!  We do not live in a correct world.  My initial guidance to the folks that asked me about this was "erase \program files\citrix\streaming client\radedeploy.exe".  All happy with myself because I KNOW that the streaming client shells to the utility to perform the deploy.

With the utility "missing", the streaming client will have no choice but to push on without the deploy.
Survey says.... didn't work. 

Instead of "pushing on", the streaming client declared failure and aborted the app launch.  Aargh.

Beautiful part of this is that I didn't have to actually try it.  Smart folks on the other side, all I had to do was keep coming up with ideas.  

Second round: Replace RadeDeploy.exe with a program that does nothing, but returns "success" return code.

Survey says.... Success!

The only utility I had handy to do this is one I wrote a very long time ago to use for remarking out RUN= statements from the Windows registry.  Source is included - please compile it yourself before putting in production.  For the programmers in the room, said program is "Hello world" minus the printf.  In the common image where the streaming client is installed, copy nop.exe to RadeDeploy.exe. 

Back to the Deploy activity

The streaming client "shells" to RadeDeploy.exe which is really "nop.exe", which returns 0 as RC.  With the Deploy "successful", the launch proceeds with no error, AND the Deploy never happens.  Perfect!

I see a future where the streaming client will be XenDesktop aware and skip the Deploy.  Until then, this technique may prove useful.  If you have other creative solutions to this problem, please share them here.

Next step is getting the RadeCache space available to everyone from a single source, read-only and fully populated at the get-go.

Joe Nord
Product Architect Application Streaming

Citrix Systems

The Citrix Application Streaming client and profiler version 5.2 are available in Beta form.  Here is a link.    MyCitrix credentials required to download.  The Streaming Client is "Offline Plug-in version 5.2". The Streaming Profiler is "Streaming Profiler version 5.2".  

These are not production, but they are pretty close to what will ship with the next XenApp release.  I note that the Streaming Client/Profiler 5.2 can be used with everything from Presentation Server 4.5 to XenApp next release.

Please give it a whirl.  The normal tech support forums should be used to share your positive experiences.

The 5.2 client will "consume" profiles created by all previous streaming profilers and the performance improvements noted below will be seen with no need to reprofile.  That said, there has been a years worth of changes in the profiler, so there can be other reasons to move up.

Things you will see:

• Second time application performance vastly improved.  Feature: "Sandbox reuse".  In the past, launching two applications from a single profile actually created two sandboxes even though those sandboxes could be considered one as they both had the same definition of isolation scope.  Starting a sandbox is a long activity, that is now skipped for second application launches, making second app launch very similar in time to locally installed.  This change was the focus of the release and should produce very happy users. 
• Sandboxes are left alive for 5 minutes after the last application terminates, improving odds for 2nd time launch compared to first.  Registry configurable period, in minutes.  Might change this to Ms.
• Integration with the Citrix Receiver to operate as a plug-in, or as a separately installed program.  This function first shipped in 1.3 which was a streaming client shipped asynchronous to XenApp releases.
• AIE* environment variables are now visible from pre-launch/post-exit scripts run outside of isolation.  This enables things like purging the per-user isolation space before/after application usage, without using RadeRunSwitches.  Some other issues with scripts were also fixed.
• A years worth of App Compatibility improvements.

The version number will be 5.2.  If you've seen my posts before, you know that I never intended to move away from 1.x, including shipping major new function in point release, like Inter-Isolation Communication at release 1.2.  Long story as to why, but the bottom line is that the streaming client after 1.3 will be 5.2.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems

I have previously written about how the Application Streaming client prunes the file system cache.  What was not discussed is how the streaming system manages the REGISTRY portion of the execution content.   This post brings attention to that need and provides a reference to a source code application that can assist with pruning the registry portion of the execution cache.

When the streaming system executes an application, it can run more than one version of the application at the same time.  New versions of the app can be used by user B when user A is still using the old version.  This way, the admin can update an application "hot" and when users closes and restart the application, they get the new version.  This covered in some details in the discussion of RADE.

What is not obvious is that while the streaming system file system cache has a high water/low water algorithm for managing the cache, it has no such algorithm for the REGISTRY content.  Peek into HKLM at the RadeCache space and you can potentially see registry content supporting MANY versions of old applications.   

Here's the execution space on the machine I am using to write this post.
Notice in my case, I'm on version "5" of the execution Target 1b3... which happens to be Microsoft Office 2007, the Vista/Win 7 execution target as published in the internal "showcase" at Citrix.  Also notice that there is no "4" because I have a strange habbit of formatting my primary development box once a month whether it needs it or not and the admin hasn't updated the profile since my last reload.  I also know that our showcase admin occasionally does admin magic to delete these old spaces, but that is getting away from the purpose of this post.

Real customers

Heard from the field a couple months ago of a customer who had taken RADE to true heart; updating applications almost daily and then streaming them to server.  After a few months, they were in the hundreds in the versioning, and while this worked fine, the registry had become polluted with all the old versions.  Notice that the file system was fine, the old versions had aged themselves out of the cache and been erased, but the registry was littered with stuff that should not be there.

Utopia

One can effectively and easily argue that the Streaming Service should observe application updates and when nobody remains using the old versions, it could happily perform house cleaning to purge out the not needed information from the installation image registry.  This would be a very good argument and I'd stand up and say, you are right, which is what happened with this customer.  Things like Rollback to earlier versions makes this more complicated, but in the general sense, managing this registry space should be no different than managing file system space.  It is more difficult though because for file system, there's an advantage that anything you erase by mistake, will automatically be put back if it is needed.   For registry, it's populated once and left alone.

To "solve" the problem, I coded a little utility to prune the registry cache.  It runs outside the confines of the streaming service and defaults to "keep 2", to enable rollback.  The SOURCE code for this utility has been posted to the Citrix Developer Network, MyCitrix account required to get in.

One can imagine a day where this function is part of the streaming service.  I imagine that day too.  Until utopia arrives, if you need to programatically prune the registry cache, this utiltiy can help.  Set it up to run once a day/week and it will keep your registry space clean.  The UI is pretty primitive, it shows to stdout a before and after record of all of the space in the RadeCache along with indicator of stuff that it erased.

What it doesn't do

The utility scans the registry content, observes all of the cache space and then recursively deletes the "old" entries.  What it doesn't do is know which GUIDs are orphans, supporting applications that are no longer published.  Interestingly, the streaming service also doesn't know which entries are no longer used.  It would be a worthy addition to also reference the "last access time" information and purge items that are over a limit.  Here, shelling to the RadeCache utility to perform the erase would be the correct action so that the official tool could be used to purge our the execution cache, both file system and registry.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems

Citrix Application Streaming runs the majority of the isolation system as a service.   Being good security citizens, the service (radesvc.exe) is run on a named user account (Ctx_StreamingSvc) rather than LOCAL_SYSTEM.  There's some fancy security name for this but what it comes down to is using the least privilege you can to help limit the potential damage you can cause should you go astray. 

The user Ctx_StreamingSvc can write to only a few locations on the machine.  As a user account, it automatically cannot write to \Windows or \Program files and cannot write to HKLM, or even HKCU for any user other than itself.  This is good.  There are some things however that just demand a service, such as populating a SINGLE application installation image no matter how many users will be logged onto the machine.

At installation, the streaming service user account is given full control of a few locations:

• \Program files\Citrix\RadeCache
• \Program files\Citrix\Deploy
• HKLM\Software\Citrix\RadeCache

These relate to the "middle layer" of the "layers of glass".  They are the SINGLE space that support multiple application executions across many apps and many users.  The ability to write to these spaces is needed to runtime populate the execution content, so this is pretty much a "must have" for the streaming service to do it's job.

Where it gets neat is is that real USER accounts have to "see" stuff in the cache so that they can layer in the isolated content, so that the layers of glass can support read-only opens to the stuff in the installation image.  In the normal Windows installation, users, read: not admins, can "see" anything beneath \Program files, so in theory, there's no action needed to pull this off. 

Being security paranoid folks, the Citrix isolation system restricts users from seeing execution content unless they are actively running the application.  The DACLs are runtime added and removed.  Here is a file system view of the DACLs applied to the RadeCache execution space, first for the service account and then for a user account.

RadeCache directory

. NT AUTHORITY\SYSTEM:(F)
  NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(F)
  BUILTIN\Administrators:(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(F)
  CREATOR OWNER:(OI)(CI)(IO)(F)
  machinename\Ctx_StreamingSvc:(F)
  machinename\Ctx_StreamingSvc:(OI)(CI)(IO)(F)

That part not too interesting.  

The GUID_v directory that holds the application content is below the RadeCache.  The below taken from my Windows Server 2003 test machine while running an application as user "U1". 

. machineordomain\U1:(RX)
  machineordomain\U1:(OI)(CI)(IO)(GR,GE)
  NT AUTHORITY\SYSTEM:(I)(F)
  NT AUTHORITY\SYSTEM:(I)(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(I)(F)
  BUILTIN\Administrators:(I)(OI)(CI)(IO)(F)
  CREATOR OWNER:(I)(OI)(CI)(IO)(F)
  machinename\Ctx_StreamingSvc:(I)(F)
  machinename\Ctx_StreamingSvc:(I)(OI)(CI)(IO)(F)

The thing to observe is that the user rights for the directory space are runtime added when the application is launched, and runtime removed when the application terminates.   Even while the application runs, the user cannot see the RadeCache directory itself.

Here's how the user sees it.

C:\Program Files\Citrix>dir radecache

Directory of C:\Program Files\Citrix\radecache

File Not Found

Same user, this time "look deeper" to see the directory space that supports the execution of the application that they are currently running.   You have to "know" the GUID to have this work because you can't directory scan the RadeCache.

C:\Program Files\Citrix>dir radecache\0561beac-e4b8-47d8-8a71-fe5752f330f9_3

Directory of C:\Program Files\Citrix\radecache\0561beac-e4b8-47d8-8a71-fe5752f330f9_3

08/07/2009  10:31 AM    <DIR>          .
08/07/2009  10:31 AM    <DIR>          ..
08/07/2009  10:30 AM    <DIR>          Device
05/06/2009  06:23 PM                 2 FontData.dat
08/07/2009  10:43 AM            24,340 Hashes.txt
05/06/2009  06:23 PM         1,295,137 InstallRoot.tab
05/06/2009  06:23 PM                 2 Preextract.txt
05/06/2009  06:23 PM            28,265 SandboxData.xml
08/07/2009  10:31 AM    <DIR>          Scripts
08/07/2009  10:43 AM            35,565 TempSandboxData_2b264684-41d7-40eb-9c69-b
cb32beba720.xml
               6 File(s)      1,383,311 bytes
               4 Dir(s)     662,564,864 bytes free

In the words of Bill Cosby, "That's kinda cool!". (1:54).  The user cannot see the RadeCache directory, but they CAN see the subdirectory.  This gets more fun when you change dir into the GUID_v directory, which succeeds, and then "cd .." to get to the parent, which fails.

Why bother?

1) Citrix Security people are paranoid and just want dev folks to write extra code

2) Prevents users from even seeing applications that are not published to them.

The answer is "2".

XenApp Server side execution

All this DACL stuff happens whether stream to client or stream to server, but the dacl benefit is important mostly for the server side case.  If there are 50 users on a XenApp server and only 10 of them have access to "very secret application", then only that specific set of 10 people will be able to even SEE the executable, much less run it.  Compare this to locally installing the "very secret application", where users can see the program files space for the application even if it isn't published to them.  In the streaming case, they can't see it in program files, and they cannot see it in the RadeCache. 

What about the file server?

The Application Hub holds the executable content and making things secret means that users must be restricted from seeing things on the Application Hub in addition to the execution machine.  Admins I have queried on this have replied "simple, I use the same groups to publish applications as I do for granting read access to the sub-directories on the Application Hub". 

This is why the profiles are all stored as sub-directories off of the root of the "profiles".  Everything related to a given profile is in a single directory.  This creates an extra step to find the .profile in the streaming profiler file open dialog, but it really helps to simplify the DACL management.  This is also why the Streaming Profiler doesn't let you enter the full path to the "save to" location.  The use of "profilename\profilename.profile" is required for this to work, so the streaming profiler prevents you from using other forms when saving.

Joe Nord

Product Architect - Application Streaming

I received an inquiry recently; can Application Streaming be used to virtualize server infrastructure apps, like Apache, SQL server and a variety of others?  I've heard the inquiry before, but have so far ignored it.  I write this post to find out if I'm somehow missing the point because as the Citrix Architect of delivering applications virtualized, I don't see any point in isolating these applications.

My response: Sure, use Provision Server.  Don't like that one, use XenServer.

Let me put it differently.  COULD you "App Streaming" style isolate the execution of a server side infrastructure application? SURE.

Should you?   In my view, no.  Why do it?

The entire PURPOSE of the server is running that application, so why isolate the application execution.  Well, it isn't really an application, the entire purpose of the machine is redirected to serving this single purpose. 

Could I STREAM the application bits in just time time for execution?  Sure. Would I ever do that?  No. I would go out of my way to make sure the execution bits were in place before runtime.  That leaves isolation. 

What about application updates

RADE provides for hot updates of applications as users run them.  In the case of server side, I could profile the application in one place, prove it works, then reboot the production servers to have then bring in the new version of the application.  This sounds like a perfect use of Provisioning Server. 

Maybe I could get the new version of the app onto the server by closing the app and relaunching it; no reboot needed.  The result is the same result, the web/database server still has to go down to come back up.

Notice this is the App Streaming architect saying "Don't run these apps with my stuff".

Okay - IT people, tell me what I'm missing.

Joe Nord

Citrix Systems

Product Architect - Application Streaming

Last week at internal Citrix Sales Engineer gathering, I was presenting stuff on Application Streaming, where I noted that Application Streaming cannot isolate 16-bit DOS applications.  Immediate response from the SEs, yes it does - I have it in production and it works.  ... Things that make you go hum.  

I went home and tested it.  Answer: The SEs are completely right.

Why?  Our file system driver is kernel mode and "sees all".  Same for the code that looks after processes coming and going.  There would be a problem with registry filtering and this would be a show stopper, except there is no registry for DOS apps.   So, I've been telling people for years that this is a product limitation, when it really isn't.

Here's a picture of Turbo Pascal, 1987 vintage, running under Citrix Application streaming.  Also note that I saved Hello.pas to the C:\Turbo directory, which shows up in per-user layer of isolation.

Enjoy.
 

Common Application Streaming question:

How much data lands in the execution cache compared to what is captured at profiling?

My quick answer to this question is 25 to 30% of the captured content will land on the execution machine at runtime.  A more correct answer is ... "it depends".   People like numbers and they need numbers, so we give them numbers.  Statistics though can be misleading.  Are they good numbers? Sure, absolutely! Read more to understand some of the complications.

To get it started, this is the wrong question.

What people really want to know is how much disk space is used to deliver this application via Application Streaming as compared to delivering it via local install?  Answer: 35.6%.  Again, read on for more details because the real cost is a bit lower than this number and gets back to that 30% space, or the real cost is quite a bit higher than this number depending on whether you're going to go offline or not.

Application selection

The selection of the application here greatly changes the outcome of the question and deliverying for OFFLINE really changes the calculation.  First, consider online execution.  If the application has only a few files, all of which are accessed at runtime, then the isolation system will populate 100% of what was profiled.  By contrast, if the application has thousands of files, most of which it never accesses, then the answer approaches zero. 

The key point is that files are brought in when they are ACCESSED.  If they are not accessed, the isolation system LIES to the application to tell it that the stuff is present when it really isn't.  When it isn't, it doesn't use any disk space and it doesn't occupy any network time being copied in from central strore. 

It is amazing to watch how much stuff applications install that they NEVER, EVER reference.

NUMBERS

The measurements that follow are taken from my primary notebook machine, running non-release Windows 7 RC1, with Citrix Application Streaming, non-released development "tips" build from about 2 days ago that flunked build test, but which seems to be working pretty good for me.  I'm connected to the "showcase" farm inside Citrix, which is where we put the stuff that isn't released yet to "use what you sell".  Hundreds of people using this by the way.  I'm running a non-release and pretty current PNAgent which I haven't changed since before Synergy conference.  This is about as "stable" an environment as I get.  Other than these items, my environment is pretty typical and should be fine for statistics.
Why Microsoft Office 2007?  Answer: It's a really big application and everyone understands it.  Is it an example of a typical application?  I'm not sure, but it is sure an excellent testcase for proving out an isolation system.

Framing the problem
There's more to it than what gets populated into the execution cache; offline/deploy and online are also part of the equation.  If the execution is "stream to server" then there's no need to deploy a copy of the execution image on the execution machine.  Disk space usage will be optimal, the low number.

For "stream to client", offline will be common, so disk usage will be higher as you have to store a COPY of everything that is on the Application Hub that supports your execution.  This storage though is a compressed version of what was captured at profiling, so it is reduced in size and you only get a copy the execution target that is right for your machine, so the size used for deploy will often be less than the disk space used on the server that stores the execution content.

Application Hub:
Our showcase farm is consuming 27.2 GB of disk space to store MS Office 2007, on the server.  Much of this is wasted space because no house cleaning has occured.  There are for example 5 version of the execution target stored which support Vista/Windows 7 and there only needs to be 1.  My admin could save 80% of this disk space and nobody would notice.  By spending the disk though, he retains the ability to perform rollback. 

LOCAL MACHINE
Deploy: 1,161,958,306 bytes. 

Notice this is a WHOLE bunch smaller than on the App Hub.  I'm pretty sure our admins flush out the old stuff to help keep this number small, but even if they don't, I format and reinstall machines often enough that this doesn't come up.  The deploy location though is COMPRESSED.  We install everything and a bunch more when profiling for the showcase farm.  The expanded number for the size of the packaged MS Office is 2,125,243,930 bytes.

RadeCache populated execution space = 321,401,139 bytes.

Drum roll for the math

Online: 321,401,139 /  1,161,958,306 = .2766 = 28% OR

Online:  321,401,139 / 2,125,243,930 = 15% (That's a nice statistic!)

Offline: (321,401,139 + 1,161,958,306) / 1,161,958,306 = 128%
Did you just say that offline streamed uses MORE than locally installed?  Yes.  We made a conscious decision for this actually.  The extra disk usage allows online and offline streaming to be largely the "same" from the view of the isolation engine.  Technically, this is called "deployed" or "not deployed".

Compare to locally installed.

I had to struggle, but I found a Citrix PM who is still has MS Office 2007 locally installed where I could get some numbers.  His machine has 904MB, 904,155,136 bytes consumed.  Notice that this is smaller than the 1.1GB that is in the CAB file that our showcase farm has.  This is expected.  Our build has some other stuff added beyond the base MS Office and also was profiled with the of MS Office install selections, selecting "everything". Among the nuggets that are "extra" in the profile are a full copy of Mozilla Firefox.  I'm not sure what it's doing in there, but adding these things does make the profile grow.

Using these "common user" numbers for locally installed and comparing to the showcase MS Office 2007 profile, we can calculate a second statistics.

Online vs common user: 321,401,139 / 904,155,136 = 35.6%

Offline vs common user: (321,401,139 + 1,161,958,306) /  904,155,136 = 164%

Fascinating numbers.  What good do they do me?  I'm not totally sure.  If you operate on the theory that streamed delivery will use about the same disk space as installed delivery, you're in the right ballpark.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

User32.dll is a magic item.  As a programmer, if you want to take the machine over, then user32.dll is your best friend.  User32 is a system DLL that gets loaded into all programs, system and user, that do anything with the GUI.  User32 has a nice side benefit that it also loads other DLLs, by name.  The list of DLLs to load is stored in the registry in a string item, AppInit_Dlls.  Yes, this space is only writable by privileged proceses, but if you can get yourself on the AppInit_Dlls list, you're golden!  This is so handy that it is a common method that viruses use to attach themselves to all the processes on a system and ... is how application isolation systems like Citrix Application Streaming do their work.

Notice above, I said ALMOST all processes link to user32.dll.

There are many processes which do not load user32.dll and if they don't, then things that load as part of AppInit_Dlls will not get loaded.  If you're in the application isolation business, this is not good because it means that you can't isolate that application. 

A common question - Does Citrix Application Streaming depend on the application loading user32.dll in order for the isolation system to hook the app's execution?

Answer: No. 

The more elaborate answer is that AIE on Presentation Server 4.0 did depend on the application loading of user32.dll, but Application Streaming does not have this limitation. 

Propeller talk on user32.dll. 

If you want to know more about hooking processes and user32.dll, here are some good and entertaining references. 

• User32.dll is really important
• AVG Virus scanner incorrectly recommends erasing user32.dll (Awesome!)
• AppInit_DLLs should be renamed Deadlock_Or_Crash_Randomly_DLLs
• How virus loads
• Microsoft Malware Protection Center blog (good reading)

Probably best for a separate post, but the second item here is really interesting.  If you think you have a virus and do a google search for "user32.dll virus" you'll get 574,000 hits!  Sometimes, it seems like the "fix" for virus is worse than the virus itself.  If you delete user32.dll, you're up a creek with no paddle!  You can hope that the Windows XP system file protection will put it back, but it's still a scary proposition. 

Consider that if you are evil, and you're inserted into system code, then the obvious next step is to hide from anti-virus.  This must be an interesting battle. 

Enjoy,

Joe Nord

This is"part 2" in a propeller-head series about the internal workings of Citrix Application Streaming.  This post covers how stuff in the registry can be "erased" during profiling without really effecting the profiling machine.  The same concepts apply at runtime where the application can erase things from the machine and have this effect only be apparent for the current user rather than the whole machine.

Part 1 covered the same topic for files.

Consider: Machine has registry space at HKLM\Software\DeleteMe (yes, notice the easy to recognize naming).  In this example, there are also a few registry items located in that space, abtly named, item 1 and item 2.

The installation program observes that this space exists and as part of its installation activity, erases it.  The job of the isolation system is to let the installer BELIEVE it erased the space, while not really erasing it, so that the streaming client can present this space as "not there", when it really is.

For the installer, I have used CMD.exe and here's the activity, captured as text.
c:\>reg query hklm\software\deleteme

HKEY_LOCAL_MACHINE\software\deleteme
    Item 1    REG_DWORD    0x1
    Item 2    REG_DWORD    0x2

c:\>reg delete hklm\software\deleteme
Permanently delete the registry key HKEY_LOCAL_MACHINE\software\deleteme (Yes/No
)? y
The operation completed successfully.

c:\>reg query hklm\software\deleteme
ERROR: The system was unable to find the specified registry key or value.

From outside of isolation, go look at the "real" registry.  You'll see:
What happened? 

Answer: The true registry was not "hurt" by the profiling activity.  The registry space that was deleted wasn't really deleted.  The captured profile though believes it was deleted and when moved to the execution machine, the Streaming Client will do similar isolation stuff to make this space appear "gone" even if that space really exists.  The "how" follows.

Package the app up and peek into the captured registry contents and you'll see the magic that makes this happen.  Recall from the file discussion, NTFS Alternate Data Streams are attached to the deleted files to describe them as erased.  The isolation system does similar activity for deleted registry.

Notice that the deleted registry space "exists" as part of the captured registry space.  It exists, but there is also this extra stuff that the isolation system references to conclude that this regsitry space is "erased" and should be masked from vision to the application.  In the case of the registry as well as for files, the erased markers are attached to the erased stuff so that the isolation system doesn't have to look far away to index things that are erased.  When a registry key is opened, the isolation system looks to the side to see if the marker is present and if it is, it "hides" this registry key from the view of the application. 

Magic?

I called this magic above.  In reality, no magic.  If the application happened to use and depend on a registry item named CitrixAIEDeletedStatus, this whole thing would not work.  Fortunately, this is a rather unique name and this makes it okay to add to registry space, where the application won't be impacted.  As a last step, the registry items that are the markers "don't exist" from the perspective of the isolated application, so this prevents confusing an application by giving it extra registry items that it did not define.

What is the other Citrix thing

Good question.  The CitrixAIEPlaceHolder item exists so that the isolation system can transport registry KEYS that have no contents.  Compare in concept to XCOPY /S and XCOPY /S /E.  Some application create registry keys (similar to file system directories) and the mere presence of the key has meaning to the application even if that key has no contents.  Since the isolation system stores and later repopulates the registry space in a manner similar to .reg files save and restore, keys with no contents get lost.   By providing an item, any item, the empty registry key is preserved from profiling system to execution system and the application "works" because it sees what it expects to see.

Disclaimers

Lots of bit head stuff here on how the isolation system works.  Don't become dependent on it.  We change these things from release to release and it is likely that the method of representing deleted registry or empty registry can and will change some day in the future.

But - if you're looking at your machine and are asking yourself: What is all the Citrix stuff?  Now you have the answer.

Joe Nord

Deleted files are an interesting topic in isolation systems.  How do you represent a file as "gone", when it is still present?  During profiling of an installation program, the isolation system has to isolate additions to the file system; this part is easy and pretty easy to understand using the "layers of glass" metaphor.  You look down from above and the highest version of the file is the version that you see.  What is more interesting to consider is that the isolation system also has to represent DELETED files; and deleted registry content.  These require placing a marker at higher levels that say to mask vision to the content that really does exist.  Deleting files is much more interesting than adding files!

Consider a view from above, looking down through the isolation system goggles:
 
I used the IIC version of this graphic because it was handy, but consider the above with only 2 layers as this is the environment during profiling.  The PHYSICAL or "REAL" disk/registry is off limits for writes.  When the installer during profiling, or when the program at runtime writes to the isolate space, the changes are reflected as writes to the higher layers of isolation (the first "writable" space, which is always on the top).  This works great for adding content because the presence of the content at the higher layer "masks" vision to the layers below.

For deleting things, this gets interesting.  Consider a program at installation which erases content from the physical machine.  Looking down from above, the installer needs to BELIEVE that the erase it commanded, really happened.  Here, the isolation system has to represent "deleted" files and registry content without really deleting the stuff from the true disk.  Failure to do this would require a VM snapshot thing and resetting of the environment before/after profiling and we didn't want that.  Instead, you can profile over and over all day long without hurting the true physical image - at least for the spaces that are isolated, which is pretty much everything during profiling.

A deleted file is represented as a real file, at a higher layer of isolation.  The real file is changed to have 0 size, but still has the same filename as the lower layer file.  The filesize reduction is just to conserve disk space, but in concept the deleted file is the same file as the lower layer; promotion from physical to isolated occurs commonly for files "changed".  To make the file "deleted", the isolation system tags the file with a NTFS Alternate Data Stream, "CITRIX_DELETED_FILE_MARKER".   If the file exists on the layer of glass, but has zero size and the required tag, then the file "disappears" from the perspective of isolated processes.  The file still exists and is even visible outside of isolation; though it still has no contents.

References:

Forked file systems (Mac)

Scary reference - ADS are evil  IMO, They are not evil, they just aren't widely understood.

Microsoft KB describing how to use ADS

Sysinternals tool for displaying Streams

The fact that ADS are rarely used by "normal" Windows applications makes them a PERFECT candidate for usage in the isolation system.  The isolation system can attach a stream to the file and feel pretty confident that this won't hurt the applications ability to manipulate that file, yet the information associated with that file hangs out and even follows the file from place to place when it is copied.  Note: this isn't exactly true - keep reading.

Profiling

During profiling, the isolation system sets the marker on files that the installation program erases.  In concept, this means that the marker is present on the execution machine at runtime because the captured layer of glass is transported to the execution machine.  In reality, this is true only in concept.  In reality, we use CAB files to hold the layer of glass and CAB files do not maintain streams.  Hum.  Problem.  What did we do?  Answer: When the application installer completes processing, the profiler goes through and "deletes the deleteds"; removing the files with zero size and the stream attached.  This means that the deleted file will not be present in the captured image, but it also means that the marker will not be present at runtime.  Problem?

First, why is this a risk?  If the file existed on the profiling machine (at true disk location), and if the installer erased it; then on the execution machine, if the file exists on the true disk, then at application runtime, the erased file will re-appear.  As the theory goes, the application at runtime will get unhappy.
Is that a problem?

At the time, I wore the FSFD dev hat and took note that darn near zero application installers actually erase things from isolated spaces during profiling.  Those that do, likely will also tollerate that the file comes back.  We also took note that even if the file did re-appear, we could always erase it again using a pre-launch script.  Bottom line: Application Streaming has shipped this way from the beginning and I have never seen a failure attributed to this behavior.

But wait, there are more problems

At runtime, the application may decide to erase things.  In general, the only spaces isolated are the \windows directory and below and the \program files spaces or more precisely, these are the locations where an ill-behaved program may want to erase things; things that could effect the execution of the program.  

It could also be that the application erases some of its configuration data from the directory to which it was installed (a bad application). In either of these cases, a deleted file marker will be generated and will land in the per-user layer of isolation.  This will mask vision to the file on the true disk.  Since this is part of the per-user space, all should be happy because the per-user space will follow the user from machine to machine as part of roaming profiles or even better, Citrix User Profile Manager!  The Win32 CopyFile API DOES maintain Streams!

Problem: Roaming profiles do not maintain ADS Streams, so the deleted markers will disappear on movement from machine to machine.  4 years in the field, has this been a problem?  Nope.  

I have asked the Citrix UPM team to maintain Streams on copy of the user profile content to/from central store which is a bit of a curiosity because it is different behavior than Roaming Profiles.   I'm not sure if it occurs yet and not even sure if they will actually implement the change.  Why?  Fundamentally, the ADS streams haven't been missed yet, they will likely be missed less in the future; except for all them MAC users.  Stick with me.

What file systems support ADS Streams?

Excellent question!  The answer is "NTFS".  The Mac file system REQUIRES streams, but we rarely get the mac file system as the foundation of a Windows machine.  FAT32 definitely doesn't support streams and I'm not sure about CDFS.

Since deleted file markers are needed for isolation processing, the Application Streaming file system code restricts it's filtering to only NTFS disk volumes.  In some ways, this is a plus.  You don't want to isolate USB drives and these tend to be FAT32, so they don't get isolated.  But, the FSFD knows the USB drive is "removable" and it assumes all removable media is for "data", so it doesn't mess with it.  In this way, the NTFS restriction actually gets in the way a bit.  Note: you can convince the FSFD to look at the USB drive as I outlined here.

What about more elaborate file systems?

Some customers have enterprise storage which is network based, but appears to the execution machine as local.  These disk volumes COULD be NTFS, but often they have a custom file system.  Do these support NTFS style ADS streams?  Answer: It depends.  Let's say they don't.  How to solve...  Answer: Convert the representation of "deleted files" from ADS Stream to something a bit more common.  Has this happened? Nope.  Is it really necessary?  I'm not sure.  Experience of the last few years says it is rare enough that the failure case probably won't come up.  With this wide post though, something tells me someone is going to say: "I've seen the failure case".

In a follow up post, I'll describe how the registry "deleted" markers are implemented.

Enjoy.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

How do you add FTAs to profiled applications that the installer forgot to install?  Answer: There are a few steps.

This has been on my list to Blog about for a while, just haven't gotten around to writing about it, mostly because  I hadn't had a chance to verify the solution would actually work.  Conveniently, Karl Muller of I-Access took care of the trial for me, and it worked, so here's the solution.  Thanks Karl.

Review

Backup a couple steps... "FTA what!"

When an application installs, it writes entries to the machine to establish associations between file extensions (multiple) and programs (single).  Each file type definition also includes a reference to the ICON that goes with the extension. This is what Windows Explorer uses to show you icons that looks like MS Word for file's that end in .docx.  

The Streaming Profiler runs the installation program under isolation and blocks the installers attempt to set FTAs on the true machine.  As part of post processing, the profiler examines all the FTA entries that the installation program TRIED to create and for each of these, creates corresponding FTA entries in the streaming profile XML data - you can see them in the .profile file, if copy to .xml and view with web browser.  Easier, just look at the properties of applications captured during profiling and ask to see the FTAs.  Yes, that's much easier.

The Access Management Console (XenApp publishing console) reads the profile information and gets a list of all the captured FTAs.  It makes these AVAILABLE for association with applications that you publish.  The default is that no FTA will be associated with the published application, so as an admin, you have to go to direct action to establish the FTA by clicking on boxes.

BUT - There's a catch, you can only establish FTA reference for the FTAs that were captured by the streaming profiler.

Say you know that .DOC9 (.DOC and the number nine), is SUPPOSED to be associated with MS Word, but you note that the installation program seems to have left that one out, or that the streaming profiler seems to have missed this one.  Let's focus on the first one.  You have an extra FTA that is SUPPOSED to be associated that seems to be missing from the list.  How do you add an extra FTA?

Option 1: Streaming Profiler, application properties, FTAs, Add

Option 2: In the Access Management Console, application properties, FTAs, Add 

Both options above would be GREAT, if only they worked.  They are both READ-ONLY.  You can see the FTAs, but you can't add a new one.  
The solution...

FTA must be captured during profiling

When profiling, tell the profiler that you want to launch an installer, it will prompt you for the path and name of the installation program, tell it "CMD".  When launch the installer, you will get a command prompt running that is running inside of isolation.  Two commands are needed at the command prompt to establish an FTA.

• assoc .doc9=Word.Document.12
• ftype Word.Document.12

If the ftype is already in place, you're done.  If not,

• ftype Word.Document.12="C:\Program Files\Microsoft Office\Office12\WINWORD.exe" /n /dde
• exit

Now, let the streaming profiler do it's thing to see what the "installer" did and it will discover a new FTA for the .doc9.  Save the profile and return to the Access Management Console.

Publishing

In the AMC, publish a new application, point it at the profile and it will see the FTA that you added.  EASY!

There's a catch

There is always a catch.  The catch here is that the FTA will exist only for new application that you publish.  Applications that you have already published have already had the FTA data sucked out of the streaming profile and placed into the XenApp publishing infrastructure.  The AMC will not "pick up" or automatically update the applications to reflect changes to the profile, so the FTAs won't exist.  If you need that FTA added to an application that is already published, it is necessary to delete the application and republish it. 

The other solution

Not recommended, but you can accomplish the same thing by hand editing the XML data in the streaming profile and saving.  This isn't the RECOMMENDED answer, but just between all of us, it will work.

Joe Nord 

Citrix TV folks have made me look really good!  Here's a link to a 9 minute video where I go through the "layers of glass" in application isolation and the fundamentals of how Application Streaming works.  The video outlines the XenApp 5.0 new capabilities for HTTP based streaming and provides guidance on wide area network architecture including Branch Repeater usage to eliminate the Application Hubs at each branch office.

I have previously blogged about the layers of glass, here, and with inter isolation communication added, here.

This video is better.

Joe Nord