Administrators are used to the idea, that running applications under Application Streaming will permit poorly written applications to run in a multi-user terminal services environment.   For example, if the application wants to write to the \Windows directory, no problem; the application will believe that it wrote there and later if it reads the same stuff, it will see what it put there and generally, the application will work. What is less known is that that Application Streaming and XenApp publishing can be used to reduce the rights of the application at execution so that it has a reduced chance of hurting the machine.

Privilege vs. Isolation

Isolation and "privilege" are different things. Running the application "isolated" does not mean that the application can't do powerful things.   An administrator privilege ISOLATED application CAN still perform privileged operations such as adding new users to the machine, marking them as administrators and adding them to the remote desktop group where the evil doer can then remotely login, as a non-isolated administrator and easily do evil things. 

Not a problem for XenApp hosted execution

To be clear, none of this is important for XenApp hosted execution.  Here, the user is already a user and stripping power from the user to get them to user power is a "nop" because they were a "user" to start with.  This discussion of "privilege" reduction is more of a Windows XP client side, or hosted desktop statement where "admin" power users are the norm.   On Windows XP, unless you're very good at locking down the machine the end user will be running as an "Administrator" and this is not desired.  How can you make this happen as little as possible?  How can you get MOST of the applications to run with the least privilege possible?

Brain damaged applications

Some applications even CHECK to see if they are admins and refuse to run if they are not.   Awesome!  If you can't figure out how to code it, demand admin rights machine wide!    You can easily hit a situation where 90% of your desktop applications will run fine without admin rights, yet you have no choice but to make the user a full blown administrator because some small subset of the applications demand admin rights; or perhaps, even really need them.

What about the "normal" applications that don't need admin rights, or at least don't need admin rights when run under isolation?  It would sure help if we could at least make the "all powerful" user be a "lowly user" for the purposes of the majority of application execution, even if the user is really an administrator.  You can, and XenApp makes this easy.  First, some history.

DropMyRigthts

Go back in time and take a look at this 2006 technet article from Microsoft on Least User Access and a description of the DropMyRights utility by Michael Howard.   Excellent stuff and here is a related set of blogs from Aaron Margolis of Microsoft who seems to have a passion for running apps as a user!   The output of this early work was a command line utility called DropMyRights which would duplicate the user's logon token, strip the powerful rights - and then use the modified token to launch the application.  Good stuff.  As an example, here is the .BAT file I used to use to launch MS Outlook.

• dropmyrights "%PROGFILES%\Microsoft Office\OFFICE11\OUTLOOK.EXE"

The idea of running apps on forced user privilege on Windows XP was not unique to App Streaming, but we did wrap pretty GUI around it and wrapped application publishing around it to make it easy to use - and then we didn't tell anyone it was there.  To be fair, most of the usage was server side, so it wasn't as important, but hosted desktops are changing this.

The XenApp publishing system makes this dropping of user rights accessible via easy to use GUI.

Access Management Console

Here's the AMC screen that controls this setting.  Notice that this "stripping of rights" is controlled in the AMC - not in the streaming profiler.  Could it be controlled in the profiler?   Sure.  Both of these tools are nice GUIs which could accomplish the same goal, so yes, it could be controlled in the profiler, but it isn't.  One could even make a really good argument that it is in the wrong place and SHOULD be in the profiler because this is where the admin is that knows more about the application.  I would agree, but it wouldn't matter, it's still in the publishing console whether or not this seems like the right place.  

 
When I wrote the draft for this post, I did it in a place without internet access, so I couldn't easily check the default.  I wrote that SURELY! the default is that we strip the rights before launching the app.  Surely, Shirley, what ever you call it, the default is the other way; by default, the launch leaves the user's token alone and launches the app using what ever power the user has according to logon.  If you CHECK the box, then the Access Management Console tells the Citrix IMA to tell the Citrix Web Interface to tell PNAgent to tell the Streaming Client that it should strip power from the user for the purposes of running this stream to client application.  Where the application will permit it, You should set the checkbox.

XenApp server side, it won't change anything;XenApp Client side, it will ensure that the application is launched using a user token that has "lower power".  Lower power is better...

Here are some other writings on Application Streaming related to this:

•  Enhancing the Security of Application Streamingfor Desktops

Enjoy!

Joe Nord

Citrix Systems Product Architect - Application Streaming

One of the first screens you will see in the Streaming Profiler wizard is a screen about "Enable User Updates" or in the earlier profilers, this was called "Enhanced security" or "Relaxed security".   Wow!  Mysterious terms!  The first thing we do in the profiler is hit the admin with a question that they don't know the answer to.  Hum.

Steps:

1. Describe the panels
2. Describe what the settings do
3. Examples of how this effects application execution
4. Guidance on how to configure the setting

Here's the panel in the streaming profiler version 5.2 (XenApp 5 Feature Pack 2):  Hot off the presses, released GA to the web download last night.

Here's the same panel in the previous streaming profiler (1.3)

What does this setting do?

Under the profiler, it doesn't do a whole lot.  It just sets a BOOLEAN that accompanies the streaming profile.  You can see via nice visual form in this streaming profiler, but if you dig down, you'll find that all this does is set a boolean in the profile XML data; at the PROFILE layer.   Changing this setting actually does more work, but I'll get to that in a minute.

Going back to the Layers of Glass, there are conceptually 3 layers of isolation.  Here's an abbreviated version.

At runtime, the applications in the isolation sandbox see a multi-layer merge of the true machine at the bottom, masked by the installation image and at the top, a per-user layer.  The per-user layer is seen "first", followed by the lower layers of isolation and finally the true disk or true registry of the machine.

The normal action is that the machine starts out pretty much clean, the streaming profiler captures the installation activity of an "installer" that writes stuff to the file system and registry.  These are packaged up to become the "blue" layer above, the installation image.  

At end user execution, the installation image is laid down on top of the execution machine and as far as the isolated applications are concerned, they are installed.  It's all a lie - they aren't really installed.

The top layer is initially "clear" or "blank".  As the programs run, they may store documents and similar, but these would generally not be in isolated space, so they don't really show up in this picture.  The application though may WRITE things to "off-limits" locations which would be caught by the isolation system and end up with storage of stuff to the per-user layer of isolation.  These land in the top layer of the isolation stack which is set up as one per-user.  This is what allows ill-behaved application to run happily under isolation on a multi-user machine when they won't happily run without isolation.  As an example, consider an application that stores settings to the program installation directory in a .INI file.  Under isolation, this will be captured and land in per user space and the application becomes runnable in a XenApp Terminal Services world where otherwise it would not work successfully.

Back to this post

If the application updates itself at runtime, the update will land in the per-user layer of isolation and this is bad.  Standard procedure when profiling application installations is to TURN OFF all automatic updates.  The application should not update itself - this should only be done in the profiling scenario where the administrator commands the update.  Recall that the isolation space is ONE and the per-user space are MANY, so we only want application content to be updated in a single place.

What does "Enable User Updates" do?

If the user downloads application updates such as .DLL updates or .EXE updates, should this be permitted?

The general answer is "NO!".  Some administrators may have a scenario where this is desired.  The common ones are users wishing to install their own plugins for isolated web browsers or install their own addons for things like Microsoft Office.

How does it work?

Put your file system filter driver writer hat on.  For isolated applications, EVERY TIME the application opens a file or tries to open a file, you get first look.  If the file open is for executable content, should this be permitted?  If "enable user updates" is "off", then file opens for RUNNING executable content from the user layer will be denied.  

The neat part here is that the isolation system distinguishes this behavior based on WHO the caller is.

If the caller is vanilla application wanting to read or write content, no problem - do what you want.  If the caller is the Windows LOADER, then this setting comes into play.  If the LOADER is trying to load executable content from the per-user layer of isolation, then the isolation system can be told to FAIL that operation, and this is what this setting controls.  Pretty neato.

One headache

The setting while stored as a profile level single property (a boolean) is implemented in the isolation system as an attribute of EACH of the isolation rules for EACH execution target of the profile.  If you set the profile level property, the streaming profiler must modify the isolation rules (many) for each Target of the profile.  This means that if you have a profile with 4 execution targets and you're editing one of them - and you set the profile level property, behind the scenes, the profiler brings the other 3 execution targets into "edit state" to make the change and will eventually write all 4 targets back to the application hub.  Going to edit state to change the rules requires unzip of the can file from the network server onto the profiler machine.  If the profile/targets are large, this can be a very painful operation to accomplish a single boolean set, but this is how it is.  If you make this change, be aware of the large behind the scenes work that the profiler is doing.  Grummble yell a bit and then it will be done.

Fun with streaming - Great entertainment in isolation circles

Turn on the -x RadeRunSwitch so you can an isolated command prompt when you launch your next favorite streamed application.  This assumes you have user updates disabled, which is the default.

cd c:\windows\system32
c:\Windows\System32>notepad.exe
< it runs >

c:\Windows\System32>type notepad.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>copy notepad.exe n.exe
        1 file(s) copied.
< file copy was successful - n.exe is at the per-user layer of isolation >

c:\Windows\System32>type n.exe
< see textual giberish - the file open succeeded for read access from CMD.exe >

c:\Windows\System32>n.exe
The system cannot find the file c:\Windows\System32\n.exe.

FIREWORKS HERE!

The isolation system LIED to the Windows Loader - returning ERROR_FILE_NOT_FOUND (2) rather than completing the loaders request to run this file from user layer of isolation.  This is what this setting does!

But wait, there's more!

c:\Windows\System32>copy n.exe notepad.exe
        1 file(s) copied.

c:\Windows\System32>notepad.exe
< it runs!! >

Why does notepad.exe succeed in the final case?  Easy, there are two notepad.exes.  At the per-user layer, there's a notepad.exe which was written on the file copy from n.exe.  We don't care what this file is, but it is executable content and it exists at the per-user layer of isolation and therefore it doesn't exist for purposes of running programs.

Since the "Enable user updates" setting is set to disable user updates, executable content at the per-user layer of isolation does not exist from the perspective of the Windows loader.  BUT - at the physical layer, there does exist a file with that name and this can satisfy the file open, without violating the isolation rules.  There could also be a file with that name at the application installation image layer.  In this example there wasn't, but there could be.  The isolation system starts at the top and goes down until it finds a hit.  If "Enhanced security" is enabled, then the per-user layer is "off-limits" for execution of executable content.

The grand result

The application "update" applied by the user may have been applied as far as the user or application is concerned, but in reality, it was not applied.  The version of the application that is running is the version that the administrator profiled.  Pretty cool stuff.

Why did we rename the setting?

Putting "security" in the title implies that this will somehow prevent users from doing things to run content that they download and this is not what it does.  If the program updates itself, then this setting will block that content from being executed.  The setting can also block user installed additions to the program (plugins), depending on the location to which they were installed - was it included as an isolation rule during profiling?

Take a web browser for example, if the user downloads executable updates to the browser, this will be captured and the user installed stuff won't run, but if the user downloads evil.exe and places it on their desktop, and then double clicks it - this will be outside of isolation so the layers here do not apply.   This is also true if the user downloads evil stuff to locations outside of isolation and launches it from the isolated application.  It will still run isolated, but it will run!  Describing this activity as "disable user updates" is more accurate than the previous words, so we've made the change.  I also hope that it removes confusion in the streaming profiler wizard.  "Enable user updates" is pretty easy to understand.

How should you create your profiles

1) Enable user updates should generally be "off".  Plugins are a rare need and where there is a real need for users to add plugins, start asking yourself if you can add those plugins at profiling to the common layer.  OR, if the use of user installed executable content is large, should this application be locally installed rather than isolated?

2) Always tell the application to NEVER update itself at runtime.

 A look to the future

Streaming dev team are discussing removing this option from a future release.  That is, "Enable user updates" will always be OFF.  I'm not sure of all the ramifications of this yet.  The question really is "how many admins are profiling their applications with user installed updates permitted"?  I hope the number is "few".

Joe Nord

Product Architect - Application Streaming

Citrix Systems

To use agent-less streaming or installed agent streaming; that is the question.

I have received inquiries recently - lots of inquiries on the same question actually - that all imply that "user mode" streaming and isolation is "better" than evil kernel driver streaming and isolation, because it doesn't have any kernel components.  This sounds exciting on first glance, but when you dig down the problem is more complex.  There are advantages to installed isolation systems and there are advantages to all user mode.  This blog shows some examples for both categories. 

To get it started, consider an isolation system's ability to load and isolate a Windows NT Service.  How can you isolate the execution of an NT Service if the isolation system itself is "user privilege".   At user rights, the isolation system lacks the privilege to start a service that isn't already installed, so it can't run and isolate services unless the service is already installed outside of isolation.  

Scalability is also a big concern.  It doesn't matter for a small system, but if you load the machine up with lots of users and lots of applications, having a separate copy of the installation system for each application becomes problematic.  

On the plus side for agent-less, it is really nice to NEVER need admin rights on the execution machine and this can help engage work from home or kiosk environments.    Another plus for agentless, the "attack surface" is smaller if there are fewer privileged components; this is the point that usually starts the conversation that "agentless is better".   Keep reading...

First, some definitions:

Agent-less streaming

Nothing is "installed" into the machine.  The isolation engine can run the profiled software without ANY components themselves being installed.  "Admin" rights are never required and the system usually embeds the isolation system and profiled application into a single .EXE image that is "executed" to run the isolation system and the isolated application.

Examples of agent-less isolation systems include ThinApp, InstallFree, XenoCode.

Installed agent streaming

With an "installed agent", parts of the isolation system are "privileged", they require an install time execution which usually includes the installation of at least one kernel mode device driver and generally also includes the installation of a NT Service that supervises the sandboxes that are active on the machine. 

Examples of installed agent isolation systems include Citrix Application Streaming and Microsoft App-V.  Even this is a simplification as the Citrix system uses kernel mode for process monitoring and file system filtering and USER MODE for registry filtering, which makes it kind of a hybrid.  App-V uses kernel mode for both.   The real key is that an installation step was required.  If you require an "install", you are a member of the installed agent camp!

Which is better?

The answer is simple: Mine! 

The real answer is that you have to ask what "better" means.  I mean, better at what?  Each of the systems have their advantages.

Consider installation

Not needing to install an agent is handy for some environments especially where the execution machine is not company managed.  USB Thumb Drives were the original player here; you can take you application anywhere!  By contrast, if you're really running in a company environment, either stream to client, running on a XenApp Server or even on a XenDesktop hosted world, the installation of an agent is not a gate - the admin controls the base image and can install any agents that they want.   For more on this, see "security" below.

Consider scalability

If you are running a heavily loaded XenApp Server.  Let's say 64-bit machines, with 64-bit Operating System, a handful of high end CPUs and enough RAM memory to run 100 concurrent users, with each user running 5 distinct isolated applications.   How many isolation spaces is that?  100 * 5 = 500 isolation spaces.  If the isolation system is 100MB, that's 500 * 100MB = 50GB of allocated virtual memory just to load the isolation engines.   I made these numbers up, but stick with me on the concept.

How many separate copies of the isolation system do you want to load?  Answer: One would be nice.  

With kernel mode or NT Service isolation systems, you'll have ONE single code space for all of the isolation sandboxes, a bit of instance memory for each sandbox and other than that, you're off and going.

This too is a over simplification.  For starters, the Citrix App Streaaming case is a hybrid.  I haven't checked the memory footprint lately, but the kernel pices are ONE and the registry and named object pieces are MANY as these are implemented inside the isolation space.  To get back on subject for this post, think of it as ONE installation system because all "installed agent" systems are in the same camp.

Does agentless mean that there is really a separate isolation engine for each application?  yes and no.   We tend to generalize this to say that the isolation engine is carried along with the to be run application.

If the agentless isolation system uses DLL references to get to the isolation engine, then the agentless system's memory load will be page table shared across the images.  This a function of the Windows NT PE Loader and memory manager.  With DLL load of the runtime, many pages of the 500 instances of the isolation system will be shared.   I don't know if the clientless systems use DLLs for the agent runtime, but if they don't, they could and this will share the load.  

Citrix App Streaming uses DLLs for the registry filtering, so the shared model here still applies (ONE).  

How to update the agent

If the isolation agent needs to be updated, how many application images do you have to update?  This is a plus and a minus for both again.  With installed agents, you update one and all the profiles/sequences benefit.

With agentless, you have to touch each of the isolation images.  Or, back to the DLL approach, you could achieve the same ONE update in agentless if you're using DLL based runtime.   

If go with the "one app, one executable to distribute" model, then the isolation system is not shared and the memory usage on a XenApp hosted model will totally suck.  Prototypes will be great, but actual performance under load will be a heavy hit to single server scalibiltiy.  When it comes time to update the isolation agent, you'll have to touch all of the profiles to get things updated; or you can skip this as a plus toward not having to maintain anything once you profile it.

Consider security

There is a perception that if kernel mode components are involved, then it's less secure.  This is mostly a statement of "attack surface".  There are many privileged components in the machine and they are all candidates for attack.  The real headache with privileged components is that they have to be "installed", but the security aspects still apply and are real.  

If you're installed and you have power, then you can do powerful things.  The corollary is that if you are not installed and you don't have power, then you CANNOT do powerful things and this is as much a plus as a minus.

Consider that many agent-less streaming systems receive "yes" check-boxes in comparison matrices when discussing isolation of NT Services.  

Either you're USER MODE or you're not user mode, you can't claim to be both.  

A step back: NT Services are installed applications, with no GUI and no direct user interaction.  Services run on either a powerful system defined account or a named user account of install time specified rights. "LOCAL_SYSTEM" and "LOCAL_SERVICE" are the common installation configurations and these equal "powerful".

In many cases, the reason that the programmer went through the pain to write a service is that they needed to do something with privilege and that was impossible from the user privilege application space.  This is why they are done in the service rather than in the application itself.

If the isolation system is agentless (user mode), and if that isolation system can load and isolate NT Services - and have that service work - then a boundary has been passed and the isolation system is no longer user mode.  Given that the agent itself would not be privileged, how can the applications that it runs be privileged?

Answer: The agentless streaming system requires the applicaiton services to be installed outside of isolation, and by installed, I mean REALLY INSTALLED.   This pretty much deletes the check-box for isolation of services.

Some NT Services likely CAN tollerate user mode execution under isolation, but for the general discussion, the answer will be that this breaks down and the service requires installation to the local machine - breaking the isolation boundary between the application and the local machine.
An "installed agent" isolation system CAN have the power to start services itself, and in this environment CAN run the service isolated.  This is only possible because the agent itself is INSTALLED and has power.  If the isolation agent is not installed, then the user cannot start isolated services and there's a pretty big gap in the claimed support of isolated services.  The services have to be outside of isolation and installed - they aren't isolated and the access to that service is not governed by the published application set.

Consider application launch performance 

Now that Application Streaming 5.2 client is out (real-soon), I can describe the greatness of 2nd time application launch.  If a sandbox is running that will support this app, there is no longer ANY need to create an isolation space for a second, third or fourth application launch.   Creating an isolation space/sandbox/bubble is an expensive operation.  The 5.2 client's ability to skip this expensive operation will provide great benefits in launch speed.  I describe this briefly here, but need to write more.   

Can agentless do things to equal this application launch performance?  Can it toss execution over the wall to already running isolation spaces?  Probably, but as an big executable based execution, by the time they run to make this decision, they will already have the isolation system mapped into memory and it's big and I'm betting, slow.  Bottom line, I'm looking foward to a new round of statistics with the 5.2 client - we should kick some major butt!  "Agent based" here has advantages, but this alone will not sway a discussion.

Consider central management
Both Citrix Application Streaming and Microsoft App-V are heavily predicated on the concept of communication with a back-end infrastructure and administrator driven management of the applications available to users and even preventing the execution of non-approved applications. 

The applications are published to users and in App-V case, users or machines and both systems communicate information back to the central authority to decribe the use of applications.   App-V can even block the launch of applications based on license metering.  All of this is enabled because of communication with back end, from an installed agent.  

Can that communication be done from an agentless system?  Probably.  Is this done?  I'm not sure.  If it is done, is that something you really want happening from a user privilege component?

What applications are available to the user and how can I trigger the update of application content?  What applications have been actively used across my whole organization and which ones are published that the users really don't care about?  What applications should I focus my support on and which ones should I deprovision without telling anyone?  Do I have enough application licenses?  

All of these questions can be answered with back end information.  Agent-based makes this easier.   Having a NT Service hanging out to collect this information and centrally report back statistics is an opportunity for central management, control and monitoring of applications.

Then again - agent-less can get most of this too. 

For example, Citrix EdgeSight monitors application usage on a machine and reports this back to make truly beautiful reports that tell the admin what has been happening on their machines.  It doesn't matter if the applications are isolated or not isolated, the EdgeSight monitoring system still sees them and can report on usage.   This happens for Citrix Application Streaming and can just as easily occur for clientless based isolation system.   

Conclusion 

Is agent-less better or is agent based better?  The answer really depends on how the whole system will be architected and what control the administrator has on where the agent will be installed.   Both have advantages.

Joe Nord

Product Architect of Citrix Application Streaming (An agent based isolation system)

Citrix Systems - Fort Lauderdale, FL

Have you ever looked into the App Streaming execution cache and said, "good grief!  What are all those GUIDs!"?  GUIDs are great, for the programmer.  For the administrator, they can be a bit more mysterious. 

I recently wrote a utility to decode the GUID data of the Application Streaming execution cache, displaying the decoded information to the console, but neater than TTY, it also updates the Windows Explorer display of that space so the "secret" information is exposed in permanent fashion and via the GUI, which is good stuff.   The utility is released to the Citrix Developer Network, in source code form so you can tweak it if you'd like. 

Download from here, look for the "Rade Cache Decode" link at the bottom of the page.

For background, I have written about the GUIDs before and how they are key to the streaming system maintaining the separation between "profiles" that you can publish and execution "targets" that the streaming system uses to actually use to run stuff.   There is more on this in this post.

Public credit

The idea for this tool actually came from very smart people at Sepago, Sascha Juch and Helge Klein, who I work with on Citrix Profile Management (the artist formerly known as Sepago Profile).   They also clarified for me that despite MSDN documentation claiming "system" attribute is necessary for this explorer magic to work, marking the directories read-only gets the job done.  Tested and true!  Thank you guys.

Update: Dieter Schmitz at Sepago is the gentleman who came up with the idea.  Thank you Dieter.

What does RadeCacheDecode do?

Application Streaming can store one or more execution Targets for each App Streaming profile. This level of indirection allows a single set of "Applications" and icons to be published even if more than one execution image is required.  This is common for a profile that supports both 32-bit and 64-bit operating systems. While the application is profiled twice to create two targets, there is only ONE profile to publish and only one set of "applications" from an Access Management Console perspective.  

A necessary consequence of this is that the execution targets have names that don't necessarily have any resemblance to the name of the profile from which they came.   Targets can be associated with execution systems for a variety of configurable options and these can change from version to version of the profile. 

That is, just because a specific target was right for Windows XP German yesterday doesn't mean that it won't also be right for English AND German tomorrow.   Even ignoring "one to many" advantages of execution targets to profiles, the streaming system STILL would use some abstract term to keep track of the execution targets. 

If you're editing a profile in the Streaming Profiler and you SAVE, all the identifiers of the execution units remain unchanged.  BUT - If you Save-As, you'll get new identifiers.   "By spec", the thing that was "saved as" is a New profile and will mature separately from the original - even if the before and after profiles have the same name.  Putting it a bit differently: Two profiles with the same name are not necessarily the same profile, the unique identifiers in the profile define the uniqueness of the execution units and in theory, or the profile itself.  The studious reader will see below where at least parts of this break down.The short version: Thanks to the use of GUIDs, the streaming cache manager will never confuse two execution targets, even if they come from different profiles that have the same name.

*What about the administrator?*GUIDs are there just to make the RadeCache space hard to digest.  After enough time looking at the Citrix internal showcase farm, you get used to the idea that the execution Target that starts with "1B3" is the MS Office 2007 profile execution target that is used for Windows Vista and Windows 7.  Type 1b3, hit tab and you get the right one.  Not ideal.  Here's what it looks like in the Windows explorer.

Okay, we know that 1B3* is the MS Office 2007 execution target for Windows Vista and Windows 7.  What are the other 5 execution targets? 

What you need is something to look up each GUID and to TELL YOU which profile it came from.  Yes, it would be BETTER if the streaming client itself had left you a clue, but it doesn't.  Notice that when the streaming client created these directories it KNEW what profile it came from and there can be at most ONE profile in the entire globe that has that GUID.  Did it leave you a hint, a small hint?  NOPE!.

Ahh, the fun life of being a XenApp adminstrator. 

HELP IS ON THE WAY!

The location of the application hub is generally "known" to the admin, but to know which GUID is associated with which profile, takes some work.  Once the running application terminates, not even the streaming client knows where the GUID came from.  It's just a directory that exists in the right place, but the link that says that the "1b3" target is somehow related to MS Office is gone, there's nothing there.  

Making it easier

For offline execution, the streaming system stores information that provides the secret decoder information required to convert GUIDs to profiles.  The Deploy space holds a copy of the Application Hub content and stores it in a place that has a fixed location (registry configurable and readable from the utility).  The RadeCacheDecode utility looks at the Deploy space, sees what execution targets exist for which profiles and uses this information to "decode" the GUID entries in the RadeCache.  This should make more sense by example.

Command line execution (readonly)

If RadeCacheDecode is run with the -r (readonly) switch, the utility processes the subdirectories of the RadeCache space, and for each tries to figure out what Deployed profile goes with that execution target.

Example output:RadeCacheDecode -r092b3450-e543-4541-837a-c374cc4e73cc_1 (GoView1067)

1b345768-25fd-45d4-b689-f3984f9221ee_5 (Office20071067)

274bd686-a305-45d9-a479-a127338586b1_1 (project20071067)

4d971270-cbac-4366-8c4a-b11f6f867a58_1 (AdditionalOffice1067)

76dc5736-0c5d-4600-a7e9-a3a41c3b5c51_6 (-not-deployed-)

a9984581-0e41-44b6-861f-b6d748dacb93_1 (Visio20071067)

Okay, we have now decoded the 4 of the 5 remaining execution Targets on my notebook.

Making it really useful!

Consider the technique that the Windows Explorer uses change each user's "Documents" directory into "My Documents".  This same activity can be used to permanently improve the readability of the RadeCache space. 

When thinking this was a good idea, I tried it out by manually creating files and while I was initially intrigued, my first conclusion was that it had negatives that outweighed the positives.  If you read the MSDN docs on the desktop.ini, it leads you to believe that the directories have to be marked "System" for this Desktop.ini stuff to work.  It isn't so.  Marking the directory readonly will do the job and with that, we have a solution!  Why is "system" bad?  Answer: it makes directories disappear from normal directory searches 

"readonly" works, so we have a solution.

This time, run the utility without the -r (readonly) switch and it will WRITE to the RadeCache space to mark all of it's subdirectories with "decoded" versions of the GUIDs.  The directories are marked readonly so the desktop.ini stuff comes into play with Windows Explorer and a file named "desktop.ini" is added to each execution Target's directory to create a "friendly name" to go with the mysterious true GUID_ver name.

The end result is that the next time the Windows Explorer browses to this space, you will see a nice version of the GUIDs - with the name of the profile placed in parens after the GUID. Sometimes it is necessary to force a refresh in Windows explorer to get it to reload the friendly names.

Here's a picture of the output

We're making good progress.  It is much easier to see these profile hinting entries than just plain GUIDs.

How does it work?

The entry above that says "not deployed" is the hint.  Since the Deploy space is at a known location, the decoder utility KNOWS where to look up the execution target GUIDs.  While the streaming client doesn't know what guid goes with what once the applications terminate, the DEPLOY space knows!  The Application Hub also knows, but unfortunately, the decoder utility doesn't know where THE (think plural) Application Hubs are located.  The Deploy space is fixed and given large offline streaming usage, this becomes pretty easy to inspect.

The utility loops through the subdirectory space of RadeCache.  For all directories that look like a GUID_v entry, it considers them an execution target and then goes looking for GUID_v content in the Deploy space.  When it finds a hit, the name of the containing directory in the Deploy space tells the decoder utility the name of the profile from which the GUID_v came.

Awesome!  Works good.

What's wrong with this picture

The studious reader will notice that the Deploy space is named based on the NAME of the profile that is deployed.  The Decode utility leverages this fact to figure out what profile a given GUID came from.

This naming in the Deploy space violates the premise that two profiles with the same name, but with different GUIDs are different profiles!  Dig into the internals of a .profile file (it is XML) and you will see that the profile also has a GUID.  GUIDs aren't just for guaranteeing uniqueness of the execution targets, they also guarantee the uniqueness of the profile layer.

Good news: the caching system is correctly using GUIDs and will "never" get confused.

The Deploy space SHOULD be done based on the GUID of the profile and all the sub-directories below the Deploy space SHOULD be GUIDs (with no version).  They SHOULD be, but they AREN'T and this means that the premise of two profiles with the same name, but different GUIDs are different is somewhat not precisely correct when considering offline content. 

In reality, this doesn't come up because admins tend to stick all of their profiles onto a single Application Hub and this means that the name of the profile is a unique key.  Still, know that this isn't right and know that I've been chasing a "fix" for this for over 2 years - so far, with no fruit.  The App Hub side is fine based on name.  The Deploy space should be GUID based.  Okay, I'm off my soapbox.

Enhancements

Good enhancements to RadeCacheDecode would expand the profile search to go beyond the Deploy space and also consult the network Application Hub.   The utility should also use the COM based SDK whose webpage it is posted in.  That is, the utility looks in subdirectories of the Deploy space to figure out which profile holds a file with the right name.  This breaks the separation of the profile as a programatic thing from looking behind the curtain.  I could have coded it that way, but it just seemed like a lot of work to open up a bunch of XML files to peek for GUIDs, when all I wanted to know was presence (is it there, or not?).

The "ultimate" enhancement to RadeCacheDecode is to change the streaming service itself so that when it creates RadeCache directories, it creates the desktop.ini file directly.  As of Streming Client version 5.2 (XenApp 5 Feature Pack 2), this does not happen - but now we know how...

To achieve almost the same thing, you could run RadeCacheDecode on a timer via some automated means.  It needs to be run with power so it can see and write to the RadeCache space, but if you do this, your desktop.ini entries should always be pretty much correct.  I coded the utility to WRITE only if the existing stuff is not "correct", so you can run it repeatably and the desktop.ini files will only be written occasionally.  This was probably overkill.

If you find it useful, let me know...

Joe Nord

Citrix Systems Product Architect - Application Streaming

The first releases of Citrix Application Streaming had isolation on the brain.  If you don't know what else to do, ISOLATE!  Don't let the app hit anything that might be important.  If you don't know if it's important or not, then it's important!  Protect it! 

This over protective behavior is being relaxed, the first step was in the XenApp XenApp 5.0 (Delaware) release where streaming client 1.2 was released which changed file system isolation to auto-ignore non-boot disk drives and soon, in the XenApp 5 Feature Pack 2 release where the version 5.2 streaming profiler and client change the default file system rule from "isolate" to "ignore".

Isolation rules review

When a user stores documents to disk under Application Streaming, the isolation system inspects the file operation to decide whether that operation should be isolated, ignored or redirected.  Details on each of these can be found here in the Citrix documentation library and here in my previous blogs.  All rules are named from the perspective of the streaming system, so "ignore" means don't change it - let it go to the file system without modification.

What isn't obvious, is what the isolation system does with disk volumes beyond the boot volume.  This behavior changed in streaming profiler version 1.2 (XenApp Delaware) so that the over paranoid behavior of the original Tarpon technology would instead leave users data alone.  Notice that I'm assuming users put data on drive D: while the OS is on drive C:.  This change has been beneficial to most customers and really helps engage the desktop, though I have seen a few that needed this to not occur and this oversight will soon be corrected to be backwardly consistent - for profiles created at the 1.1 level. 

To be clear, space beneath the user profile is and always has been "ignored", which means the normal places for users to store documents have no isolation collisions.  It's the non-normal places that require attention.

In App Streaming 5.2 which isn't out yet, the isolation will be further relaxed.

In addition to ignoring non-boot disk volumes, the isolation system will change it's default rule from "isolate" to "ignore".  Notice that this will happen only for NEW profiles.  If you have existing profiles, their behavior is already defined.  

Example user of Citrix showcase internal farm

Consider an example user, we will call him "Nabeel", because - well, that is his real name.  Nabeel is a Citrix executive and he travels a lot.  When he travels, he does lots of presentations and ... uses Citrix Application Streaming to run MS Office 2007.

In the example case, he was on a 3 week trip to Asia where he visited lots of Citrix sites and lots of customers.  With the magic of Application Streaming, he was able to use MS PowerPoint "offline" on the airplane to refine his presentations and also update the PPTs throughout the journey.  All of this worked super! 

Presentations updated all over the world, presented all over the world.  Everything was well received, and then he came home....

When he returned to Fort Lauderdale, he used Windows explorer to zip up all the files from the 3 weeks of presentations and e-mailed them out to the people he had met.  The receiving people noted that the presentations were mere skeletons.  CONTENT FREE if you will.  Everything worked on for many weeks was "as it was" 5 weeks earlier before he put Application Streaming onto the notebook.  OOOPS!

My phone rings...

What happened?  Answer: the files were all stored to the \Citrix2007 directory and since this is not a space that is "known safe", the isolation system isolated it.  From the view of the powerpoint application under isolation, it saw the correct and current version of the document.  But look at the real disk and the files were their original selves.   BAD.

Step 1: Where did the good files go?

That one is easy:

Consider the layers of glass.  The answer is that the file is in the top layer of the layers of isolation.  Find it in  %APPDATA%\Citrix\RadeCache\GUID\Device\C\Citrix2007

Step 2: How to make this not happen again...

Harder.  Insufficient answer is to add an ignore rule for Citrix2007 to the profile.  No good.

Not a problem for stream to server

Before going deeper, it should be said that this problem never comes up for stream to server.  In a server environment, "users" do not have the power to create directories off of the ROOT.  Users can store documents to %USERPROFILE% and that's about it.  Server side, this problem doesn't exist.

Stream to desktop

Client side, "users" tend to be "administrators" and this creates new problems.  "Users" like storing stuff to folders off of the root.  This makes it really easy to know what you need to backup.  

The isolation system not knowing how to handle the \Citrix2007 directory isolated the operations and this is far too over protective.

App Streaming 5.2 (Yellowtail)

In the upcoming release, the default file system rule for new profiles is changed from "isolate" to "ignore".  The default rule set then includes \Windows, \Program files and similar "important" directories as places that should be isolated at runtime.  I'll add that the profiling time rules and the execution time rules have to vary a bit to make this occur.  The streaming system takes care of this automatically.  Profiling time remains paranoid by default, isolating most everything.  Runtime becomes much more relaxed - making the isolated application execution more consistent in behavior with locally installed, while maintaining "protection" of spaces that the application should not be allowed to write per the layers of glass.

File system permissions still apply

If the user tries to write to a space that they aren't allowed, the file system permissions remain in place.  A user will only be able to write to or read from the real \Citrix2007 if they have DACLs in place that make this permissible.  This is a file system statement, not a isolation system.  In the "ignore" case, what happens is that the isolation system sends the I/O operation down to the file system without change and the file system will then decide if the user is allowed to access those files. 

This change greatly improves the offline streaming experience.

Joe Nord - Citrix Systems Product Architect - Application Streaming

Learn more about Citrix XenApp 5 Feature Pack 2

• Official Press Release - http://citrix.com/English/NE/news/news.asp?newsID=1857726
• XenApp 5 Feature Pack 2 Web Site - http://citrix.com/xenapp/featurepack2
• XenApp 5 Feature Pack 2 Executive Video - http://citrix.com/xenapp/fp2/video
• XenApp 5 Feature Pack 2 Webinar - http://citrix.com/xenapp/fp2/techtalk
• XenApp feature matrix by platform, version and edition - http://citrix.com/xenapp/comparativematrix
• XenApp Expert Series videos for this release - http://citrix.com/xenapp/fp2/expertseries
• XenApp 5 Feature Pack 2 Blogs- http://community.citrix.com/blogs/tag/xa5fp2
• Download XenApp technology previews - http://citrix.com/xenapp/techpreviews
• XenApp Product Page - http://citrix.com/xenapp/

Follow XenApp on | | |

I received an interesting inquiry recently.  Given my post on Never Deploy to XenApp Servers, how do you stop deploy from happening for hosted desktop?   Excellent question!

Here's the situation.  Happy XenApp Customer (they are all happy) who uses Application Streaming to deliver applications to their XenApp servers as well as stream to client, primarily for offline access.  The applications are published to the user as offline enabled and this works swell to trigger the deploy operation that is required to execute the application when disconnected from the corporate network and also works swell to not trigger Deploy in the XenApp hosted case.

Quick refresher on Deploy

The Deploy space is a COPY of all the needed stuff from the Application Hub, but it is local to the execution machine.  The Deploy space makes it possible for the streaming client to support application usage even when disconnected from the network.  The Deploy space is generally in \Program files\Citrix\Deploy and the execution cache space is generally in \Program files\Citrix\RadeCache. 

At runtime, the streaming client runtime fills the execution space as needed by copying content from the Application Hub - or if Deployed, from the Deploy space.  Other than one being remote and the other being local, the activity of the client is the same whether "offline" or "online".  Technically, "Deploy" enables streaming from one side of the hard disk to the other and this keeps life pretty simple from a programming standpoint.   I thought it up back in the Tarpon development and I'm still pretty happy with myself.

Deploy is automatically not done on XenApp Servers

It may seem like a good idea to Deploy to the XenApp Servers, but I have already squashed that myth and apparently did a good job there because the corollary, "Never Deploy to XenDesktop" has also been heard.

Back to the customer.  They have XenApp running good and are engaging XenDesktop.  Application Streaming is delivering the applications to the XenApp servers as well as to the end user machines (notebooks). 

Here's the thing that's not always obvious.  The streaming client KNOWS that Deploying to XenApp Servers is a bad idea.  If you publish an application to a bunch of users and offline enable the applications AND if you publish the applications to hosted execution, a single user can use both without triggering a Deploy on the XenApp server.  

If the user is running the application server side, the streaming client is still involved but it knows that it is running ON a XenApp Server, so it SKIPS the Deploy.  This is the right thing to do.  If the admin doesn't like that, they can still command a Deploy to occur on the server by directly calling the deploy utility, RadeDeploy.exe, but again, you shouldn't do that.

The problem

The customer is now engaging XenDesktop and have connected the hosted desktops to the App Streaming infrastructure from the existing XenApp configuration.  Here, the streaming client is running on the hosted desktop and ... when performing its app launch concludes that this is a client side execution and background copies down all the bits for offline execution. This is bad.

Here's a good write up on the percentage of space that is used for online vs. offline execution.

The point: Lots of disk writes start happening to copy the Deploy content down from the Application Hub to place them in the Deploy space where they will exist and get copied again into the RadeCache.  All of this is bad to the layers of cake.  We want to minimize disk WRITES.  Disk writes are backed up either by Xen/Other Virtual Machine manager or by Provisioning Services.  Either way, large writes by large numbers or users are bad.  Beyond being slow, these writes will occupy space in the per-user write back cache; which is discarded on logoff, so this whole thing will repeat on the next logon.  

The solution

The Deploy should not occur!  But it does.  In a "correct" world, the streaming client would be XenDestop aware as it is XenApp aware.  If running "hosted", don't Deploy!  We do not live in a correct world.  My initial guidance to the folks that asked me about this was "erase \program files\citrix\streaming client\radedeploy.exe".  All happy with myself because I KNOW that the streaming client shells to the utility to perform the deploy.

With the utility "missing", the streaming client will have no choice but to push on without the deploy.
Survey says.... didn't work. 

Instead of "pushing on", the streaming client declared failure and aborted the app launch.  Aargh.

Beautiful part of this is that I didn't have to actually try it.  Smart folks on the other side, all I had to do was keep coming up with ideas.  

Second round: Replace RadeDeploy.exe with a program that does nothing, but returns "success" return code.

Survey says.... Success!

The only utility I had handy to do this is one I wrote a very long time ago to use for remarking out RUN= statements from the Windows registry.  Source is included - please compile it yourself before putting in production.  For the programmers in the room, said program is "Hello world" minus the printf.  In the common image where the streaming client is installed, copy nop.exe to RadeDeploy.exe. 

Back to the Deploy activity

The streaming client "shells" to RadeDeploy.exe which is really "nop.exe", which returns 0 as RC.  With the Deploy "successful", the launch proceeds with no error, AND the Deploy never happens.  Perfect!

I see a future where the streaming client will be XenDesktop aware and skip the Deploy.  Until then, this technique may prove useful.  If you have other creative solutions to this problem, please share them here.

Next step is getting the RadeCache space available to everyone from a single source, read-only and fully populated at the get-go.

Joe Nord
Product Architect Application Streaming

Citrix Systems

The Citrix Application Streaming client and profiler version 5.2 are available in Beta form.  Here is a link.    MyCitrix credentials required to download.  The Streaming Client is "Offline Plug-in version 5.2". The Streaming Profiler is "Streaming Profiler version 5.2".  

These are not production, but they are pretty close to what will ship with the next XenApp release.  I note that the Streaming Client/Profiler 5.2 can be used with everything from Presentation Server 4.5 to XenApp next release.

Please give it a whirl.  The normal tech support forums should be used to share your positive experiences.

The 5.2 client will "consume" profiles created by all previous streaming profilers and the performance improvements noted below will be seen with no need to reprofile.  That said, there has been a years worth of changes in the profiler, so there can be other reasons to move up.

Things you will see:

• Second time application performance vastly improved.  Feature: "Sandbox reuse".  In the past, launching two applications from a single profile actually created two sandboxes even though those sandboxes could be considered one as they both had the same definition of isolation scope.  Starting a sandbox is a long activity, that is now skipped for second application launches, making second app launch very similar in time to locally installed.  This change was the focus of the release and should produce very happy users. 
• Sandboxes are left alive for 5 minutes after the last application terminates, improving odds for 2nd time launch compared to first.  Registry configurable period, in minutes.  Might change this to Ms.
• Integration with the Citrix Receiver to operate as a plug-in, or as a separately installed program.  This function first shipped in 1.3 which was a streaming client shipped asynchronous to XenApp releases.
• AIE* environment variables are now visible from pre-launch/post-exit scripts run outside of isolation.  This enables things like purging the per-user isolation space before/after application usage, without using RadeRunSwitches.  Some other issues with scripts were also fixed.
• A years worth of App Compatibility improvements.

The version number will be 5.2.  If you've seen my posts before, you know that I never intended to move away from 1.x, including shipping major new function in point release, like Inter-Isolation Communication at release 1.2.  Long story as to why, but the bottom line is that the streaming client after 1.3 will be 5.2.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems

I have previously written about how the Application Streaming client prunes the file system cache.  What was not discussed is how the streaming system manages the REGISTRY portion of the execution content.   This post brings attention to that need and provides a reference to a source code application that can assist with pruning the registry portion of the execution cache.

When the streaming system executes an application, it can run more than one version of the application at the same time.  New versions of the app can be used by user B when user A is still using the old version.  This way, the admin can update an application "hot" and when users closes and restart the application, they get the new version.  This covered in some details in the discussion of RADE.

What is not obvious is that while the streaming system file system cache has a high water/low water algorithm for managing the cache, it has no such algorithm for the REGISTRY content.  Peek into HKLM at the RadeCache space and you can potentially see registry content supporting MANY versions of old applications.   

Here's the execution space on the machine I am using to write this post.
Notice in my case, I'm on version "5" of the execution Target 1b3... which happens to be Microsoft Office 2007, the Vista/Win 7 execution target as published in the internal "showcase" at Citrix.  Also notice that there is no "4" because I have a strange habbit of formatting my primary development box once a month whether it needs it or not and the admin hasn't updated the profile since my last reload.  I also know that our showcase admin occasionally does admin magic to delete these old spaces, but that is getting away from the purpose of this post.

Real customers

Heard from the field a couple months ago of a customer who had taken RADE to true heart; updating applications almost daily and then streaming them to server.  After a few months, they were in the hundreds in the versioning, and while this worked fine, the registry had become polluted with all the old versions.  Notice that the file system was fine, the old versions had aged themselves out of the cache and been erased, but the registry was littered with stuff that should not be there.

Utopia

One can effectively and easily argue that the Streaming Service should observe application updates and when nobody remains using the old versions, it could happily perform house cleaning to purge out the not needed information from the installation image registry.  This would be a very good argument and I'd stand up and say, you are right, which is what happened with this customer.  Things like Rollback to earlier versions makes this more complicated, but in the general sense, managing this registry space should be no different than managing file system space.  It is more difficult though because for file system, there's an advantage that anything you erase by mistake, will automatically be put back if it is needed.   For registry, it's populated once and left alone.

To "solve" the problem, I coded a little utility to prune the registry cache.  It runs outside the confines of the streaming service and defaults to "keep 2", to enable rollback.  The SOURCE code for this utility has been posted to the Citrix Developer Network, MyCitrix account required to get in.

One can imagine a day where this function is part of the streaming service.  I imagine that day too.  Until utopia arrives, if you need to programatically prune the registry cache, this utiltiy can help.  Set it up to run once a day/week and it will keep your registry space clean.  The UI is pretty primitive, it shows to stdout a before and after record of all of the space in the RadeCache along with indicator of stuff that it erased.

What it doesn't do

The utility scans the registry content, observes all of the cache space and then recursively deletes the "old" entries.  What it doesn't do is know which GUIDs are orphans, supporting applications that are no longer published.  Interestingly, the streaming service also doesn't know which entries are no longer used.  It would be a worthy addition to also reference the "last access time" information and purge items that are over a limit.  Here, shelling to the RadeCache utility to perform the erase would be the correct action so that the official tool could be used to purge our the execution cache, both file system and registry.

Enjoy, 

Joe Nord

Product Architect - Application Streaming

Citrix Systems

Citrix Application Streaming runs the majority of the isolation system as a service.   Being good security citizens, the service (radesvc.exe) is run on a named user account (Ctx_StreamingSvc) rather than LOCAL_SYSTEM.  There's some fancy security name for this but what it comes down to is using the least privilege you can to help limit the potential damage you can cause should you go astray. 

The user Ctx_StreamingSvc can write to only a few locations on the machine.  As a user account, it automatically cannot write to \Windows or \Program files and cannot write to HKLM, or even HKCU for any user other than itself.  This is good.  There are some things however that just demand a service, such as populating a SINGLE application installation image no matter how many users will be logged onto the machine.

At installation, the streaming service user account is given full control of a few locations:

• \Program files\Citrix\RadeCache
• \Program files\Citrix\Deploy
• HKLM\Software\Citrix\RadeCache

These relate to the "middle layer" of the "layers of glass".  They are the SINGLE space that support multiple application executions across many apps and many users.  The ability to write to these spaces is needed to runtime populate the execution content, so this is pretty much a "must have" for the streaming service to do it's job.

Where it gets neat is is that real USER accounts have to "see" stuff in the cache so that they can layer in the isolated content, so that the layers of glass can support read-only opens to the stuff in the installation image.  In the normal Windows installation, users, read: not admins, can "see" anything beneath \Program files, so in theory, there's no action needed to pull this off. 

Being security paranoid folks, the Citrix isolation system restricts users from seeing execution content unless they are actively running the application.  The DACLs are runtime added and removed.  Here is a file system view of the DACLs applied to the RadeCache execution space, first for the service account and then for a user account.

RadeCache directory

. NT AUTHORITY\SYSTEM:(F)
  NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(F)
  BUILTIN\Administrators:(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(F)
  CREATOR OWNER:(OI)(CI)(IO)(F)
  machinename\Ctx_StreamingSvc:(F)
  machinename\Ctx_StreamingSvc:(OI)(CI)(IO)(F)

That part not too interesting.  

The GUID_v directory that holds the application content is below the RadeCache.  The below taken from my Windows Server 2003 test machine while running an application as user "U1". 

. machineordomain\U1:(RX)
  machineordomain\U1:(OI)(CI)(IO)(GR,GE)
  NT AUTHORITY\SYSTEM:(I)(F)
  NT AUTHORITY\SYSTEM:(I)(OI)(CI)(IO)(F)
  BUILTIN\Administrators:(I)(F)
  BUILTIN\Administrators:(I)(OI)(CI)(IO)(F)
  CREATOR OWNER:(I)(OI)(CI)(IO)(F)
  machinename\Ctx_StreamingSvc:(I)(F)
  machinename\Ctx_StreamingSvc:(I)(OI)(CI)(IO)(F)

The thing to observe is that the user rights for the directory space are runtime added when the application is launched, and runtime removed when the application terminates.   Even while the application runs, the user cannot see the RadeCache directory itself.

Here's how the user sees it.

C:\Program Files\Citrix>dir radecache

Directory of C:\Program Files\Citrix\radecache

File Not Found

Same user, this time "look deeper" to see the directory space that supports the execution of the application that they are currently running.   You have to "know" the GUID to have this work because you can't directory scan the RadeCache.

C:\Program Files\Citrix>dir radecache\0561beac-e4b8-47d8-8a71-fe5752f330f9_3

Directory of C:\Program Files\Citrix\radecache\0561beac-e4b8-47d8-8a71-fe5752f330f9_3

08/07/2009  10:31 AM    <DIR>          .
08/07/2009  10:31 AM    <DIR>          ..
08/07/2009  10:30 AM    <DIR>          Device
05/06/2009  06:23 PM                 2 FontData.dat
08/07/2009  10:43 AM            24,340 Hashes.txt
05/06/2009  06:23 PM         1,295,137 InstallRoot.tab
05/06/2009  06:23 PM                 2 Preextract.txt
05/06/2009  06:23 PM            28,265 SandboxData.xml
08/07/2009  10:31 AM    <DIR>          Scripts
08/07/2009  10:43 AM            35,565 TempSandboxData_2b264684-41d7-40eb-9c69-b
cb32beba720.xml
               6 File(s)      1,383,311 bytes
               4 Dir(s)     662,564,864 bytes free

In the words of Bill Cosby, "That's kinda cool!". (1:54).  The user cannot see the RadeCache directory, but they CAN see the subdirectory.  This gets more fun when you change dir into the GUID_v directory, which succeeds, and then "cd .." to get to the parent, which fails.

Why bother?

1) Citrix Security people are paranoid and just want dev folks to write extra code

2) Prevents users from even seeing applications that are not published to them.

The answer is "2".

XenApp Server side execution

All this DACL stuff happens whether stream to client or stream to server, but the dacl benefit is important mostly for the server side case.  If there are 50 users on a XenApp server and only 10 of them have access to "very secret application", then only that specific set of 10 people will be able to even SEE the executable, much less run it.  Compare this to locally installing the "very secret application", where users can see the program files space for the application even if it isn't published to them.  In the streaming case, they can't see it in program files, and they cannot see it in the RadeCache. 

What about the file server?

The Application Hub holds the executable content and making things secret means that users must be restricted from seeing things on the Application Hub in addition to the execution machine.  Admins I have queried on this have replied "simple, I use the same groups to publish applications as I do for granting read access to the sub-directories on the Application Hub". 

This is why the profiles are all stored as sub-directories off of the root of the "profiles".  Everything related to a given profile is in a single directory.  This creates an extra step to find the .profile in the streaming profiler file open dialog, but it really helps to simplify the DACL management.  This is also why the Streaming Profiler doesn't let you enter the full path to the "save to" location.  The use of "profilename\profilename.profile" is required for this to work, so the streaming profiler prevents you from using other forms when saving.

Joe Nord

Product Architect - Application Streaming

I received an inquiry recently; can Application Streaming be used to virtualize server infrastructure apps, like Apache, SQL server and a variety of others?  I've heard the inquiry before, but have so far ignored it.  I write this post to find out if I'm somehow missing the point because as the Citrix Architect of delivering applications virtualized, I don't see any point in isolating these applications.

My response: Sure, use Provision Server.  Don't like that one, use XenServer.

Let me put it differently.  COULD you "App Streaming" style isolate the execution of a server side infrastructure application? SURE.

Should you?   In my view, no.  Why do it?

The entire PURPOSE of the server is running that application, so why isolate the application execution.  Well, it isn't really an application, the entire purpose of the machine is redirected to serving this single purpose. 

Could I STREAM the application bits in just time time for execution?  Sure. Would I ever do that?  No. I would go out of my way to make sure the execution bits were in place before runtime.  That leaves isolation. 

What about application updates

RADE provides for hot updates of applications as users run them.  In the case of server side, I could profile the application in one place, prove it works, then reboot the production servers to have then bring in the new version of the application.  This sounds like a perfect use of Provisioning Server. 

Maybe I could get the new version of the app onto the server by closing the app and relaunching it; no reboot needed.  The result is the same result, the web/database server still has to go down to come back up.

Notice this is the App Streaming architect saying "Don't run these apps with my stuff".

Okay - IT people, tell me what I'm missing.

Joe Nord

Citrix Systems

Product Architect - Application Streaming

Last week at internal Citrix Sales Engineer gathering, I was presenting stuff on Application Streaming, where I noted that Application Streaming cannot isolate 16-bit DOS applications.  Immediate response from the SEs, yes it does - I have it in production and it works.  ... Things that make you go hum.  

I went home and tested it.  Answer: The SEs are completely right.

Why?  Our file system driver is kernel mode and "sees all".  Same for the code that looks after processes coming and going.  There would be a problem with registry filtering and this would be a show stopper, except there is no registry for DOS apps.   So, I've been telling people for years that this is a product limitation, when it really isn't.

Here's a picture of Turbo Pascal, 1987 vintage, running under Citrix Application streaming.  Also note that I saved Hello.pas to the C:\Turbo directory, which shows up in per-user layer of isolation.

Enjoy.
 

Common Application Streaming question:

How much data lands in the execution cache compared to what is captured at profiling?

My quick answer to this question is 25 to 30% of the captured content will land on the execution machine at runtime.  A more correct answer is ... "it depends".   People like numbers and they need numbers, so we give them numbers.  Statistics though can be misleading.  Are they good numbers? Sure, absolutely! Read more to understand some of the complications.

To get it started, this is the wrong question.

What people really want to know is how much disk space is used to deliver this application via Application Streaming as compared to delivering it via local install?  Answer: 35.6%.  Again, read on for more details because the real cost is a bit lower than this number and gets back to that 30% space, or the real cost is quite a bit higher than this number depending on whether you're going to go offline or not.

Application selection

The selection of the application here greatly changes the outcome of the question and deliverying for OFFLINE really changes the calculation.  First, consider online execution.  If the application has only a few files, all of which are accessed at runtime, then the isolation system will populate 100% of what was profiled.  By contrast, if the application has thousands of files, most of which it never accesses, then the answer approaches zero. 

The key point is that files are brought in when they are ACCESSED.  If they are not accessed, the isolation system LIES to the application to tell it that the stuff is present when it really isn't.  When it isn't, it doesn't use any disk space and it doesn't occupy any network time being copied in from central strore. 

It is amazing to watch how much stuff applications install that they NEVER, EVER reference.

NUMBERS

The measurements that follow are taken from my primary notebook machine, running non-release Windows 7 RC1, with Citrix Application Streaming, non-released development "tips" build from about 2 days ago that flunked build test, but which seems to be working pretty good for me.  I'm connected to the "showcase" farm inside Citrix, which is where we put the stuff that isn't released yet to "use what you sell".  Hundreds of people using this by the way.  I'm running a non-release and pretty current PNAgent which I haven't changed since before Synergy conference.  This is about as "stable" an environment as I get.  Other than these items, my environment is pretty typical and should be fine for statistics.
Why Microsoft Office 2007?  Answer: It's a really big application and everyone understands it.  Is it an example of a typical application?  I'm not sure, but it is sure an excellent testcase for proving out an isolation system.

Framing the problem
There's more to it than what gets populated into the execution cache; offline/deploy and online are also part of the equation.  If the execution is "stream to server" then there's no need to deploy a copy of the execution image on the execution machine.  Disk space usage will be optimal, the low number.

For "stream to client", offline will be common, so disk usage will be higher as you have to store a COPY of everything that is on the Application Hub that supports your execution.  This storage though is a compressed version of what was captured at profiling, so it is reduced in size and you only get a copy the execution target that is right for your machine, so the size used for deploy will often be less than the disk space used on the server that stores the execution content.

Application Hub:
Our showcase farm is consuming 27.2 GB of disk space to store MS Office 2007, on the server.  Much of this is wasted space because no house cleaning has occured.  There are for example 5 version of the execution target stored which support Vista/Windows 7 and there only needs to be 1.  My admin could save 80% of this disk space and nobody would notice.  By spending the disk though, he retains the ability to perform rollback. 

LOCAL MACHINE
Deploy: 1,161,958,306 bytes. 

Notice this is a WHOLE bunch smaller than on the App Hub.  I'm pretty sure our admins flush out the old stuff to help keep this number small, but even if they don't, I format and reinstall machines often enough that this doesn't come up.  The deploy location though is COMPRESSED.  We install everything and a bunch more when profiling for the showcase farm.  The expanded number for the size of the packaged MS Office is 2,125,243,930 bytes.

RadeCache populated execution space = 321,401,139 bytes.

Drum roll for the math

Online: 321,401,139 /  1,161,958,306 = .2766 = 28% OR

Online:  321,401,139 / 2,125,243,930 = 15% (That's a nice statistic!)

Offline: (321,401,139 + 1,161,958,306) / 1,161,958,306 = 128%
Did you just say that offline streamed uses MORE than locally installed?  Yes.  We made a conscious decision for this actually.  The extra disk usage allows online and offline streaming to be largely the "same" from the view of the isolation engine.  Technically, this is called "deployed" or "not deployed".

Compare to locally installed.

I had to struggle, but I found a Citrix PM who is still has MS Office 2007 locally installed where I could get some numbers.  His machine has 904MB, 904,155,136 bytes consumed.  Notice that this is smaller than the 1.1GB that is in the CAB file that our showcase farm has.  This is expected.  Our build has some other stuff added beyond the base MS Office and also was profiled with the of MS Office install selections, selecting "everything". Among the nuggets that are "extra" in the profile are a full copy of Mozilla Firefox.  I'm not sure what it's doing in there, but adding these things does make the profile grow.

Using these "common user" numbers for locally installed and comparing to the showcase MS Office 2007 profile, we can calculate a second statistics.

Online vs common user: 321,401,139 / 904,155,136 = 35.6%

Offline vs common user: (321,401,139 + 1,161,958,306) /  904,155,136 = 164%

Fascinating numbers.  What good do they do me?  I'm not totally sure.  If you operate on the theory that streamed delivery will use about the same disk space as installed delivery, you're in the right ballpark.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

User32.dll is a magic item.  As a programmer, if you want to take the machine over, then user32.dll is your best friend.  User32 is a system DLL that gets loaded into all programs, system and user, that do anything with the GUI.  User32 has a nice side benefit that it also loads other DLLs, by name.  The list of DLLs to load is stored in the registry in a string item, AppInit_Dlls.  Yes, this space is only writable by privileged proceses, but if you can get yourself on the AppInit_Dlls list, you're golden!  This is so handy that it is a common method that viruses use to attach themselves to all the processes on a system and ... is how application isolation systems like Citrix Application Streaming do their work.

Notice above, I said ALMOST all processes link to user32.dll.

There are many processes which do not load user32.dll and if they don't, then things that load as part of AppInit_Dlls will not get loaded.  If you're in the application isolation business, this is not good because it means that you can't isolate that application. 

A common question - Does Citrix Application Streaming depend on the application loading user32.dll in order for the isolation system to hook the app's execution?

Answer: No. 

The more elaborate answer is that AIE on Presentation Server 4.0 did depend on the application loading of user32.dll, but Application Streaming does not have this limitation. 

Propeller talk on user32.dll. 

If you want to know more about hooking processes and user32.dll, here are some good and entertaining references. 

• User32.dll is really important
• AVG Virus scanner incorrectly recommends erasing user32.dll (Awesome!)
• AppInit_DLLs should be renamed Deadlock_Or_Crash_Randomly_DLLs
• How virus loads
• Microsoft Malware Protection Center blog (good reading)

Probably best for a separate post, but the second item here is really interesting.  If you think you have a virus and do a google search for "user32.dll virus" you'll get 574,000 hits!  Sometimes, it seems like the "fix" for virus is worse than the virus itself.  If you delete user32.dll, you're up a creek with no paddle!  You can hope that the Windows XP system file protection will put it back, but it's still a scary proposition. 

Consider that if you are evil, and you're inserted into system code, then the obvious next step is to hide from anti-virus.  This must be an interesting battle. 

Enjoy,

Joe Nord

This is"part 2" in a propeller-head series about the internal workings of Citrix Application Streaming.  This post covers how stuff in the registry can be "erased" during profiling without really effecting the profiling machine.  The same concepts apply at runtime where the application can erase things from the machine and have this effect only be apparent for the current user rather than the whole machine.

Part 1 covered the same topic for files.

Consider: Machine has registry space at HKLM\Software\DeleteMe (yes, notice the easy to recognize naming).  In this example, there are also a few registry items located in that space, abtly named, item 1 and item 2.

The installation program observes that this space exists and as part of its installation activity, erases it.  The job of the isolation system is to let the installer BELIEVE it erased the space, while not really erasing it, so that the streaming client can present this space as "not there", when it really is.

For the installer, I have used CMD.exe and here's the activity, captured as text.
c:\>reg query hklm\software\deleteme

HKEY_LOCAL_MACHINE\software\deleteme
    Item 1    REG_DWORD    0x1
    Item 2    REG_DWORD    0x2

c:\>reg delete hklm\software\deleteme
Permanently delete the registry key HKEY_LOCAL_MACHINE\software\deleteme (Yes/No
)? y
The operation completed successfully.

c:\>reg query hklm\software\deleteme
ERROR: The system was unable to find the specified registry key or value.

From outside of isolation, go look at the "real" registry.  You'll see:
What happened? 

Answer: The true registry was not "hurt" by the profiling activity.  The registry space that was deleted wasn't really deleted.  The captured profile though believes it was deleted and when moved to the execution machine, the Streaming Client will do similar isolation stuff to make this space appear "gone" even if that space really exists.  The "how" follows.

Package the app up and peek into the captured registry contents and you'll see the magic that makes this happen.  Recall from the file discussion, NTFS Alternate Data Streams are attached to the deleted files to describe them as erased.  The isolation system does similar activity for deleted registry.

Notice that the deleted registry space "exists" as part of the captured registry space.  It exists, but there is also this extra stuff that the isolation system references to conclude that this regsitry space is "erased" and should be masked from vision to the application.  In the case of the registry as well as for files, the erased markers are attached to the erased stuff so that the isolation system doesn't have to look far away to index things that are erased.  When a registry key is opened, the isolation system looks to the side to see if the marker is present and if it is, it "hides" this registry key from the view of the application. 

Magic?

I called this magic above.  In reality, no magic.  If the application happened to use and depend on a registry item named CitrixAIEDeletedStatus, this whole thing would not work.  Fortunately, this is a rather unique name and this makes it okay to add to registry space, where the application won't be impacted.  As a last step, the registry items that are the markers "don't exist" from the perspective of the isolated application, so this prevents confusing an application by giving it extra registry items that it did not define.

What is the other Citrix thing

Good question.  The CitrixAIEPlaceHolder item exists so that the isolation system can transport registry KEYS that have no contents.  Compare in concept to XCOPY /S and XCOPY /S /E.  Some application create registry keys (similar to file system directories) and the mere presence of the key has meaning to the application even if that key has no contents.  Since the isolation system stores and later repopulates the registry space in a manner similar to .reg files save and restore, keys with no contents get lost.   By providing an item, any item, the empty registry key is preserved from profiling system to execution system and the application "works" because it sees what it expects to see.

Disclaimers

Lots of bit head stuff here on how the isolation system works.  Don't become dependent on it.  We change these things from release to release and it is likely that the method of representing deleted registry or empty registry can and will change some day in the future.

But - if you're looking at your machine and are asking yourself: What is all the Citrix stuff?  Now you have the answer.

Joe Nord

Deleted files are an interesting topic in isolation systems.  How do you represent a file as "gone", when it is still present?  During profiling of an installation program, the isolation system has to isolate additions to the file system; this part is easy and pretty easy to understand using the "layers of glass" metaphor.  You look down from above and the highest version of the file is the version that you see.  What is more interesting to consider is that the isolation system also has to represent DELETED files; and deleted registry content.  These require placing a marker at higher levels that say to mask vision to the content that really does exist.  Deleting files is much more interesting than adding files!

Consider a view from above, looking down through the isolation system goggles:
 
I used the IIC version of this graphic because it was handy, but consider the above with only 2 layers as this is the environment during profiling.  The PHYSICAL or "REAL" disk/registry is off limits for writes.  When the installer during profiling, or when the program at runtime writes to the isolate space, the changes are reflected as writes to the higher layers of isolation (the first "writable" space, which is always on the top).  This works great for adding content because the presence of the content at the higher layer "masks" vision to the layers below.

For deleting things, this gets interesting.  Consider a program at installation which erases content from the physical machine.  Looking down from above, the installer needs to BELIEVE that the erase it commanded, really happened.  Here, the isolation system has to represent "deleted" files and registry content without really deleting the stuff from the true disk.  Failure to do this would require a VM snapshot thing and resetting of the environment before/after profiling and we didn't want that.  Instead, you can profile over and over all day long without hurting the true physical image - at least for the spaces that are isolated, which is pretty much everything during profiling.

A deleted file is represented as a real file, at a higher layer of isolation.  The real file is changed to have 0 size, but still has the same filename as the lower layer file.  The filesize reduction is just to conserve disk space, but in concept the deleted file is the same file as the lower layer; promotion from physical to isolated occurs commonly for files "changed".  To make the file "deleted", the isolation system tags the file with a NTFS Alternate Data Stream, "CITRIX_DELETED_FILE_MARKER".   If the file exists on the layer of glass, but has zero size and the required tag, then the file "disappears" from the perspective of isolated processes.  The file still exists and is even visible outside of isolation; though it still has no contents.

References:

Forked file systems (Mac)

Scary reference - ADS are evil  IMO, They are not evil, they just aren't widely understood.

Microsoft KB describing how to use ADS

Sysinternals tool for displaying Streams

The fact that ADS are rarely used by "normal" Windows applications makes them a PERFECT candidate for usage in the isolation system.  The isolation system can attach a stream to the file and feel pretty confident that this won't hurt the applications ability to manipulate that file, yet the information associated with that file hangs out and even follows the file from place to place when it is copied.  Note: this isn't exactly true - keep reading.

Profiling

During profiling, the isolation system sets the marker on files that the installation program erases.  In concept, this means that the marker is present on the execution machine at runtime because the captured layer of glass is transported to the execution machine.  In reality, this is true only in concept.  In reality, we use CAB files to hold the layer of glass and CAB files do not maintain streams.  Hum.  Problem.  What did we do?  Answer: When the application installer completes processing, the profiler goes through and "deletes the deleteds"; removing the files with zero size and the stream attached.  This means that the deleted file will not be present in the captured image, but it also means that the marker will not be present at runtime.  Problem?

First, why is this a risk?  If the file existed on the profiling machine (at true disk location), and if the installer erased it; then on the execution machine, if the file exists on the true disk, then at application runtime, the erased file will re-appear.  As the theory goes, the application at runtime will get unhappy.
Is that a problem?

At the time, I wore the FSFD dev hat and took note that darn near zero application installers actually erase things from isolated spaces during profiling.  Those that do, likely will also tollerate that the file comes back.  We also took note that even if the file did re-appear, we could always erase it again using a pre-launch script.  Bottom line: Application Streaming has shipped this way from the beginning and I have never seen a failure attributed to this behavior.

But wait, there are more problems

At runtime, the application may decide to erase things.  In general, the only spaces isolated are the \windows directory and below and the \program files spaces or more precisely, these are the locations where an ill-behaved program may want to erase things; things that could effect the execution of the program.  

It could also be that the application erases some of its configuration data from the directory to which it was installed (a bad application). In either of these cases, a deleted file marker will be generated and will land in the per-user layer of isolation.  This will mask vision to the file on the true disk.  Since this is part of the per-user space, all should be happy because the per-user space will follow the user from machine to machine as part of roaming profiles or even better, Citrix User Profile Manager!  The Win32 CopyFile API DOES maintain Streams!

Problem: Roaming profiles do not maintain ADS Streams, so the deleted markers will disappear on movement from machine to machine.  4 years in the field, has this been a problem?  Nope.  

I have asked the Citrix UPM team to maintain Streams on copy of the user profile content to/from central store which is a bit of a curiosity because it is different behavior than Roaming Profiles.   I'm not sure if it occurs yet and not even sure if they will actually implement the change.  Why?  Fundamentally, the ADS streams haven't been missed yet, they will likely be missed less in the future; except for all them MAC users.  Stick with me.

What file systems support ADS Streams?

Excellent question!  The answer is "NTFS".  The Mac file system REQUIRES streams, but we rarely get the mac file system as the foundation of a Windows machine.  FAT32 definitely doesn't support streams and I'm not sure about CDFS.

Since deleted file markers are needed for isolation processing, the Application Streaming file system code restricts it's filtering to only NTFS disk volumes.  In some ways, this is a plus.  You don't want to isolate USB drives and these tend to be FAT32, so they don't get isolated.  But, the FSFD knows the USB drive is "removable" and it assumes all removable media is for "data", so it doesn't mess with it.  In this way, the NTFS restriction actually gets in the way a bit.  Note: you can convince the FSFD to look at the USB drive as I outlined here.

What about more elaborate file systems?

Some customers have enterprise storage which is network based, but appears to the execution machine as local.  These disk volumes COULD be NTFS, but often they have a custom file system.  Do these support NTFS style ADS streams?  Answer: It depends.  Let's say they don't.  How to solve...  Answer: Convert the representation of "deleted files" from ADS Stream to something a bit more common.  Has this happened? Nope.  Is it really necessary?  I'm not sure.  Experience of the last few years says it is rare enough that the failure case probably won't come up.  With this wide post though, something tells me someone is going to say: "I've seen the failure case".

In a follow up post, I'll describe how the registry "deleted" markers are implemented.

Enjoy.

Joe Nord

Product Architect - Application Streaming

Citrix Systems

How do you add FTAs to profiled applications that the installer forgot to install?  Answer: There are a few steps.

This has been on my list to Blog about for a while, just haven't gotten around to writing about it, mostly because  I hadn't had a chance to verify the solution would actually work.  Conveniently, Karl Muller of I-Access took care of the trial for me, and it worked, so here's the solution.  Thanks Karl.

Review

Backup a couple steps... "FTA what!"

When an application installs, it writes entries to the machine to establish associations between file extensions (multiple) and programs (single).  Each file type definition also includes a reference to the ICON that goes with the extension. This is what Windows Explorer uses to show you icons that looks like MS Word for file's that end in .docx.  

The Streaming Profiler runs the installation program under isolation and blocks the installers attempt to set FTAs on the true machine.  As part of post processing, the profiler examines all the FTA entries that the installation program TRIED to create and for each of these, creates corresponding FTA entries in the streaming profile XML data - you can see them in the .profile file, if copy to .xml and view with web browser.  Easier, just look at the properties of applications captured during profiling and ask to see the FTAs.  Yes, that's much easier.

The Access Management Console (XenApp publishing console) reads the profile information and gets a list of all the captured FTAs.  It makes these AVAILABLE for association with applications that you publish.  The default is that no FTA will be associated with the published application, so as an admin, you have to go to direct action to establish the FTA by clicking on boxes.

BUT - There's a catch, you can only establish FTA reference for the FTAs that were captured by the streaming profiler.

Say you know that .DOC9 (.DOC and the number nine), is SUPPOSED to be associated with MS Word, but you note that the installation program seems to have left that one out, or that the streaming profiler seems to have missed this one.  Let's focus on the first one.  You have an extra FTA that is SUPPOSED to be associated that seems to be missing from the list.  How do you add an extra FTA?

Option 1: Streaming Profiler, application properties, FTAs, Add

Option 2: In the Access Management Console, application properties, FTAs, Add 

Both options above would be GREAT, if only they worked.  They are both READ-ONLY.  You can see the FTAs, but you can't add a new one.  
The solution...

FTA must be captured during profiling

When profiling, tell the profiler that you want to launch an installer, it will prompt you for the path and name of the installation program, tell it "CMD".  When launch the installer, you will get a command prompt running that is running inside of isolation.  Two commands are needed at the command prompt to establish an FTA.

• assoc .doc9=Word.Document.12
• ftype Word.Document.12

If the ftype is already in place, you're done.  If not,

• ftype Word.Document.12="C:\Program Files\Microsoft Office\Office12\WINWORD.exe" /n /dde
• exit

Now, let the streaming profiler do it's thing to see what the "installer" did and it will discover a new FTA for the .doc9.  Save the profile and return to the Access Management Console.

Publishing

In the AMC, publish a new application, point it at the profile and it will see the FTA that you added.  EASY!

There's a catch

There is always a catch.  The catch here is that the FTA will exist only for new application that you publish.  Applications that you have already published have already had the FTA data sucked out of the streaming profile and placed into the XenApp publishing infrastructure.  The AMC will not "pick up" or automatically update the applications to reflect changes to the profile, so the FTAs won't exist.  If you need that FTA added to an application that is already published, it is necessary to delete the application and republish it. 

The other solution

Not recommended, but you can accomplish the same thing by hand editing the XML data in the streaming profile and saving.  This isn't the RECOMMENDED answer, but just between all of us, it will work.

Joe Nord 

Citrix TV folks have made me look really good!  Here's a link to a 9 minute video where I go through the "layers of glass" in application isolation and the fundamentals of how Application Streaming works.  The video outlines the XenApp 5.0 new capabilities for HTTP based streaming and provides guidance on wide area network architecture including Branch Repeater usage to eliminate the Application Hubs at each branch office.

I have previously blogged about the layers of glass, here, and with inter isolation communication added, here.

This video is better.

Joe Nord

Today I learned that Citrix posts its XenApp build debug symbols to a publically accessible symbols server.  Cool!  This is mostly to make life easy for the Citrix support folks, but what ever the reason, this means that the symbols are in a place that you can see them!

If you are familiar with WinDBG or Debugging Tools for Windows, then you know that the Microsoft Symbol server is a welcomed item in understanding the state of any machine you are inspecting.  Microsoft posts all the symbols for every component of every released version of Windows to msdl.microsoft.com/download/symbols. Set up the _NT_SYMBOL_PATH environment variable and the debugger does the rest to fetch Windows symbols from the internet.  Yeah, old news. 

I recently learned that Citrix also posts the XenApp symbols!  And to this I say, down right neighborly of us! 

Here's the link that says how to set it up, CTX118622.

What are symbols?

When building a program, the compiler does its thing and then the linker does its thing to actually create the executable content.  When putting the program together, one of the things that the linker does is produce a table of program symbols and their location in memory (symbols).  Programmers use these to figure out what's going on where they can then see nice "strings" rather than hex numbers. I say programmers, but it is really the debugger that does the work. 

There are two choices in symbols, the "FULL" and the "STRIPPED".  Simply put, the "full" ones are better, much like source code access.  The stripped ones though are still very helpful and it is these that are on the Microsoft download server ... and the Citrix download page. 

Neat thing of today, I've been writing code at Citrix for a long time and never known that the symbols were available to the outside.  NEATO.  This sure beats debugging stack traces with only hex locations of functions.

If you're not debugging systems on a regular basis, you may still find this useful for analyzing crash dumps.  One of the first things you do after a failure is windbg -z memory.dmp and then !analyze -v.  Give the debugger symbols and you can get a more precise location of a failure.  Good stuff.

Joe Nord

The "layers of cake" presents great opportunities for Citrix Delivery Center storage optimization.  Rather than store separate images for each machine, you can have ONE image for maintaining the operating system, ONE location which holds the application images, and ONE location for the User Profile.  This is valuable for XenApp, but it is especially important in XenDesktop where the number of machine images multiplies to be a really big number. 

This post describes the "problems" of getting applications into this layered world and discusses multiple solutions; including several that are not ideal.  It concludes with short term and long term recommendations.

Here's a slide that I presented at the Citrix Synergy conference in Las Vegas last week.
 
Even has candles!  Provisioning Services provides the bottom layer.  Application Streaming provides the middle layer and Citrix Profile Manager provides the per-user settings as the top layer of this multi-layer cake.  Getting this to move from "vision" to "reality" though involves system design and piecing this into your larger data center configuration is a complex operation.  The benefit though is increased flexibility and improved management of your systems as you are managing only images and assigning these as workloads to machines. 

Why does Application Streaming provide the middle layer
In this discussion, the "Streaming" aspects of Application STREAMING are really overrated.  The important fact is that the base operating system image focuses on being the operating system and the application layer focuses on providing applications.  Application Streaming is well suited to this need primarily because of the just in time application delivery - delivery based on which USERS will actually visit the machine.  To implement the separation in the layers of cake, the lowest layer cannot have knowledge of the application set and this means that the applications have to be RUNTIME provided to the execution machine.  The Citrix XenApp publishing does exactly this, both for Stream to Server (XenApp) and for Stream to Desktop (XenDesktop).  In both cases, the machine is "vanila".  A user logs in, application icons are placed onto the machine via the Plugin for Hosted Apps (aka PNAgent) and eventually the user clicks on something to cause the application to run.  With "streamed" applications, the applications are presented to the user "just in time" for their use.

Where it gets complicated
WOW, this is awesome.  I can have my cake and eat it too!  The details though are subject suitable for significant debate of best solution.  In particular the delivery of applications to the middle (application) layer. 

Provisioning Sevices will provide the base operating system image to the bottom layer and this layer will be absent most applications.  Sure, some foundation applications that are used by all users may be in this layer, but the advantage in the layers of cake is separating the applications from the operating system and this motivates us to want the apps delivered via Application Streaming regardless of benefits of isolation. 

Provisioning Services implements ONE to MANY by having a single disk image that it distributes to MANY "machines".  These machines may be real machines, or they may be virtual machines.  Either way, they get a MAC address and this defines them as a "machine".  Since this is a disk image and used by many, PVS must implement "write-back" cache so that machines using the shared image can update that image at runtime.  Important changes to the single image will be performed in "single" mode of PVS, but runtime changes to the disk must be maintained so that PVS can runtime support the temporary changes that the host operating system and applications will make the the "disk".   For each machine that PVS is running, it maintains a write-back cache.

The PVS write-back cache is "temporary".  When the machine shuts off, it is discarded.  This is what we "want", but it also "must be" because PVS is a block based implementation of a disk and it really has no idea what is stored in what block on the disk.  It doesn't even know what the "files" are.  With numerous machines "writing" conflicting changes to the disk at the same time, the only way out is to write cache these and then discard the cache when the machine shuts off, or in the XenDesktop case, when the user logs off.

Why is this a problem

If the Application Streaming application content (RadeCache) is runtime populated, then users will experience a first time application launch penalty each time the machine is booted.  In the XenDesktop case, this will be experienced on each logon.  The RadeCache must be populated and while it holds only a portion of what the application "installed", it is still enough data that the population of this space is a sore-point for first time use of applications.  In server case, or stream to a physical desktop, the first time launch is a one time cost.  The second time launch is much more important.  BUT, in the layers of cake the first time launch penalties become a every logon penalty.  The first experience on each logon is that the user behavio will be "slow" and then this "slow" performance will be repeated each time the user logs off and logs back in.  NOT GOOD.

It gets worse.  The PVS system write-back cache is about to get hit with LOTS of stuff.   As an example, consider MS Office 2007 on the notebook from whcih I'm writing this post.  The populated stuff to support MS office is about 400MB of disk space.  This will populate into the PVS write back cache and that's alot of disk space if you multiply it by 1000 machines.  More to consider, the Application streaming System does a network read to get the data from the Application Hub, it then writes it to the "local" disk, which in this case, isn't local.  The PVS write-back system then has to send that data across the network, back to the PVS server, who has to maintain the data on a per-machine basis; and potentially later, send that data BACK across the network to the execution machine to support execution.  I'm not sure of the numbers, but if you load it heavily enough, this will start to be a burden on almost any network.

How to solve

First a bit of background on the streaming system storage of application content.  The disk storage is the key here.  Everything that represents the application gets added to the streaming execution cache, which defaults to disk location, \program files\citrix\RadeCache.  There is a GUID beneath here to hold the execution content for each "profile" or execution "target" which represents the installed application that the streaming system is keeping track of.  Technically, a versioned GUID, GUID_version where version is used to allow "hot" update of applications.

You COULD populate the RadeCache into your base PVS disk, but this rather defeats the separation between Operating System and Application images.  This solution is "out".

The better solution says that the RadeCache should not be stored on the same disk volume that is the bottom layer.  Instead, the RadeCache should be moved to another disk volume and this disk volume should be common and shared across all the using systems (XenApp servers or XenDesktop clients).  Now, the maintenance of the operating system image can be ONE; and done via Provisioning Services and the maintenance of the applications can be ONE; and done via the Application Streaming profiler + the shared RadeCache execution space.

Getting more complicated
Much of what follows are multiple approaches to solving this problem.  The "correct" answer will depend on your network configuration and the tools that you have available to assist with representing a shared network resource as a local system disk volume.

PHYSICAL MACHINES

The easy solution.  In XenApp for example, even if using PVS as boot source, you likely still have a physical disk volume on the execution machine.  If yes, configure that space as write-back cache for PVS and the complications are improved.  The extra network hops will go away.   I'm not positive, but it is likely that this write back cache still resets on each boot, so there will still be first-time launch penalties.  With that, we should instead focus on machines that have no physical disk as the solution here will work for both configurations. 

We need a central source that can be mounted into all execution machines and treated as streaming runtime execution storage.  We want the "RadeCache" to be "fully populated" so that the streaming system NEVER needs to do a runtime cache fill.  In this way, the "streaming" aspects of Application Streaming are handed off to other technologies that will make the RadeCache appear as present even when they are remoted.  A side benefit of this is that App Streaming file based cache fills then get replaced with a page based system.

Application Hub

In a way, we already have a central space that holds the application content, it is the Application Hub.   Generally, all the profiles are stored beneath some "top" location and each profile contains the execution content to support that application on what ever the execution system may be.  One complication is that the "execution stuff" is stored inside a .CAB file.  While this is handy for compression and handy for portability to offline execution, it is not handy for access at runtime, pretending that the cab contents are actually present on the referencing machine.  It WOULD SURE BE HELPFUL if the CAB file were replaced with a subdirectory that has the same name, but is an uncabbed representation of the execution content.   In this way, the streaming system could redirect the local machine RadeCache\GUID_v to the Application Hub, profilename\GUID_v directory.  Do this and the problem is SOLVED.  Only problem, it doesn't work. 

Redirecting the execution content to network source

The "LINKD" command in Vista command prompts provides the foundation.    You need to use the /D option to create a directory junction.  Cool part here is that you can redirect the directory space to another volume and that volume CAN BE ON A NETWORK!  My experiments say that this works on Windows Vista, but that it does not work on Windows XP.  Given most XenDesktop configurations are Windows XP, this is a complication.  Note that the foundation reparse points that make this magic possible having been present in NTFS since Windows 2000.  The ability to convince the OS that this is present appears to only work on Vista and above.  I wrote some programs to exercise this back in the early Tarpon days; none bore fruit, but then again, Windows 2000 was a required operating system at the time, so more options may be available.

But wait, I can mount a volume into an empty directory on Windows XP.  Yes, yes, you can, but you can only mount the ROOT of that volume.  On Vista, you can mount subdirectory to subdirectory -> Much more useful!

The foundation commands are: As an admin:

CD \Program files\Citrix\RadeCache

mkdir \\apphub\share\profiles\profilename\GUID_v  (and then uncab the cab contents into that dir).

MKLINK /D GUID_v \\apphub\share\profiles\profilename\GUID_v

DIR GUID_v  ( Local directory, then you see everything on the network server )

In theory, DONE!  The streaming system will redirect stuff into the RadeCache, the I/O manager and NTFS will convert those into reparse points where they will ultimatley go back down the stack, this time as a network file system operation and all will be happy.

But, runtime, the Application Streaming system gets unhappy.  I have tested this only with the non-released streaming client that I'm using on all my machines, including the machine I'm using to write this post.  There is some chance that this will actually work on the currently released stuff.  The stuff "you don't have yet" does registry MOUNTS rather than loading of registry information.  This is far more efficient at runtime, but it is also dependent on the operating system supporting the mount and when asked to mount a registry hive from a non-local disk location, the system responds, very simply, "NO".  DUDE!  Nothing is local!  You think C: is better than that redirected space, trust me, it's all bad!  Please push on and give it a shot!  Nope.  Doesn't happen.

Using a second disk volume
Provisioning Server focuses on providing the operating system image; read this as "one volume" per machine.  What if you had a second "local" disk volume?  Yes, that would work!  You could tell the Application Streaming system to move the RadeCache to the second disk volume and all would be happy!  The streaming system already comes with a utility to move the cache, it is ClientCache.exe and is included and installed with the streaming client.  Here's a link that describes how to use it.  So far so good.  The streaming system though will insist that this "new" location be on a "local" disk. 

The second disk volume would contain only a fully populated view of a conceptual RadeCache of a machine that has ALL applications fully streamed.  Requires the admin to create this second disk image and requires that this image be maintained when applications are updated, but all of these things are containable and might even make sense as we all get used to maintaining the layers of cake.

How do you MOUNT a second disk volume?

Nicely, this disk volume does NOT have to exist at machine boot.  It only needs to exist BEFORE the user has a chance to launch any "streamed" applications.  If we assume that they do that in their startup folder, then this means that we need the second disk in place as a function of logon.  It could be in a system logon script.

The mounted disk volume must be LOCAL.  The streaming system insists on it.  It isn't that the streaming system is against storing execution content on remote systems, it just insists on it - which is interestingly very similar to the registry mount rocks I threw earlier in this post. 

The streaming system file system filter driver observes all file system volume mounts.  If it's "local" and if its "NTFS", it gets involved in the I/O stack and can use this space as a place to store the execution content.  

The App Streaming system spends alot of time lying to applications about what is present.  Our trick is to LIE to the LYING system to fool it into believing a disk volume is "local" when everything is really remote.

How to lie?

Some customers have elegant disk systems that allow them to mount extra "drives".   If so, then the ideal such system would allow mount of space into N machines where there is ONE backing store.  In utopia, this space is read/write.  Only the Streaming Caching service will write to it and any writes done for any user on any given machine would then benefit all users on ALL machines.  ELEGANT!

I belive such systems exist, but I have not yet seen it successfully implemented.  One complication is that the streaming system insists that the disk volume be formatted NTFS.  If the file sytem isn't NTFS, we get out of the way.  In this case, we don't want to get out of the way and given numerous machines accessing a single shared source, the odds of this having NTFS as the file system are ... remote.  It will though have most everything we need.  If you have one of these, shoot me a line and we'll work with you to get the streaming filter willing to put the streaming cache into that space. 

VHD, ISO

XenServer, VMWare, Hyper-V

The machine virtualization technologies also provide tools to mount disk volumes.  If we can mount a single disk volume, local disk, into multiple machines, ideal.  If this supports WRITES, better.  At logoff, would need to merge this common RadeCache back to a master store to become the base image for additional users.  All of this would work - but it is all very involved.

Ideal world

In an easier world, the Application Streaming system itself would permit redirecting the RadeCache execution space to a central network location, on the Application Hub.  That does not exist.  The fact that I'm talking about it though should tell you that I want to solve this problem.  I've been avoiding talking about it because I am not yet satisified with the answer.  This post though attempts to discuss your NEAR term solutions.

What should I do?

Near term, populate the RadeCache into the base PVS image.  I know this isn't ideal, but it is a workable solution and solves the performance aspects.  Longer term, investigate mounting .ISO and VHD and look into abilities to leverage storage arrays to provide "local" disks that are really remote.   Longer term, understand that this is a common need and solving it, is broadly important. 

Joe Nord

Product Architect- Application Streaming and User Profile Manager

Citrix System, Fort Lauderdale, FL

I recommend: Never PreDeploy to XenApp Servers; never PreDeploy to XenDesktop clients.  Always PreDeploy to notebooks.

Background: Application Streaming supports server side execution, client side execution and client side execution that can go offline.  These are controlled with publishing; should the application be available for offline or not, and via client side utilities (RadeDeploy) to optionally predeploy the execution content to the execution machine.  Notice that PreDeploy is REQUIRED and automatic for offline, but that Deploy is optional for online. 

What does "online" mean?

Online execution means that the streaming client has everything it needs to run the application available via the network.  This means that it can "see" the Web Interface and can "see" the Application Hub.  By contrast, "offline" means that execution can still happen when the Web Interface and Application Hub are not available.

In programmer speak, this is a 2*2 matrix with 4 possible states; one of which is invalid, offline without Deploy.  I argue that Deploy + online, while valid, should not be used if the execution machine is inside the data center.  If the network is "solid", don't Deploy.  The corollary, if the machine is "sketchy", always publish for offline, with automatic deploy.  This would get the number of states down from 3 to 2, which makes programmers happy.

What is Deploy

The guts of the streaming client are unaware of whether execution is server side or client side, online or offline.  The execution engine brings execution content into the RadeCache (\Program files\Citrix\RadeCache) and fetches that execution content from a CAB file on a file server or web server (App Hub).  In the online case, the cab file location is REALLY on a server.  In the offline case, the streaming engine is pointed to the Deploy location on the local machine and does "local streaming".  I call this streaming from one side of the hard disk to another; it was my idea back at the beginning and I must say that 4 years later, I'm pretty happy with myself.  The caching activities in the streaming service and the kernel mode stuff in the file system driver are completely oblivious to "offline/online" and the streaming happens with the same code path in all cases; happy.

Deploy is implemented by copying the execution content from the Application Hub and placing a copy of the Application Hub contents onto the execution machine; in \program files\citrix\deploy.  Looking back, I wish we had put the RadeCache and Deploy directories one level deeper, but this is where they exist. 

How to deploy

There are two ways to deploy.  In the automatic case, the admin publishes an application and, to some users (you), publishes that application as available for offline.  When you refresh applications in PNAgent (plugin for hosted apps), PNAgent gets a list of all of the applications that are available offline. 

The administrator has a choice when publishing.  In the Access Management Console, the admin can say that the Pre-Deploy should happen immediately, or they can state that the Deploy should happen on the FIRST use of the application.  I have not yet found a single admin who has gone with the Deploy on application refresh.  Network storms at 8:05am are not how they want to start their day.  This means that applications are not available for offline until the application has been run online, at least once.  

Eventually, the user (you) clicks on an icon to run the application.  The application is run as if "online" and in the background, the server side content is xcopied down to the Deploy location on the execution machine.  This automatic action does not occur if execution is on a XenApp server. 

The first use of the application continues in "online" configuration and it isn't until the next fresh launch of applications from that profile that the offline content is used.  Looking back, it would have been better to have completely copied the offline content to the execution machine before ever running the application.  The user feedback could then be ... "Copying your offline content, please go get a cup of coffee while I transfer 2GB to your machine".  Instead, the user is faced with online execution, competing  copying execution content from the file server into the RadeCache AND the streaming system copying the whole CAB file from the file server into the Deploy location and this presents a less than ideal first time experience, if the application is large.  Yes, the offline deploy is done on an "idle thread", but single user, 2GHz + dual-CPU machines spend lots of time idle, so it competes.  Eventually, you get the execution content local and further streaming from that profile is local to the execution machine AND available for offline. 

How to PreDeploy to XenApp server

First, don't!  Second, you could use RadeDeploy.exe to command the deploy to happen, and if you do, it will occur.  This would normally be done by software management system.  The streaming client running on that server will at runtime look for the deploy content and if it is there, it will use it. 

WHY NOT Deploy to servers!

It isn't about the first execution.  Yes, the first execution will be faster if you are PreDeployed.  The application update though will be SLOWER.  You first deploy.  This happens once.  Later, you will update the application numerous times.  The update in "online" scenario is more efficient than the update in "offline".  I should note that I really mean that the update is more efficient without PreDeploy.  Online/Offline is not the trigger, the trigger is Deploy/NotDeploy - then again, one shouldn't deploy without offline, so they really are the same state.

Consider a large application, say 1GB in size.  If you PreDeploy it, you will have everything available for cache fills based on copies across the local disk; efficient.  If instead, you don't PreDeploy it, some small percentage of the application will be runtime copied into the execution cache.  

When it comes time to update the applications MOST of the execution content will be unchanged.  The Streaming Client will "KNOW" what files are the same, and which are different and it will "KNOW" this without a deep look at the files inside the CAB.  In the online case, all of the "same" content can be immediatley promoted from version 1 to version 2 with no network involvement and this happens whether deployed or not.  In the "deployed" case,  the streaming system though needs to bring down EVERYTHING and this is very inefficient. 

In Streaming Client 1.1, this resulted in a full copy of the CAB file from the network server to the Deploy location on the client.  Notice that it is a single file (the cab); the client copied the whole thing and this is not friendly to the network and worse, it is very bad if the user is remotely connected over a VPN; where Deploy SHOULD be used.  

In Streaming Client 1.2 (XenApp 5.0), the streaming client was enhanced to bring down only the "deltas" from version "old" to version "new".  This is much more efficient network wise and brings Deploy case on-par with the online case, but it is still less efficient.  Ultimately, the whole cab file is needed for both "old" and "new".  The streaming system knows what has to exist, but the CAB file format does not allow update in place.  To save network usage, the streaming client trades client side CPU and Disk activity to save network.  This means that the Deploy update consists of a full un-cab of the CAB file, apply the updates, and full re-cab.  For a 1GB application, this can take minutes!  10s of minutes?   What is my machine doing!!  AArgh!

In a XenApp hosted environment, there is no need to pay the un-cab/re-cab penalty; the most efficient update is directly inside the execution cache and this has existed since the first release of Application Streaming.  All of the above rocks thrown, are we going to make the Deploy/Offline case more efficient?  Certainly!

What should I do to optimize first time launch on the XenApp server?

Turn on "-e" RadeRun switch.  Here is a KB article with more details.  Execute one application from each profile, ONCE.  Finally, be sure to then turn off -e.  Yes, I say turn it off because if you don't you will un-pack everything from the CAB on EVERY launch of any application for every user - which will be dog slow.   

You can achieve the same thing by running RadeRun.exe directly from the command line, specifying the -e switch.  This best done by automated systems, when the users aren't around.

With the single run with "-e", everything will be extracted from the Application Hub and placed into the streaming execution cache.  The streaming system will still do it's "just in time" programming, but it will pretty much never conclude that a cache fill is needed and the result is that execution will not suffer a first time launch penalty.

Enjoy, 

Joe Nord

Citrix Systems, Product Architect for Application Streaming and User Profile Manager.