10 Jun 2008 05:27 PM EDT

Filling the execution cache is an obvious job of the streaming service and is an obvious item to discuss.  Since the obvious item is not as interesting as the unobvious, deleting from the cache is covered in this post in detail.

Warning: Geek speak is turned on for this post.   I hope people find it useful. 

The Application Streaming client has the mission of maintaining its execution caches which support the execution of applications under Application Streaming.  Yes, that's rhetorical.  The streaming client has the same cache management logic regardless of its execution platform and manages the cache in exactly the same manner for both server side execution and client side.

Most administrators understand pretty well that the middle layer of the isolation spaces represents the "installed application" and this layer is "not really present" at runtime.   I've posted a few other blogs that show a graphic of this, so I won't repeat it here but you should be able to find it up or down a few screens.
The installed contents (installation image) are presented to the application as if they are "present" and are only truly brought into the execution machine when the application actually "touches something".  That is, the arduous job of actually copying installation image content from the central server to the execution machine is put off until the last possible moment, or as a more elegant term, "just in time populated" into the cache.  This is classic computer science stuff: If you put work off long enough, you may be able to avoid it completely.  In college, I did my homework with this same methodology.

At runtime, the execution cache starts empty and grows and then retains its contents for future executions.  The cache is maintained even across boots so that the later sessions will have execution content "already there" and avoid the need for a cache fill.  Cache fills equal network traffic, equals slow, so we prefer to have 2nd time launch experience rather than a first time.  Notice that cache fills are also shared across users.  Consider a Presentation Server hosted execution where content is added to the cache for UserA.  The execution cache is shared across the users, so the cache addition for UserA, will benefit UserB.  The "first time launch penalty" of using streaming and isolation really applies only to the first user on the entire server, and there are ways to reduce that impact.

Deleting content from the cache

When running an unlimited number of different applications, for an unlimited number of potential users, from an unlimited number of separate streaming profiles, the streaming service must support them using finite space in which to run the applications.

How much space does it get? 
What is the maximum size of the RadeCache?

The answer is ...  that it depends.  The RadeCache limits are implemented as a size constrained high water, low water system where the low and high limits are determined at streaming client installation and possibly overridden by an administrator.  The max size limit (high water mark) is installation time written to the registry and defaults to 5% of allocated disk space of the primary boot volume ( e.g. C: ), or a minimum of 1GB.  The low water limit defaults to 95% of cache size.

Right - what does that mean?

The Streaming Service (RadeSvc.exe) keeps track of the space being used by the execution cache and where that value grows ABOVE the high water mark, the service unleashes a "purge thread", or "delete thread" to free some space.  Notice that it does not fail or even delay the cache fill for the application that triggered the growth beyond the high water mark.   The application that is trying to run is immediately put back to doing useful work and this means that the cache is temporarily allowed to grow beyond the high water mark.  The delete activity is done by the streaming service too, but it is done on a separate thread from the cache fill logic.   The delete actions are also done at idle priority.  The delete thread is persistent though and will continue chugging along until the deleting is finished.

The streaming service is aware of the cache contents of ALL of the execution caches and the size of each file in each target, and for each file, the age of when that file was most recently accessed.    These execution caches are all of the subdirectories of \Program Files\Citrix\RadeCache.  Notice that there can be many caches and even cache content for applications that are no longer published to any user that is presently logged onto that server or possibly even for applications that are no longer published to anyone anywhere.  The point is that "new stuff" is more important than "old stuff" and eventually, the old stuff will age its way out of the cache. 

If someone runs an application that uses old stuff, the old stuff is new again and is no longer the best candidate for erase.  Low tech, but easy to implement which should read as "it doesn't go boom".

Low water mark

The cache manager delete thread repeatedly deletes old things from the cache until a "low water" mark is achieved.   Once low water mark is achieved, the delete thread goes back to sleep.   New cache fills can be occurring concurrently with cache deletes and all the while, the streaming service has to keep track of what is in the cache.  This is one of the reasons that: If you want to purge things from the cache, you should use tools like RadeCache.exe to do the delete rather than just deleting the files. This lets the streaming service retain its self stated importance as the keeper of the cache and lets it know what is presently occupying space in the cache.  Those are good things, but more importantly, it protects against windows in time where an application can start opening a file and then the file getting erased by the deleter before the file open has occurred.   This would be a long aside, so I'll get back on subject of deleting stuff from the cache.

Default low water mark is 5% below full.  Strangely, it's not implemented as 95% of full, which is bazaar, but that's how it is. This value is configurable via registry, it defaults to "5" and you generally shouldn't mess with it.  If you do, understand that it is how much space should be FREE before the delete thread stops deleting stuff rather than the percent full after the deletion stops.

Low disk avoidance

In addition to filling the cache and deleting stuff from the cache, the cache manager also tries to notice when the disk is crazy full and takes steps to be a good citizen.  The default here is 200MB. If you're down to this limit of disk space, the cache manager will dispatch the purge logic to start freeing up space on the disk and otherwise to try to keep the machine from buckling.  Once the minimum disk space is achieved, the delete thread goes back to sleep.

Thrash avoidance

Notice that deleting stuff for purposes of managing the cache is a good thing.  Deleting lots of things to help the system not run out of disk space is also a good thing.  It's possible though that all this good citizen stuff will result in endless cache deletes followed by fills followed by cache deletes, followed by fills, ...

The cache manager includes thrash detection logic to prevent this scenario which says:  If EVERYTHING in the cache was touched in the last 24 hours, then we're thrashing, don't delete anything. This may not be perfect - actually, it isn't perfect - but it is an attempt to do the right thing and effectively support streamed applications while automating the cache cleanup. 

Registry keys
Here are the registry keys that control the streaming service cache management.  They are all items in HKLM\Software\Citrix\Rade 

Registry item     Type       Default and comments

CacheLimitMb      REG_DWORD  5% of disk space or 1GB.
                             Value is in MB (mega bytes) even though
                             key name indicates Mb (mega bits).

CacheLowWaterPct  REG_DWORD  5.
                             Value is percent of cache to make full
                             when start deleting

LowDiskThreshold  REG_DWORD  200.  Value is in MB.

Enjoy. 

Joe Nord
Product Architect, Application Streaming. 
Citrix Systems, Fort Lauderdale, FL

Permalink | Comments (4) |

Comments  (Hide Comments)

Joe, real fan of yours my friend. Awesome post, as always.

cheers,
Gus

Posted by Gus Pinto at Jun 10, 2008 20:17 | Reply To This

Joe,

is there any way for pre populating the cache (\RadeCache) from the pre deployed .cab file (\Deploy)? It's really a problem if you have a big cab file (~ 700mb) which is extracted at the first launch on a server.

Thanks,

Reinhard

Posted by Anonymous at Jun 22, 2008 08:27 | Reply To This

You'd like to think that running RadeDeploy.exe will pre-populate the execution cache, or at least have an option for pre-populating the execution cache.  It does not.  Instead, all it does is copy the content from the network server to the Deploy folder.  At runtime, the "cache fills" still occur to place content into the execution space below RadeCache.

The solution is to manually pull the FILES out of the CAB file and place them into the RadeCache subdirectory that identifies the Target.  That is the GUID_version of the CAB file is the same GUID_version of the directory name in the RadeCache.  I have previously written about why you should not delete the RadeCache directory itself (DACLS).  The directories below RadeCache CAN be created and deleted without hurting the operation of the streaming client, though you should not do this while the streaming system is running applications or profiling and when you get done, a reboot would be a good measure or at least a start/stop of the streaming service (radesvc).

Inside the CAB file are a few "root" level files.  These are the ones that the streaming profiler uses to manage what is in the Target.  Ignore those.  The important stuff is in the Device subdirectory.

The Device folder in the CAB file holds the start of disk content.  A subdirectory for each drive letter and then the disk structure for the installation activity of each of those drives.  Everything related to the installation disk activity is held below Device and the RadeCache also has a Device space.

To Pre-Populate the RadeCache, create a directory below \Program Files\Citrix\RadeCache\ with name of directory to match the filename for the CAB file.  Then, extract all the files from the Device space of the CAB and place them into the created directories Device space (recursive).  That's it, you're done - with disk.

To know though, if you are using digitally signed profiles, you just bypassed the signature check on the cache fill.   

Pre-populating the registry is more involved and I'll suggest isn't worth the trouble.  The first launch will populate it for all the users and the majority of TIME is spent in actually copying the disk content (many 100s of MB).  The first time populate is also optimized to use volatile registry space and other actions to make it relatively quick.   The backing store for the registry content is in the Deploy location, so its local and will be populated into the registry without network involvement.

So, start with disk.  If that's enough, good.  If it's not enough, ...

The registry contents are stored inside a .TAB file at the ROOT of the CAB file.  Why .TAB?  Let's not worry about that. 

Compare .TAB to .REG.  They are similar except that REG files have fully qualified paths in them and these are no good for isolation (the runtime location and the profiling time location are different).  The paths are stripped to make all entries relative path and then the whole lot are compressed? or packed together in a form that allows for backward/forward compatibility.  Did I mention that it isn't ascii 09 chars?    To get the contents of this registry, open the profile in the Streaming Profile, RMB/Update Target, the registry contents will be extracted to the profiling temp space where you could extract it and copy to other location, after adjusting all the registry positions to be the correct target space.  Like I said at the start, let the streaming client do this for you.  

We have it on the roadmap to make RadeDeploy optionally pre-populate the disk and registry isolation space.  On the map here means beyond Delaware release.

-Joe 

Posted by Joseph Nord at Jun 22, 2008 10:02 | Reply To This

Joe,

thanks for fast response. It's really detailed! Do you have any scripts which accomplish this? If you have many servers with many streamed apps... it's a pain

Posted by Anonymous at Jun 22, 2008 11:32 | Reply To This