Post to Twitter |
Comments (19) |
Views (9394) |
I recommend: Never PreDeploy to XenApp Servers; never PreDeploy to XenDesktop clients. Always PreDeploy to notebooks.
Background: Application Streaming supports server side execution, client side execution and client side execution that can go offline. These are controlled with publishing; should the application be available for offline or not, and via client side utilities (RadeDeploy) to optionally predeploy the execution content to the execution machine. Notice that PreDeploy is REQUIRED and automatic for offline, but that Deploy is optional for online.
What does "online" mean?
Online execution means that the streaming client has everything it needs to run the application available via the network. This means that it can "see" the Web Interface and can "see" the Application Hub. By contrast, "offline" means that execution can still happen when the Web Interface and Application Hub are not available.
In programmer speak, this is a 2*2 matrix with 4 possible states; one of which is invalid, offline without Deploy. I argue that Deploy + online, while valid, should not be used if the execution machine is inside the data center. If the network is "solid", don't Deploy. The corollary, if the machine is "sketchy", always publish for offline, with automatic deploy. This would get the number of states down from 3 to 2, which makes programmers happy.
What is Deploy
The guts of the streaming client are unaware of whether execution is server side or client side, online or offline. The execution engine brings execution content into the RadeCache (\Program files\Citrix\RadeCache) and fetches that execution content from a CAB file on a file server or web server (App Hub). In the online case, the cab file location is REALLY on a server. In the offline case, the streaming engine is pointed to the Deploy location on the local machine and does "local streaming". I call this streaming from one side of the hard disk to another; it was my idea back at the beginning and I must say that 4 years later, I'm pretty happy with myself. The caching activities in the streaming service and the kernel mode stuff in the file system driver are completely oblivious to "offline/online" and the streaming happens with the same code path in all cases; happy.
Deploy is implemented by copying the execution content from the Application Hub and placing a copy of the Application Hub contents onto the execution machine; in \program files\citrix\deploy. Looking back, I wish we had put the RadeCache and Deploy directories one level deeper, but this is where they exist.
How to deploy
There are two ways to deploy. In the automatic case, the admin publishes an application and, to some users (you), publishes that application as available for offline. When you refresh applications in PNAgent (plugin for hosted apps), PNAgent gets a list of all of the applications that are available offline.
The administrator has a choice when publishing. In the Access Management Console, the admin can say that the Pre-Deploy should happen immediately, or they can state that the Deploy should happen on the FIRST use of the application. I have not yet found a single admin who has gone with the Deploy on application refresh. Network storms at 8:05am are not how they want to start their day. This means that applications are not available for offline until the application has been run online, at least once.
Eventually, the user (you) clicks on an icon to run the application. The application is run as if "online" and in the background, the server side content is xcopied down to the Deploy location on the execution machine. This automatic action does not occur if execution is on a XenApp server.
The first use of the application continues in "online" configuration and it isn't until the next fresh launch of applications from that profile that the offline content is used. Looking back, it would have been better to have completely copied the offline content to the execution machine before ever running the application. The user feedback could then be ... "Copying your offline content, please go get a cup of coffee while I transfer 2GB to your machine". Instead, the user is faced with online execution, competing copying execution content from the file server into the RadeCache AND the streaming system copying the whole CAB file from the file server into the Deploy location and this presents a less than ideal first time experience, if the application is large. Yes, the offline deploy is done on an "idle thread", but single user, 2GHz + dual-CPU machines spend lots of time idle, so it competes. Eventually, you get the execution content local and further streaming from that profile is local to the execution machine AND available for offline.
How to PreDeploy to XenApp server
First, don't! Second, you could use RadeDeploy.exe to command the deploy to happen, and if you do, it will occur. This would normally be done by software management system. The streaming client running on that server will at runtime look for the deploy content and if it is there, it will use it.
WHY NOT Deploy to servers!
It isn't about the first execution. Yes, the first execution will be faster if you are PreDeployed. The application update though will be SLOWER. You first deploy. This happens once. Later, you will update the application numerous times. The update in "online" scenario is more efficient than the update in "offline". I should note that I really mean that the update is more efficient without PreDeploy. Online/Offline is not the trigger, the trigger is Deploy/NotDeploy - then again, one shouldn't deploy without offline, so they really are the same state.
Consider a large application, say 1GB in size. If you PreDeploy it, you will have everything available for cache fills based on copies across the local disk; efficient. If instead, you don't PreDeploy it, some small percentage of the application will be runtime copied into the execution cache.
When it comes time to update the applications MOST of the execution content will be unchanged. The Streaming Client will "KNOW" what files are the same, and which are different and it will "KNOW" this without a deep look at the files inside the CAB. In the online case, all of the "same" content can be immediatley promoted from version 1 to version 2 with no network involvement and this happens whether deployed or not. In the "deployed" case, the streaming system though needs to bring down EVERYTHING and this is very inefficient.
In Streaming Client 1.1, this resulted in a full copy of the CAB file from the network server to the Deploy location on the client. Notice that it is a single file (the cab); the client copied the whole thing and this is not friendly to the network and worse, it is very bad if the user is remotely connected over a VPN; where Deploy SHOULD be used.
In Streaming Client 1.2 (XenApp 5.0), the streaming client was enhanced to bring down only the "deltas" from version "old" to version "new". This is much more efficient network wise and brings Deploy case on-par with the online case, but it is still less efficient. Ultimately, the whole cab file is needed for both "old" and "new". The streaming system knows what has to exist, but the CAB file format does not allow update in place. To save network usage, the streaming client trades client side CPU and Disk activity to save network. This means that the Deploy update consists of a full un-cab of the CAB file, apply the updates, and full re-cab. For a 1GB application, this can take minutes! 10s of minutes? What is my machine doing!! AArgh!
In a XenApp hosted environment, there is no need to pay the un-cab/re-cab penalty; the most efficient update is directly inside the execution cache and this has existed since the first release of Application Streaming. All of the above rocks thrown, are we going to make the Deploy/Offline case more efficient? Certainly!
What should I do to optimize first time launch on the XenApp server?
Turn on "-e" RadeRun switch. Here is a KB article with more details. Execute one application from each profile, ONCE. Finally, be sure to then turn off -e. Yes, I say turn it off because if you don't you will un-pack everything from the CAB on EVERY launch of any application for every user - which will be dog slow.
You can achieve the same thing by running RadeRun.exe directly from the command line, specifying the -e switch. This best done by automated systems, when the users aren't around.
With the single run with "-e", everything will be extracted from the Application Hub and placed into the streaming execution cache. The streaming system will still do it's "just in time" programming, but it will pretty much never conclude that a cache fill is needed and the result is that execution will not suffer a first time launch penalty.
Enjoy,
Joe Nord
Citrix Systems, Product Architect for Application Streaming and User Profile Manager.
Comments (19)
May 14, 2009
Anonymous says:
Joe, What are you recommendations for Application Streaming deployment or pre-d...Joe,
What are you recommendations for Application Streaming deployment or pre-deployment in a XenApp environment standard image provisioned using Provisioning Server?
- Chris
May 14, 2009
Joseph Nord says:
Recommendations here are complicated and forth-coming. Too big to write he...Recommendations here are complicated and forth-coming. Too big to write here. Look for a KB or white paper in a couple months. Short version, need to get the RadeCache out of the PVS provided image so that writes to the RadeCache do not end up on PVS per-machine write back cache.
May 15, 2009
Jane Cassell says:
We are in the process of developing a system similar to that Chris mentions - we...We are in the process of developing a system similar to that Chris mentions - we cant wait a couple of months to make the decision to pre-cache or not
By then we will be in production!!!
Can you give us some ideas - we are provisioning images to hardware and using the local hard disk as cache but would rather not have to re-stream all the apps to each provisioned image each time that they are rebooted - approx every 2 days - we are talking approx 150-200 servers!
Thanks
Jane
May 15, 2009
Joseph Nord says:
Hum. I'll see what I can get out in blog form QUICK, with something more o...Hum. I'll see what I can get out in blog form QUICK, with something more official to follow as a KB.
May 14, 2009
Simon Bramfitt says:
Joe Can you clarify the recommendation on raderun -e followed by app execution....Joe
Can you clarify the recommendation on raderun -e followed by app execution. It sounds like you are suggesting that just launching the app and closing it are enough to unpack the CAB, I was under the impression that to unpack you had to launch the app and then execute a workflow that touched all parts of the app to fully unpack it.
Please tell me I am wrong on this
Regards
Simon
May 14, 2009
Joseph Nord says:
> It sounds like you are suggesting that just launching the app and closing i...> It sounds like you are suggesting that just launching the app and closing it are enough to unpack the CAB
Yes. With -e, the Streaming System will do exactly that. Unpack everything into the execution cache BEFORE launching the application. In theory, this means that no runtime streaming will occur. In reality, eventually, the streaming service may delete things from the cache (house cleaning) and then the user could trigger something to cause a re-fill. This though would be not too common.
Anonymous replies:
May 14, 2009
Helge Klein says:
Hi Joe, very interesting insights! Thanks for sharing! HelgeHi Joe,
very interesting insights! Thanks for sharing!
Helge
Jun 12
Jacobo Carballada says:
Is possible to use raderun -e in a non interactive script? I am investigatin...Is possible to use raderun -e in a non interactive script?
I am investigating what it should be the best way to pre-populate the applications to xenapp servers. I am trying to do it during the startup script but it doesnt seem to work. I still need to further investigate this but I am wondering if anyone came across already with this issue and if there is any working solution for that.
Cheers
Jun 12
Joseph Nord says:
Can the utility be run? Definately. Will the utility be able to "se...Can the utility be run? Definately.
Will the utility be able to "see" the Application Hub? That is a bigger question.
What user context does the non-interactive script run in? That user will require read access to the directory on the server holding the execution content for the profile, either UNC or HTTP/S. The raderun -e "pull" from the server will use the credentials of the "user" and if this is not a real user, they may not have visibility to the server. A solution: Define a domain account with no interactive logon access, and give it access to the share, and run the non-interactive script using that account. raderun -e will run in that context and the streaming service will then have vision to the server for purposes of fully populating the cache.
The write to the execution store on the local machine is done by the local user, Ctx_StreamingSvc. The service borrows the user token to get to the content on the server since the streaming service itself lacks privilege to see anything on the network.
A trick in this case though is that the application will actually RUN - and you don't really want this. I mean, you have to get the application to terminate. You could command this with time delay and a "kill", but this isn't very good. One option is to add an "app" to the profile, which is a .bat file with one line "exit". Run that app via the raderun -e and it will immediatley terminate, after it fully populates the cache, both disk and registry. This could then run on a schedule and would result in no real-user paying first time launch penalty.
In "next" streaming client, the registry storage is converted to a hive and a MOUNT rather than a .REG like read/populate -- much faster. This means that the relative benefit of this complex approach vs. the simple approach of unzipping the file contents, will diminish. This method though is still more "technically correct".
Jul 08
Johan Greefkes says:
Adding to this, you could create a simple profile with this on-line batch file i...Adding to this, you could create a simple profile with this on-line batch file in it. Then link all the profiles you want to pre-execute to this profile.
In your startup scheduled task, run:
raderun.exe -e /app:CacheHook /package:\\server\share\profile
Jul 10
Anonymous says:
If you use the -e option to raderun it only pre-extracts all the files for t...If you use the -e option to raderun it only pre-extracts all the files for the application you are targeting but it doesnt pre-extract all the files for all the linked profiles.
Aug 04
Rodney Beaudry says:
I have two questions.. 1)If I am working on a large profile that has multiple a...I have two questions..
1)If I am working on a large profile that has multiple applications inside (Office 2007)... will the "-e" option extract all the contents of the CAB if i launch say Word or will just the files needed for word extract?
2) Can the "-e" be specified through the ACM when publishing a specific streamed application instead of using the RadeRunSwitches?
Thanks Joeseph! Lots of great information here!
Aug 04
Joseph Nord says:
RadeRun -e will extract the full contents of the CAB before the application ...RadeRun -e will extract the full contents of the CAB before the application is launched. This means that launching "Word" will also extract Excel.exe and will also extract OneNote.exe.
The thing to be aware of of is that -e should NOT be left in place. -e tells the streaming system to extract EVERYTHING and it will do this on every application launch regardless of whether some or all of the files are already in the cache. I have had a support escalation come my way where a customer found app launch to be slow and the cause was that they left -e in place.
> Can the "-e" be specified through the ACM when publishing a specific streamed application instead of using the RadeRunSwitches?
It cannot. That said, consider that -e should not be run as a usual event.
The -C, -D options make more sense here as a per-application property. Admins want to obliterate user content on each launch so it always starts fresh. This is a common request. Implementing though hits many groups in the company though, so it is hard to get on the map. Short term solution is to nuke the cache via a pre-launch bat file - which has its own issues, but will get easier in an upcoming release.
Aug 04
Rodney Beaudry says:
Thanks for the information Joseph! One last thing .. if I want to pre-fill ...Thanks for the information Joseph!
One last thing .. if I want to pre-fill the Cache using the -e option in the RadeRunSwitches does it require a server reboot after the registry is modified?
Aug 04
Joseph Nord says:
The registry item is read on each application launch. No reboot required.The registry item is read on each application launch. No reboot required.
Aug 06
Alex Nikonchuk says:
Joseph, Thanks for a very helpful post. I do however still have a burning...Joseph,
Thanks for a very helpful post. I do however still have a burning question: Is there a way to automate/script populating of RadeCache on a large farm? We're about to have a 200+ server farm, with a fairly large set of apps. The first-run penalty is not something "our clients" can tolerate, so we're looking for a way to script the pre-cache of the apps on each server. I've tried a number of "raderun -e" tricks, but none of them seem to work in a non-interactive "scheduled task" way (if you have a trick for this, it'll be awesome if you share
I was however wondering if there's a way to just "extract" the CAB's without using the raderun ? cab extraction is easily scriptable, and a few MFCOM calls easily produce a list of profile locations.
Any thoughts will be greately appreciated.
Thanks Joseph
Aug 06
Joseph Nord says:
Hi Alex. The easiest way to help the first time launch penalty is to have ...Hi Alex. The easiest way to help the first time launch penalty is to have the RadeCache file system space fully populated at time of first application launch. Using RadeRun -e will get you FILE and REGISTRY pre-populated, but the largest time savings will be for files. How to do...
Open the CAB up with Windows Explorer and sort by PATH, you will see a handful of ROOT level files which are the App Streaming metadata that accompanies the execution target. Ignore those. You will also see a DEVICE subdirectory and below there another directory for each installation disk drive letter written at installation time - generally speaking "C".
To help first time launch, you CAN extract and copy the contents of the Device\ onto equivalent location on the execution machines, that is extract and copy to the RadeCache space on the 200 servers.
Example technique using Microsoft CabArc utility:
mkdir GUID_v; chdir GUID_v; cabarc -x \\server\share\profiles\profilename\GUID_v.cab.
Then follow this up with an erase of all the root level files, the meta data - let the streaming system decide which ones it needs and let it populate them at first time launch.
You will note that this does not get the installation image REGISTRY, and this is true. Loading the registry though is a single file extract from the cab at runtime (InstallRoot.tab), followed by a .REG style registry load - which does take time, but is a small time compared to the file system load. More: In future streaming clients (think "soon"), this registry load will become a registry mount and any advantage in pre-loading the registry content will go away, so I recommend focus on files.
When done with the extract, check the owner and DACLs of the created files. They should be the same as if the streaming service itself populated the cache.
Owner of files should be: Ctx_StreamingSvc (account used to run the streaming service) and the streaming service should have full rights to the files. I believe all the DACLs inherit from the Parent directory, the RadeCache, but make sure the streaming has all permissions on the files, so that it can delete the files in the cache as they age.
One final note on CAB files. While the CAB files produced by the streaming profiler are "standard" CAB files and cab be extracted using CABARC, you SHOULD NOT use cabarc to create or modify the CAB files that will be used by Citrix Applicaiton Streaming. WHY? We do not use the default cab creation techniques. The default CAB creation (example, by cabarc) produce CAB files that do not lend themselves well to having files removed one at a time; instead cabarc optimizes for compression and assumes all files will be extracted in one go. To make it efficient for single file extracts, we use a 1 file per compression block in the cab creation and this is not "normal". CabArc and similar utilties can read the stuff we create just fine, but the creation is different, ours is optimized for single file extracts instead of compression. Probably worth a separate blog, but bottom line, let the profiler do the cab creation.
Key: The streaming service accepts "magic" as a cache population technique. You perform the magic, but be sure to make the DACLs end up right so that the streaming system can later manage the cache.
Aug 06
Rodney Beaudry says:
Hi Joseph, I am have been working hard to try and stream office 2007 to server ...Hi Joseph,
I am have been working hard to try and stream office 2007 to server and during my testing I always receive the following error with any new user that launches one of the published apps within the profile.
"An Error occured while caching a file required for your application. (0xE000385)."
I dug up this Citrix document http://support.citrix.com/article/CTX119053 and the workarounds of restarting the streaming service does work but for obvious reasons is unacceptable come roll out time to production. The solution states that I should pre-deploy the application.
After reading this great post i am not so sure that is a good idea and i am thinking that pre-filling the radecache is the way to go.
Thoughts?
Aug 07
Joseph Nord says:
I expect this means that the Citrix Streaming Service (username=Ctx_StreamingSvc...I expect this means that the Citrix Streaming Service (username=Ctx_StreamingSvc) does not have WRITE access to the RadeCache directory, or the specific subdirectory supporting that profile.
You can confirm this with Filemon from sysinternals and also by checking the dacls. Recall that the streaming system does not run as local system; instead running on a named user account, which has less power. The named account has write access only to specific dacl permitted locations, RadeCache and Deploy on the file system, and RadeCache HKLM space in the registry.
It also says that the service is trying to create things at runtime even with full extract, this will happen for the meta-data files at the root such as the isolation rules, sandboxdata.xml.
Filemon/procmon will answer all. Somewhere in the trace, you'll see a error for the streaming service on an attempt to create a file in the RadeCache\GUID_v space, probably for sandboxdata.xml.
To Fix: Compare the DACLs of a default installation to the GUID_v directories you created, they should be the same. Focus on container inheritance. Conveniently, I reinstalled 1.2 client yesterday, here's a DACL rundown of my test machine (Windows Server 2003, 32-bit). This will be the same for 1.3.1.
Details on DACLs moved to a new blog entry.
http://community.citrix.com/blogs/citrite/josephno/2009/08/07/App+Streaming+-+Dynamically+applied+DACLs
My bet is that you are missing the Ctx_StreamingSvc entry at the RadeCache level. Recreate using ClientCache.exe utility; installed with the streaming client.
Add Comment