Jump to content
Welcome to our new Citrix community!
  • 0

NGEN filling up App Layers


Olly Thompson1709156630

Question

Wondering if anyone else if having a similar issue? when coming to finish an App Layer, a script is run from JGSpiers: https://www.jgspiers.com/citrix-app-layering-preparation-script/

 

This generally works well, but we're finding that (intermittently) the NGEN process is taking up to 60 minutes and consuming around 5GB of disk space on each layer. 

 

We have run NGEN update /force in a new version of the OS layer (That finished in 11 minutes and took up just over 100MB space), but this had made no difference. Even creating a new app layer and immediately running the command causes the issue above. ELM is currently running at v.1812, but the issue is outstanding. I must also add that the problem is not consistent, although it does affect around 90% of new App Layers.

 

It's almost as though the NGEN update process is running the force command on its own, but even then i wouldn't expect so much storage to be taken up on a layer with nothing installed.

 

Any assistance is appreciated. Logged with Citrix, but as yet, they've not been much help.

Link to comment

Recommended Posts

  • 1

We've been able to reproduce this here.  We don't have an ETA for when we might be able to fix it, because it doesn't make a whole lot of sense to us either.  However, we've been able to figure out one useful thing: it only happens if you use the Connector Cache.  If you edit your connector and set the Cache Size to 0, or if you create another connector with the Cache set to 0, you won't see this.  When we use the cache, we change how we build boot images, and that apparently matters.  So if this is a problem, and you don't want to just wait out the background process, you can try with the cache disabled.

  • Like 2
Link to comment
  • 0

App Layering's only requirement is that .Net recompilations be finished before you finalize.  We have no requirement that you run NGEN manually, we're perfectly happy if your product installers, and Windows background processes, do that for you.  If manually running NGEN is giving you problems, don't do it.  That may sound cavalier, but since it's not a required step, and not actually solving a problem for you, let it go.

 

I don't think we have any other reports of this, so I don't know what might be making your system special.  Make sure you're up to the latest App Layering, of course.  Has this been a day-1 occurrence for you?

Link to comment
  • 0

@Gunther Anderson if that is he case, you may wish to update the best practice guide: https://support.citrix.com/article/CTX225952 as it states clearly under Application install to run NGEN update

 

This hasn't happened since day 1 (that we've noticed), but rather has come in at some point in the last few weeks. Strangely enough i booted up a PVS vDisk today and ran NGEN update which went through absolutely fine. Also, as the OS layer runs through with no issues, it's almost as if the newly created App Layers are not referencing the OS layer correctly

Link to comment
  • 0

@George - A new OS layer was created and a full ngen run on this layer (took about 12 minutes). Then created an App Layer off this OS layer and ngen update filled the disk. 

 

Have a case raised with Citrix which has been escalated a couple of times. Provided full verbose logs from ngen as well as UniDesk logs and their response is that they don't have a conclusive answer. I ran the ngen update on 2 seperate App Layer VMs at the same time which were identical - one worked and one didn't. Citrix advised they could see invalidated data in the log0.txt log on the failed VM, but don't know why it's happening, but it is causing many writes to the App Layering disk. 

 

Essentially, i've been told that i shouldn't run ngen in the App Layers (unless absolutely necessary), although this goes against their own advice (which is now going to be changed). If i do need to run ngen update in an App Layer and i notice it filling the disk, increase the App Layer disk size from 10Gb and re-try.

 

It's a total cop-out from my point of view. This environment is running full SSD storage and a perfectly good connection to the storage, so the issue is with the way in which the Layers look at the underlying OS disk and yet Citrix are wiping their hands of it.

Link to comment
  • 0

Totally agree - I've used your own script in many deployments now without issue.

 

This layer filling issue is unfortunately putting a very poor view of Citrix onto the customer. At the moment, it looks as though we'll be ripping App Layering out next week. It's a pity, because with the full SSD and really good network the customer has, this has been the best running deployment i've done yet

Link to comment
  • 0

Hello,

I have the same problem.

I have observed also that the problem is caused by using cache for connector:

- When I create new App Layer (without cache) and run ngen the process is fast and not consume much disk space.

- When I create new App Layer with enabled cache, ngen  takes a very long time and takes up a lot of disk space.


In both cases, of course, I used the same OS Layer.

Link to comment
  • 0

I am also having this issue.  I just started a new App Layering environment in a working XenDesktop 7.15 LTSR CU2 environment.

 

If I try to seal a layer, it gives me the finalizing error:

 

a Microsoft NGen operation is in progress in the background {0}

 

I started using the following script:

 

https://www.jgspiers.com/citrix-app-layering-preparation-script/

 

ngen just runs forever and fills up the 10gb App Layer.  I was able to seal the OS Layer and Platform Layer just fine, but for some reason it just doesn't work for the App Layers.  Is this a known bug with recent versions?  I'm on ELM 19.1.0.17 now, should I upgrade to resolve the error?  Does anybody have any solution to this?

 

Any assistance would be extremely appreciated.

Link to comment
  • 0

Unfortunately, Citrix were not particularly keen on investigating or fixing this issue at the time - even with the multitude of logs i sent over. They could see that the data was not being validated, but would not work to resolve.

 

The customer ended up pulling ELM which is a pity because they had full SSD storage and lightning network so it was by far the best performing one i've implemented (ngen issue aside).

Link to comment
  • 0
On 4/8/2019 at 0:50 PM, Gunther Anderson said:

We've been able to reproduce this here.  We don't have an ETA for when we might be able to fix it, because it doesn't make a whole lot of sense to us either.  However, we've been able to figure out one useful thing: it only happens if you use the Connector Cache.  If you edit your connector and set the Cache Size to 0, or if you create another connector with the Cache set to 0, you won't see this.  When we use the cache, we change how we build boot images, and that apparently matters.  So if this is a problem, and you don't want to just wait out the background process, you can try with the cache disabled.

 

Thank you for the quick response.  I can confirm that this workaround resolved the issue and doesn't seem like too much of a hassle for now.  I will monitor this thread and would appreciate an update once the bug has been resolved.  I'm sure other Google searchers would appreciate it as well, considering that this is one of the first results that come up.  :)

Link to comment
  • 0

More Information:

I am experiencing the same problem.    I first experienced the problem on my Office 2016 Layer, but I figured out that I am experiencing the problem on ANY layer that I run NGEN on.

 

I do not have problems when I NGEN from the \framework\v4 directory.     I only see the problem from the \framework64\v4 directory.

 

I systematically increased my layer size.    I went from 10GB to 15GB to 20GB to 25GB.    It fills up the entire layer no matter the size.

 

ALSO:

I originally thought that I must have forgot to NGEN the OS Layer.    I reopened the OS layer and ran an NGEN again.   I then closed the layer and opened new App Layers.    This did NOT solve the problem.

 

My Environment:

- Citrix 7.18 

- Dell servers

- Dell iSCSI Disk

- NVIDIA TESLA GPUs

- Citrix HyperVisor

 

Link to comment
  • 0

I'm not 100% convinced the setting of the cache size to 0 is the issue. When I originally did this the performance returned for doing an 'ngen update' and finalizing of the layer. The size of the app was remaining relatively small. But as I progressed and built more and more layers, performance of 'ngen update' slowed and the layers grew in size. Examples:

 

  • JRE 8 Update 66 - 4.3GB (Max Layer Size 5GB)
  • Adobe Shockwave - 4.3GB (Max Layer Size 5GB)
  • SQL Server Native Client 2005 - 4.2GB (Max Layer Size 5GB)

I've done these layers before on a different OS Layer (same Operating System though) and they were around the 2-2.5GB mark. I've now rebooted my ELM appliance to build a Platform Layer that will have Server VDA, WEM Agent, Workspace App and the Citrix Quality Indicator. Will be interesting to compare the final size of this layer compared to these very small footprint installs. Will provide an update when my testing is completed.

 

Thanks,

 

Matt

Link to comment
  • 0

Is there any update on progress on resolving this issue? I notice it is not listed in known issues in the documentation at this stage. Do we know what version of ELM this crept into play and if so, is there a way to downgrade the appliance to a known working version? Really want to be able to restore the use of the cache again as it is painfully slow to create/modify layers.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...