Jump to content
Welcome to our new Citrix community!

Question about cookie consistency and firmware upgrade


Ross Helfand

Recommended Posts

Hello,

 

have a question about CookieConsistency.  I was upgrading from Firmware 10.x (yes, I know how old that is!) and as soon as I did we started getting application errors.  I was browsing this forum and I see the answer Rhonda provided in this thread:  https://discussions.citrix.com/topic/409740-cookie-validation/  but I'm still a little confused.

 

I also see that there's a note at the bottom of this page:
https://docs.citrix.com/en-us/citrix-adc/current-release/application-firewall/cookie-protection/cookie-consistency-check.html

talking about Sessionless Cookie Consistency, and I assume that's what happened to us.

 

To summarize the problem, I upgraded one Netscaler in our HA pair and swithced over.  When I did, we started having trouble accessing a page that is protected by an APPFW, which we have set like so:

-cookieConsistencyAction block log stats

I rolled back the change and started testing on another set of Netscalers and found some syslog messages:

Cookie validation failed for .EPAYMENTEXPIRETIME <blocked>
Cookie validation failed for .EPAYMENTUSERNAME <blocked>
Cookie validation failed for .FDAT <blocked>
Cookie validation failed for citrix_ns_id_REDACTED.OURSITE.com_%2F_wat <blocked>
# then after another try:
Cookie validation failed for citrix_ns_id_REDACTED.OURSITE.com_%2F_wlf <blocked>

We hd some other cookies "exempted" using:

bind appfw profile APPFW_PCI -cookieConsistency EPAYMENTEXPIRETIME
# etc ...

And low and behold, when I explicitly added the blocked cookies (including the periods!) to the exempt list, I was able to hit the page.

 

I am not an application guy, so I'm just trying to get a better understanding of what changed between 10.x and 11.x that broke our application.  I don't understand why the application is working in firmware 10.x now.  It seems like the "-cookieConsistencyAction block" should still be causing cookie validation failures.

 

Thanks!

Link to comment
Share on other sites

What build did you upgrade to?  A lot of things changed with how sessionization features work post 11.1 compared to how it used to work in the 10/9 days.

 

Between 10 and 11.x (Post 11.1 and later for sure), the way cookie consistency validation works changed.  The original sessionization cookie isn't needed with new cookie consistency checks (which version that started at 11, 11.1 or 12.1 I can't say).

 

And just like when you go from Cookie Consistency check OFF to ON, your existing cookies that users alreayd have on their endpoints are being re-presented to the ADC without being validated so they create immediate false positives.  To fix, you have to clear all existing cookies on the endpoint and then enable feature so all app cookies are signed by the ADC and protected from that point forwards.

 

My guess is that when you upgraded the feature, you had the old tracking cookies in use and they were no longer valid with the new build. Exempting the cookies is one way; but clearing all the endpoint cookies, then enabling the new cookie protection feature might allow your existing app to function under the NEW protection model.  

So the question is did you clear all client cookies AND reset the browser when you switched to OLD build to NEW build to see if it worked?  If not, old keys create false violations.

 

If you did clear cookies and close and reopen test browser and even after this the new appfw blocks the old app, then the issue may be something about the legacy app or the legacy browser in use might be incompatible with the new security expectations. 

  

Link to comment
Share on other sites

12 hours ago, Ross Helfand said:

We hd some other cookies "exempted" using:

One last thought, it's not clear from your post but did you have cookies exempted on the 10.x build before the upgrade, then when you upgraded they were not exempted, so you had new block events.  Then you added them to the exemptions, and then things worked?

 

Any cookie on the exempt list was not protected and therefore will not cause violations. If they were exempted on 10.x pre-upgrade. But not exempted post upgrade, then these cookies were in violation on the new system.  Exempt them just like they were pre-upgrade to avoid issues. Exempted cookies are not tracked adn are not protected.  But either the cookies are being used client side OR they do not match the web domain or something else. 

 

If I misunderstood and these were never exempted pre upgrade and you only did it post upgrade to get around the issue, then try clearing cookies before testing again.  

Link to comment
Share on other sites

Hi Rhonda,

 

Thanks as always for taking the time to answer.  This story is a little convoluted which is why I was light on details.  My end goal was trying to get to build-13.0-87.9 which was the latest when I started this journey.  This application does credit card processing, so everyone is a little afraid to touch it.  ?   I hope that explains some of the strange details below.

 

What I did was:

 

Primary site HA pair:

- upgraded the secondary node from build-10.5-69.5_nc -> build-12.1-65.17_nc_64 -> build-13.0-87.9_nc_64 (this was the upgrade process recommended to me when I opened a Citrix support case).

- ran 'force failover' and did some testing to see that we had a strange new error

- switched back to the other Netscaler running 10.5 which is where it's been ever since

 

Secondary (DR site) pair (same APPFW config as in our primary site):

- This pair has been on NS13.0: Build 82.42.nc since they were built.  We had not yet tested the application (I know)

- Did some offline testing and saw we were getting the same error as when I switched over to 13.x

- Downgraded one node from NS13.0: Build 82.42.nc -> build-12.1-65.17_nc_64 -> build-11.1-65.12_nc and tested the application using new incognito browser windows every time.  Always got the same error

- Downgraded from build-11.1-65.12_nc -> build-10.5-69.5_nc and the error went away

 

We figured out it was cookies by getting one of the application guys to add some extra logging and determining that one of the important cookies was missing.  That led me down the path of setting "-cookieConsistencyAction log stats" (removing the "block") and seeing that the application started working.  Then I re-enabled "block" and finally found the syslog messages with the errors.

 

To answer a couple of your other questions, I always used a new private/incognito browser when trying to hit this particular page.  And yes, there were about seven cookies that were exempted using the "bind appfw profile APPFW_PCI -cookieConsistency COOKIENAME" syntax.

 

What I thought was kind of odd was that the "Cookie validation" errors were all similar to the names of the cookies that were already exempted from 10.x, but they had leading periods.  It would not surprise me if 10.x firmware saw the exemption line as a REGEX and therefore "EPAYMENTEXPIRETIME" and ".EPAYMENTEXPIRETIME" were both exempt.  I cannot find 10.x documentation any more, but I see that there is an optional "-isRegex" argument to the "bind appfw profile" command in the new firmware.

 

Thanks,

Ross

 

Link to comment
Share on other sites

Ok.  Any cookies exempted in original profile were exempted because AppFw was probably seeing them as violations to begin with (client side modifications or others or they didn't match website domain). So the exemption was needed to avoid breaking legitimate features.

 

Therefore you are likely to still need exemptions on your new profile.

 

Whether cli OR gui if the cookie name does not exactly match a static string YOU must enable the -isregex option to match on a pattern.  

Any browser header viewer should show you if the cookie name being set is epayment or some variation.  (The isregex for cookie name would likely have been present in 10.x/9.x as well.)

 

If you have blocking off and learning on and you test a new access tot he app; for session cookies it is IMPORTANT to close any existing browser instances before testing.  You should then identify any cookies (with new or old names) that are causing false positives and then you would need to exempt them from protection.  Tell your app team that the cookies exempted were not protected before and they will remain unprotected going forwards.  

 

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...