Jump to content
Welcome to our new Citrix community!

Netscaler Integrated Cacheing for HTTP callouts only


Josh Slaney

Recommended Posts

Hello,   I have implemented an HTTP callout in a responder policy to check a web page and look for a particular phrase.   I'm trying to limit the number of requests that get sent to the callout server by setting the "Cache Expiration Time(in secs)" setting within the callout policy.   According to this guide: https://docs.citrix.com/en-us/citrix-adc/current-release/appexpert/http-callout/caching-http-callout-responses.html I need to configure integrated caching on the appliance.   Once the feature is enabled it uses the Content Group: "calloutContentGroup" for caching the responses.  However, enabling this feature turns on some default Response and Request policy labels that are bound at the Default global levels.   I need to ensure nothing is getting cached except the callouts.  What is the best way to enable the feature and ensure that only the callout traffic is getting cached? I dont see that any of the cache policy below seem to store in "calloutContentGroup"   If I unbind the cache policies globally, will that impact the callout caching?  The callout policy is set to send to a non-addressable VIP on the same load balancer.   A couple of options I've been entertaining:

 

Option A:

1. Unbind the Default Global caching Request/Response PolicyLabel policies

2. Run some tests to confirm if the netscaler is caching the callout requests/responses

 

Option B:

1. Unbind the Default Global caching Request/Response PolicyLabel policies

2. Bind the PolicyLabel policies on the Non-Addressable Callout VIP that lives on the same netscaler.

 

The following Policy Labels gets applied at the global level when the feature is enabled:

add cache contentGroup DEFAULT
set cache contentGroup NSFEO -maxResSize 1994752
add cache contentGroup BASEFILE -relExpiry 86000 -weakNegRelExpiry 600 -maxResSize 256 -memLimit 2
add cache contentGroup DELTAJS -relExpiry 86000 -weakNegRelExpiry 600 -insertAge NO -maxResSize 256 -memLimit 1 -pinned YES
add cache contentGroup ctx_cg_poc -relExpiry 86000 -weakNegRelExpiry 600 -insertAge NO -maxResSize 500 -memLimit 256 -pinned YES
add cache policy _nonGetReq -rule "!HTTP.REQ.METHOD.eq(GET)" -action NOCACHE
add cache policy _advancedConditionalReq -rule "HTTP.REQ.HEADER(\"If-Match\").EXISTS || HTTP.REQ.HEADER(\"If-Unmodified-Since\").EXISTS" -action NOCACHE
add cache policy _personalizedReq -rule "HTTP.REQ.HEADER(\"Cookie\").EXISTS || HTTP.REQ.HEADER(\"Authorization\").EXISTS || HTTP.REQ.HEADER(\"Proxy-Authorization\").EXISTS || HTTP.REQ.IS_NTLM_OR_NEGOTIATE" -action MAY_NOCACHE
add cache policy _uncacheableStatusRes -rule "! ((HTTP.RES.STATUS.EQ(200)) || (HTTP.RES.STATUS.EQ(304)) || (HTTP.RES.STATUS.BETWEEN(400,499)) || (HTTP.RES.STATUS.BETWEEN(300, 302)) || (HTTP.RES.STATUS.EQ(307))|| (HTTP.RES.STATUS.EQ(203)))" -action NOCACHE
add cache policy _uncacheableCacheControlRes -rule "((HTTP.RES.CACHE_CONTROL.IS_PRIVATE) || (HTTP.RES.CACHE_CONTROL.IS_NO_CACHE) || (HTTP.RES.CACHE_CONTROL.IS_NO_STORE) || (HTTP.RES.CACHE_CONTROL.IS_INVALID))" -action NOCACHE
add cache policy _cacheableCacheControlRes -rule "((HTTP.RES.CACHE_CONTROL.IS_PUBLIC) || (HTTP.RES.CACHE_CONTROL.IS_MAX_AGE) || (HTTP.RES.CACHE_CONTROL.IS_MUST_REVALIDATE) || (HTTP.RES.CACHE_CONTROL.IS_PROXY_REVALIDATE) || (HTTP.RES.CACHE_CONTROL.IS_S_MAXAGE))" -action CACHE -storeInGroup DEFAULT
add cache policy _uncacheableVaryRes -rule "((HTTP.RES.HEADER(\"Vary\").EXISTS) && ((HTTP.RES.HEADER(\"Vary\").INSTANCE(1).LENGTH > 0) || (!HTTP.RES.HEADER(\"Vary\").STRIP_END_WS.SET_TEXT_MODE(IGNORECASE).eq(\"Accept-Encoding\"))))" -action NOCACHE
add cache policy _uncacheablePragmaRes -rule "HTTP.RES.HEADER(\"Pragma\").EXISTS" -action NOCACHE
add cache policy _cacheableExpiryRes -rule "HTTP.RES.HEADER(\"Expires\").EXISTS" -action CACHE -storeInGroup DEFAULT
add cache policy _imageRes -rule "HTTP.RES.HEADER(\"Content-Type\").SET_TEXT_MODE(IGNORECASE).STARTSWITH(\"image/\")" -action CACHE -storeInGroup DEFAULT
add cache policy _personalizedRes -rule "HTTP.RES.HEADER(\"Set-Cookie\").EXISTS || HTTP.RES.HEADER(\"Set-Cookie2\").EXISTS" -action NOCACHE
add cache policy ctx_images -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS_INDEX(\"ctx_file_extensions\").BETWEEN(101,150)" -action CACHE -storeInGroup ctx_cg_poc
add cache policy ctx_web_css -rule "HTTP.REQ.URL.ENDSWITH(\".css\")" -action CACHE -storeInGroup ctx_cg_poc
add cache policy ctx_doc_pdf -rule "HTTP.REQ.URL.ENDSWITH(\".pdf\")" -action CACHE -storeInGroup ctx_cg_poc
add cache policy ctx_web_JavaScript -rule "HTTP.REQ.URL.ENDSWITH(\".js\")" -action CACHE -storeInGroup ctx_cg_poc
add cache policy ctx_web_JavaScript-Res -rule "HTTP.RES.HEADER(\"Content-Type\").CONTAINS(\"application/x-javascript\")" -action CACHE -storeInGroup ctx_cg_poc
add cache policy ctx_NOCACHE_Cleanup -rule TRUE -action NOCACHE
add cache policylabel _reqBuiltinDefaults -evaluates REQ
add cache policylabel _resBuiltinDefaults -evaluates RES
bind cache policylabel _reqBuiltinDefaults -policyName _nonGetReq -priority 100 -gotoPriorityExpression END
bind cache policylabel _reqBuiltinDefaults -policyName _advancedConditionalReq -priority 200 -gotoPriorityExpression END
bind cache policylabel _reqBuiltinDefaults -policyName _personalizedReq -priority 300 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _uncacheableStatusRes -priority 100 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _uncacheableVaryRes -priority 200 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _uncacheableCacheControlRes -priority 300 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _cacheableCacheControlRes -priority 400 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _uncacheablePragmaRes -priority 500 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _cacheableExpiryRes -priority 600 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _imageRes -priority 700 -gotoPriorityExpression END
bind cache policylabel _resBuiltinDefaults -policyName _personalizedRes -priority 800 -gotoPriorityExpression END
bind cache global NOPOLICY -priority 185883 -gotoPriorityExpression USE_INVOCATION_RESULT -type REQ_DEFAULT -invoke policylabel _reqBuiltinDefaults
bind cache global NOPOLICY -priority 185883 -gotoPriorityExpression USE_INVOCATION_RESULT -type RES_DEFAULT -invoke policylabel _resBuiltinDefaults

 

 

 

Link to comment
Share on other sites

1) Caching on callouts is only good for if the value you are requesting the callout to return is NOT changing on a per user basis.  

Example A: if using callout to return a list of IPs on current blacklist, so the ADC can compare if current IP is on current blacklist or not (when you only expect callout list to change every 10 minutes or so), this would be cacheable.

Example B:  if you use the callout to take an client ip from the adc (on a per request basis) and the callout compares against its list and returns a yes on blacklist/no its not value, then this callout results changes per callout request (ip passed in) and SHOULD NOT be cached.

 

So, if the "phrase" you look for is something that doesn't change per user request or callout invocation, it is possibly cacheable. 

 

2) Next, cache management.

You could treat the callout web site as a regular web lb vserver and do full cache policy management on the lb vserver. Useful if you have complicated cache determinataions.

OR, you can just use the callout's cache request for "x time" setting.   And it will just refresh on the interval (which sounds like what you are doing).

In either case, yes, Integrated Caching is being used and since the "default" policies means some content on other apps will be cached, you should use policies to control caching of anything else.

 

All the built in cache policies tend to be low priority on the response time global default bind point.

Therefore, if you create an explicit cache_pol_nocache with a NOCACHE action and you bind it to any vserver, then the vserver binding is higher precedence than the global defaults and this should prevent caching for all vservers.

 

If you create the nocache policy and bind it at priority 10 on the global default response time bind point, it will be more important than the other defaults and should prevent caching (but I don't know where the callout cache behavior falls in that precedent.    But its harder to remove when you don't need it here.

Any vserver you want to have caching on, just remove the nocache policy at the vserver level and use custom policies at the vserver bindings (or the global defaults apply.)

 

Advanced engine precedence:  global override is more important than vserver (cs vserver processes before lb vserver tiers), and vserver is more important than global default.

 

Also, remember to change caching behavior:

1) disable ns feature ic

2) flush cache

3) change policies

4) re-enable ns feature ic

 

 

  • Like 1
Link to comment
Share on other sites

1 hour ago, Rhonda Rowland1709152125 said:

1) Caching on callouts is only good for if the value you are requesting the callout to return is NOT changing on a per user basis.  

Example A: if using callout to return a list of IPs on current blacklist, so the ADC can compare if current IP is on current blacklist or not (when you only expect callout list to change every 10 minutes or so), this would be cacheable.

Example B:  if you use the callout to take an client ip from the adc (on a per request basis) and the callout compares against its list and returns a yes on blacklist/no its not value, then this callout results changes per callout request (ip passed in) and SHOULD NOT be cached.

 

So, if the "phrase" you look for is something that doesn't change per user request or callout invocation, it is possibly cacheable. 

 

2) Next, cache management.

You could treat the callout web site as a regular web lb vserver and do full cache policy management on the lb vserver. Useful if you have complicated cache determinataions.

OR, you can just use the callout's cache request for "x time" setting.   And it will just refresh on the interval (which sounds like what you are doing).

In either case, yes, Integrated Caching is being used and since the "default" policies means some content on other apps will be cached, you should use policies to control caching of anything else.

 

All the built in cache policies tend to be low priority on the response time global default bind point.

Therefore, if you create an explicit cache_pol_nocache with a NOCACHE action and you bind it to any vserver, then the vserver binding is higher precedence than the global defaults and this should prevent caching for all vservers.

 

If you create the nocache policy and bind it at priority 10 on the global default response time bind point, it will be more important than the other defaults and should prevent caching (but I don't know where the callout cache behavior falls in that precedent.    But its harder to remove when you don't need it here.

Any vserver you want to have caching on, just remove the nocache policy at the vserver level and use custom policies at the vserver bindings (or the global defaults apply.)

 

Advanced engine precedence:  global override is more important than vserver (cs vserver processes before lb vserver tiers), and vserver is more important than global default.

 

Also, remember to change caching behavior:

1) disable ns feature ic

2) flush cache

3) change policies

4) re-enable ns feature ic

 

 

For further clarification on item #1.  I have an a responder policy set match on any source IP that has a negative IP reputation category and  if that matches  perform a callout to a web server to check an "Allow List".  Each callout is a url string followed by the Source IP address in the callout policy.  The url stem expression in the callout config is similar to this:

"/urlstring1/urlstring2/"+CLIENT.IP.SRC

The callout web server is able to take each individual URL request and responds with "yes" or "no" based off the URL request string.  So if the CLIENT.IP.SRC portion of the URL is on a list it responds with "yes". Otherwise it responds with no.  The reason I'm looking turning on callout caching is because often times I see hundreds of hits over in a very short time from the same IP. For example: someone runs a scan on a website and sends 500 requests over a 2 minute span from the same IP.  I don't want the callout to peform 500 different checks to the web server for that single URL. I'd rather just cache that for several minutes.   I have seen quite a few false positives with the IP reputation and it tends to take 24-48 hours for them to recategorize an IP. This has impacted some of our users with the false positives.  So my main goal here is to use this callout list as a 2nd check for us to quickly "allowlist" certain IP if they are impacted by a  mis-categorization. 

I have confirmed the callout only happens if a negative IP reputation category is matched with the 1st part of the policy.  The responder policy looks similar to this:

CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(SCANNERS)&&(SYS.VSERVER("CALLOUT _VIP").STATE.EQ(UP)&&SYS.HTTP_CALLOUT(CALLOUT_POLICY).CONTAINS("no"))

This will match if a source IP matches on the scanners repuation category AND the VIP is up and the callout return contains no.  I've set the cache timeout to 3 minutes and confirmed that it appears to be working as expected.  For example,  if the callout is sent with  /urlstring1/urlstring2/1.1.1.1 and I get a response of "no" it will continue to match on that policy for the next 3 minutes even if i add 1.1.1.1 to my allowlist on the web server.

 

For Item #2. I removed the default global binding and my caching still appears to be working for the set designated time frame in the Callout settings.  However, I am not seeing any cached objects on the appliance like I was before.  I reapplied the default policy at the VIP level for the CALLOUT_VIP and I still don't see any hits.   I suspect I will need to follow the steps you listed here

"Also, remember to change caching behavior:

1) disable ns feature ic

2) flush cache

3) change policies

4) re-enable ns feature ic"

Your suggestion on setting a no cache policy at the global level with a lower priority is a good idea.  Also setting it at the VIP level is do-able, but I'd rather avoid that due to the sheer # of VIP that i have.  

 

Maybe I'm not understanding completely this phrase: "

OR, you can just use the callout's cache request for "x time" setting.   And it will just refresh on the interval (which sounds like what you are doing).

In either case, yes, Integrated Caching is being used and since the "default" policies means some content on other apps will be cached, you should use policies to control caching of anything else."

-Can I just remove the default global bind points for the global policy?

-If i do that will the "x time" setting still work?  The reason I ask is because I did see some hits when they were bound globally.  When I looked at the cached objects on the appliance I could see the URL for the callout.  After removing the global bindings, it appears the cacheing is still working, but I no longer see the cache object for the callout URL.  I will try to do the steps above to see if that helps.

Link to comment
Share on other sites

First:

I *would not* remove default policy bindings as you change the default policy behavior of the adc.   Means possibly unexpected results during upgrades and unexpected results as default behavior isn't present if you engage support or need caching elsewhere later.  You would have to restore settings to get back to "expected" results.

 

But if you use a nocache policy, you can turn off caching without changing default behavior; remove a custom policy in future if you need to.  So a higher priority/importance policy trumps lower priority policies.  Higher priority means lower index (in my case).  Policy priority 10 nocache (high importance) trumps policy to cache at priority 8000 (very low importance).

 

The policy allows you to turn off caching without changing default behaviors.  If you don't want to tag every vserver, then bind the nocache to the global default at priority 10; leaving the regular global default policies at their normal low priority 8000-ish range.  If the callout cache is being caught by this, then we only have to change its policy and not all other features. Both methods work, but I don't want to change fundamentals if I don't have to; bypass sure, but change I typically don't.

 

Second:

If you need complex parameterized caching rules, then writing caching policies and binding the callouts lb vserver would give you full cache control of callout responses.

 

Alternatively, the callouts built in setting for "cache for x amount of time" is in place of this as a simple rotate cache every 5 minutes (for example).  So you don't need to write your own callout cache policies, just use this. you still need a nocache policy to assign to keep other content from being cached.

 

The problem is this cache timeout on the callout, can't distinguish a YES for IP1 from a NO from IP2, IF i'm understanding how your callout works. You invoke callout and it returns a value and for next 5 minutes all callouts requests are cached, regardless of IP. 

Instead, If you wanted to cache based on which IP we are requesting callout for, you would have to write parameterized cache policies which is more complex than you probably want but we would cache the ip1 results separate from ip2 results POTENTIALLY.

 

My concern is that a flat cache behavior and the callout takes an IP and spits out yes vs no.  You are caching first response.

If IP1 (valid user) gets a NO, and you cache this.

Then the next 10 IPs that come in also are assigned NO whether on list or not, because you have cached callout result. You're not evaluating per IP.

 

If that's not what's happening great.  But caching per request parameter is a more complex cache policy config.

 

 

 

 

 

Link to comment
Share on other sites

15 hours ago, Rhonda Rowland1709152125 said:

First:

I *would not* remove default policy bindings as you change the default policy behavior of the adc.   Means possibly unexpected results during upgrades and unexpected results as default behavior isn't present if you engage support or need caching elsewhere later.  You would have to restore settings to get back to "expected" results.

 

But if you use a nocache policy, you can turn off caching without changing default behavior; remove a custom policy in future if you need to.  So a higher priority/importance policy trumps lower priority policies.  Higher priority means lower index (in my case).  Policy priority 10 nocache (high importance) trumps policy to cache at priority 8000 (very low importance).

 

The policy allows you to turn off caching without changing default behaviors.  If you don't want to tag every vserver, then bind the nocache to the global default at priority 10; leaving the regular global default policies at their normal low priority 8000-ish range.  If the callout cache is being caught by this, then we only have to change its policy and not all other features. Both methods work, but I don't want to change fundamentals if I don't have to; bypass sure, but change I typically don't.

 

Second:

If you need complex parameterized caching rules, then writing caching policies and binding the callouts lb vserver would give you full cache control of callout responses.

 

Alternatively, the callouts built in setting for "cache for x amount of time" is in place of this as a simple rotate cache every 5 minutes (for example).  So you don't need to write your own callout cache policies, just use this. you still need a nocache policy to assign to keep other content from being cached.

 

The problem is this cache timeout on the callout, can't distinguish a YES for IP1 from a NO from IP2, IF i'm understanding how your callout works. You invoke callout and it returns a value and for next 5 minutes all callouts requests are cached, regardless of IP. 

Instead, If you wanted to cache based on which IP we are requesting callout for, you would have to write parameterized cache policies which is more complex than you probably want but we would cache the ip1 results separate from ip2 results POTENTIALLY.

 

My concern is that a flat cache behavior and the callout takes an IP and spits out yes vs no.  You are caching first response.

If IP1 (valid user) gets a NO, and you cache this.

Then the next 10 IPs that come in also are assigned NO whether on list or not, because you have cached callout result. You're not evaluating per IP.

 

If that's not what's happening great.  But caching per request parameter is a more complex cache policy config.

 

 

 

 

 

As always, you're an awesome resource for Netscaler.  I didn't even think to check simulating multiple IP addresses with different results to see if the same cache response was used for both regardless of the outcome.  I do see  cache objects for each unique individual URL on another netscaler where i have already enabled this with the defaults for the calloutContentGroup.  In your opinion would this indicate each unique URL response is cached?

 

for example:

> show cache object -group calloutContentGroup    
0x00005500601e00004230  calloutContentGroup     GET     //hostname.host.com:443/urlsring1/urlstring2/162.142.125.41
0x00005500655900004258  calloutContentGroup     GET     //hostname.host.com:443/urlsring1/urlstring2/162.142.125.60
0x000005507bd600004281  calloutContentGroup     GET     //hostname.host.com:443/urlsring1/urlstring2/162.142.125.58

Link to comment
Share on other sites

Without seeing your callout, I assumed it was handling it differently.  That looks like its working as you expected, so I think so.  (Glad to have been mistaken.)

 

Since the Callout returns data either as a response or a body content, you should be able to do a quick test in a browser.

What you could do is test your callout manually in a browser:

http://<calloutvip>/pagename?ip=value (or whatever your callout request does, if it passes data in header, you would need a request modifier plugin like tamper data or a proxy simulate)

Then you could see what the browser returns for IP1 vs IP2.  And you would be able to confirm the cached by ADC response through the via header.

 

If because the request URL is different, means the browser is caching separate responses, then the callout caching is GREAT for this scenario.

If the URL was the same and only one response was being cached regardless of IP, that would be the problem.

 

It looks like you are getting the former result, but you might just "try" the callout directly in a browser to see.

Link to comment
Share on other sites

19 minutes ago, Rhonda Rowland1709152125 said:

Without seeing your callout, I assumed it was handling it differently.  That looks like its working as you expected, so I think so.  (Glad to have been mistaken.)

 

Since the Callout returns data either as a response or a body content, you should be able to do a quick test in a browser.

What you could do is test your callout manually in a browser:

http://<calloutvip>/pagename?ip=value (or whatever your callout request does, if it passes data in header, you would need a request modifier plugin like tamper data or a proxy simulate)

Then you could see what the browser returns for IP1 vs IP2.  And you would be able to confirm the cached by ADC response through the via header.

 

If because the request URL is different, means the browser is caching separate responses, then the callout caching is GREAT for this scenario.

If the URL was the same and only one response was being cached regardless of IP, that would be the problem.

 

It looks like you are getting the former result, but you might just "try" the callout directly in a browser to see.

The URL Stem expression for the callout is going to make it so each unique source IP address ends up as a different URL request:

"/urlstring1/urlstring2/"+CLIENT.IP.SRC

 

With regards to my Responder policy: 

CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(SCANNERS)&&(SYS.VSERVER("CALLOUT _VIP").STATE.EQ(UP)&&SYS.HTTP_CALLOUT(CALLOUT_POLICY).CONTAINS("no"))

 

I have bracketed the 2nd half of the policy because I want that evaluation to happen only if the VIP is up.  I know parenthesis change the order of evaluation.  If i definitely want to ensure the 1st half of the policy is matched before attempting the 2nd half, would it be better to change this policy to the following and bracketing the 1st portion of the policy?

(CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(SCANNERS))&&(SYS.VSERVER("CALLOUT _VIP").STATE.EQ(UP)&&SYS.HTTP_CALLOUT(CALLOUT_POLICY).CONTAINS("no"))

Link to comment
Share on other sites

[TL/DR version: The parentheses don't matter much here, the order of clauses really, really does. I originally thought it was just about the callout being third and then I saw the callout vserver UP test and thought you had an issue with an undefined callout return and then finally realized, I think you've done it correctly...but unpacked the reasoning below anyway.]

I would still *test* the down state scenarios to verify the results match expectations.

 

First:  So on caching: I think you are now good. Sorry for the earlier misunderstanding.

 

On policy processing:

1 hour ago, Josh Slaney said:

With regards to my Responder policy: 

CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(SCANNERS)&&(SYS.VSERVER("CALLOUT _VIP").STATE.EQ(UP)&&SYS.HTTP_CALLOUT(CALLOUT_POLICY).CONTAINS("no"))

 

Next:  Boolean logic and compilers...

If you have a clause:  A && B, in most situations B is only evaluated if A is true. If A is false, there is no way the clause "A&&B" is ever true so B isn't tested at all.

 

So, for something like this:

(A && B) && C

is same as

A && B && C

Technically you could make it:  A && (B && C), but unless you starting including OR clauses, these should be equivalent. This third one is probably the best to guarantee though.** (I wish there was a better way; but make sure the callout is robust and/or use a backup vserver to spit a safety response is another potential way to handle this.)

 

The parentheses don't matter in this clause. Neither hurt nor help.  (If we started seeing ORs things would be different).

Anytime A is false, B and C will never be evaluated, so no hope of true valuation.

The evaluation is usually in clause order and the parentheses for this particular expression won't change that.

 

So, for a regular*** set of comparisons (A is someip and B is somepage and C is something else), C will never evaluate unless A && B are true (and they will evaluate first in clause order).

Callout down is not necessarily regular.

But you've worked around this (i'm unpacking why below).  Because, I first thought it would not do what you wanted and as I got to the end of the explanation and I realized it should.

 

The potential problem you have is this:

You are using your B clause to confirm the callout is UP.  But your issue is in the C clause, the callout itself. (that's where I realized I was wrong)

 

Anytime you have a callout defined it has a destination by either vserver (which is an ip:port combo) or a specific destination ip:port.

And anytime the destination is DOWN, the callout is down.

And anytime the callout object is DOWN, the callout invocation result (whether boolean or text-based) is UNDEFINED.

 

By default, the responder feature and policies default UNDEFINED action is NOOP (do nothing).  Unless you change it to drop/reset.

If your policy expression only had the Callout in it, then the following three things would be possible.

Reminder:  A TRUE responder action is going to DROP traffic for this example...

- Callout True means IP on blacklist and Responder will DROP

- Callout False means IP not on blacklist and Responder will not apply; traffic is allowed.

- Callout Down results in Callout Expression evaluation UNDEFINED; and responder will not apply (NOOP); traffic is allowed and not stopped by feature.

 

When you are in a compound expression, the UNDEF trumps all:

For A && B && C expressions, the different combos of true and false can be evaulated.

T && T && F is false

T && T && T is true

 

But an undefined result in any clause, results in the policy taking an undefined evaluation. However, you **should** be preventing this with the clause order you mentioned above:

T && undef && ??? would be an undefined result.

T && T && undef would be an undefined result.

T && F && n/a  should result in a FALSE because the vserver test should fail, making the result of the callout unnecessary so its undefined result shouldn't be seen.

 

 

 

 

 

 

 

Link to comment
Share on other sites

35 minutes ago, Rhonda Rowland1709152125 said:

[TL/DR version: The parentheses don't matter much here, the order of clauses really, really does. I originally thought it was just about the callout being third and then I saw the callout vserver UP test and thought you had an issue with an undefined callout return and then finally realized, I think you've done it correctly...but unpacked the reasoning below anyway.]

I would still *test* the down state scenarios to verify the results match expectations.

 

First:  So on caching: I think you are now good. Sorry for the earlier misunderstanding.

 

On policy processing:

 

Next:  Boolean logic and compilers...

If you have a clause:  A && B, in most situations B is only evaluated if A is true. If A is false, there is no way the clause "A&&B" is ever true so B isn't tested at all.

 

So, for something like this:

(A && B) && C

is same as

A && B && C

Technically you could make it:  A && (B && C), but unless you starting including OR clauses, these should be equivalent. This third one is probably the best to guarantee though.** (I wish there was a better way; but make sure the callout is robust and/or use a backup vserver to spit a safety response is another potential way to handle this.)

 

The parentheses don't matter in this clause. Neither hurt nor help.  (If we started seeing ORs things would be different).

Anytime A is false, B and C will never be evaluated, so no hope of true valuation.

The evaluation is usually in clause order and the parentheses for this particular expression won't change that.

 

So, for a regular*** set of comparisons (A is someip and B is somepage and C is something else), C will never evaluate unless A && B are true (and they will evaluate first in clause order).

Callout down is not necessarily regular.

But you've worked around this (i'm unpacking why below).  Because, I first thought it would not do what you wanted and as I got to the end of the explanation and I realized it should.

 

The potential problem you have is this:

You are using your B clause to confirm the callout is UP.  But your issue is in the C clause, the callout itself. (that's where I realized I was wrong)

 

Anytime you have a callout defined it has a destination by either vserver (which is an ip:port combo) or a specific destination ip:port.

And anytime the destination is DOWN, the callout is down.

And anytime the callout object is DOWN, the callout invocation result (whether boolean or text-based) is UNDEFINED.

 

By default, the responder feature and policies default UNDEFINED action is NOOP (do nothing).  Unless you change it to drop/reset.

If your policy expression only had the Callout in it, then the following three things would be possible.

Reminder:  A TRUE responder action is going to DROP traffic for this example...

- Callout True means IP on blacklist and Responder will DROP

- Callout False means IP not on blacklist and Responder will not apply; traffic is allowed.

- Callout Down results in Callout Expression evaluation UNDEFINED; and responder will not apply (NOOP); traffic is allowed and not stopped by feature.

 

When you are in a compound expression, the UNDEF trumps all:

For A && B && C expressions, the different combos of true and false can be evaulated.

T && T && F is false

T && T && T is true

 

But an undefined result in any clause, results in the policy taking an undefined evaluation. However, you **should** be preventing this with the clause order you mentioned above:

T && undef && ??? would be an undefined result.

T && T && undef would be an undefined result.

T && F && n/a  should result in a FALSE because the vserver test should fail, making the result of the callout unnecessary so its undefined result shouldn't be seen.

 

 

 

 

 

 

 

You nailed it! I was having undefined results if the callout VIP was down or having issues.  So my decision was to avoid any undefined hits altogether by adding the status check on the VIP.  I realize that nothing will match on the responder reputation policies unless that VIP is "UP", but I'm okay with that.  Thanks for the explanation on the policy processing.  My testing indicates that it does have to match the first half of policy before the callout ever takes place.  

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...