Jump to content
Welcome to our new Citrix community!

HTTP callout parameter limitations?


Recommended Posts

We currently proxy some services through an external bot detection service to help prevent data mining. I have written an http callout .net service to query an internal database of data miner IPs that the external proxy is unable to catch. 

 

When the traffic comes in from the external bot detection service, CLIENT.IP.SRC obviously contains the proxy service IP and the XFF header contains the actual client IP, a comma and the proxy service IP again. I need to be able to use the XFF + regex value as the parameter to pass to the service. XFF to catch the 2 IPs followed by regex to strip out the comma and second ip leaving the true ip.

 

If, for the parameter, I enter CLIENT.IP.SRC, that value is correctly gathered and passed to the service as it should be. However, if I place HTTP.REQ.HEADER(“X-FORWARDED-FOR”).REGEX_MATCH(workingregextograbip) as the parameter, or even just XFF alone, which the ADC accepts as valid syntax, the value passed is “true”.

 

Are these objects disallowed in the parameter field? Does anyone know if there is a limitation of what ADC functions or headers can be used as a callout parameter? Is there a better way of extracting the IP that I need to send to the service?

 

I have considered a REQ header rewrite before the callout that would swap the CLIENT.IP.SRC for the XFF + regex and then a post callout RES rewrite to return the value. That would get super messy as I would have to store the contents of what the regex removed in a variable somewhere as the external proxy IP is not fixed, but is a range.

 

What am I missing here?

 

Thanks in advance!

Edited by rtuttle772
Edited for clarification.
Link to comment
Share on other sites

I kind of resolved this myself. For some reason, using HTTP.REQ.HEADER(“X-FORWARDED-FOR”).BEFORE_STR(“,”) is accepted and returns the correct value. Can anyone help me understand why when using XFF alone or in conjunction with REGEX would return a value of “true” instead of the value that the above DOES return correctly?

 

Thanks!

Link to comment
Share on other sites

Share your actual expression and a sample of the header you want to process.  As there may be alternate ways to do what you want.

 

Keep in mind:

regex_match is a boolean operator that returns true/false for whether the pattern was found or not.  Just like doing eq("") or contains(""), they say whether value is found or not; they don't extract a value meeting a certain condition.

 

Regex_Select and Before_RegEx/After_RegEx are the operators you need if you want to extract strings matching a regex pattern when the regular string processing operators can't get the job done.

  • Like 1
Link to comment
Share on other sites

Thanks Rhonda,

 

The information you provided was exactly what I was looking for. In running a test against some other headers to operate against, I have the following if you care to assist.  Below I am checking the "somepath" in the GET request against the "somepath" in the Referer header. 

 

HEADER - 

 

GET /somepath/morepath/details.aspx?d=U_yPZkz_EyqG3oWT70F98mmQA836tHoKCMPxUkzWFRKz3IdDiucczWic9wvMJrT1SlSmEVItFu9YNwWC5mLUxZKRjLE3bYlbQ4c6yte_A3bb7cZe0&t=27117261 HTTP/1.1\r\n

 

Host: foobar.com

User-Agent: Mozilla Firefox/3.0.3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Referer: https://foobar.com/somepath/morepath/Info.aspx?12345

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Date: Thu, 09 Oct 2008 18:25:00 GMT

Cookie: sessionid=100xyz

 

 

- Responder POLICY

 

HTTP.REQ.URL.CONTAINS("details.aspx") && HTTP.REQ.URL.PATH.GET(1)!=HTTP.REQ.HEADER("Referer").VALUE(0).REGEX_SELECT(re~(?<=https://foobar.com/).*(?=/morepath)~)

 

 

FACTS:

By itself, HTTP.REQ.URL.PATH.GET(1) - correctly identifies "somepath" in the GET request

 

By itself, HTTP.REQ.HEADER("Referer").VALUE(0).REGEX_SELECT(re~(?<=https://foobar.com/).*(?=/morepath)~) -seems to  incorrectly identify the instance of  "somepath" in the GET request and not the one in the Referer header

 

When I run the policy through the expression tester with somepath in both header values, I get false, If I change one of the values to something else, I get true. However, when I bind to a VIP to test, even matching values in those fields are marked as true. 

 

Any ideas?

 

Thanks!

 

 

 

Link to comment
Share on other sites

Sorry, for the delay was off and didn't have access to a NetScaler.

I think there is a much easier way to do this than with the regex you are trying to use.  And to be honest, I didn't know you could do <expr1> == <expr2> or <expr1> != <expr2) type clauses, which is actually pretty cool...and I don't know why I didn't realize that would work before.  

 

So for the following URL and sample headers:

GET /somepath/morepath/details.aspx?d=U_yPZkz_EyqG3oWT70F98mmQA836tHoKCMPxUkzWFRKz3IdDiucczWic9wvMJrT1SlSmEVItFu9YNwWC5mLUxZKRjLE3bYlbQ4c6yte_A3bb7cZe0&t=27117261 HTTP/1.1\r\n

 

Host: foobar.com

User-Agent: Mozilla Firefox/3.0.3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Referer: https://foobar.com/somepath/morepath/Info.aspx?12345

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Date: Thu, 09 Oct 2008 18:25:00 GMT

Cookie: sessionid=100xyz

 

Try the following expressions (as an example): (I used == just to test the results instead of !=)

http.req.url.path.get(1) is going to isolate the first directory in the path of the original URL request:  somepath

http.req.header("referer") is going to retrieve the entirety of the Referer header:  https://foobar.com/somepath/morepath/Info.aspx?12345

 

Instead of using the regex operator, typecast the contents of the referer header to a URL datatype and then do a URL comparison like so:

http.req.url.path.get(1) == http.REQ.HEADER("referer").TYPECAST_HTTP_URL_T.PATH.GET(1)

somepath == somepath

 

The trick is that you aren't going to be able to make it case-insensitive in this model though, unless you do this:

http.req.url.path.get(1).set_text_mode(ignorecase) == http.REQ.HEADER("referer").TYPECAST_HTTP_URL_T.PATH.GET(1).set_text_mode(ignorecase)

 

This does work in the evaluator; it would need to be tested in the evaluator.

 

For the actual test you are trying to run above, make the following changes:

So change your original expression from this: 

HTTP.REQ.URL.CONTAINS("details.aspx") && HTTP.REQ.URL.PATH.GET(1)!=HTTP.REQ.HEADER("Referer").VALUE(0).REGEX_SELECT(re~(?<=https://foobar.com/).*(?=/morepath)~)

 

to:  (I would recommend the parentheses to limit the scope of the != clause)

http.req.url.set_text_mode(ignorecase).contains("/details.aspx") && (http.req.url.path.get(1).SET_TEXT_MODE(ignorecase) != http.REQ.HEADER("referer").TYPECAST_HTTP_URL_T.PATH.GET(1).SET_TEXT_MODE(ignorecase))

 

The original regex operator you tried is being messed up by the  apparent scope of the (?<=) operator. Or this could just be an issue with the evaluator, not being able to handle this particular expression. In either case, there is an easier way to do it without regex_select.

 

Next:

In general, this should be more efficient to process than the regex...but the question is what is your expression going to do.

So if you first directory in the current URL doesn't match the first directory in the Referer header, then you want to trigger this callout?    IF that's the case you should be okay.

But test both a valid and invalid trigger on real traffic.

 

Finally, the Evaluator:

I'm on a 12.1 build and while some of the evaulator issues were fixed in 12.1 a few still remain. So regarding these comments:

 

On 7/3/2019 at 7:15 PM, Joseph Tuttle said:

FACTS:

By itself, HTTP.REQ.URL.PATH.GET(1) - correctly identifies "somepath" in the GET request

 

By itself, HTTP.REQ.HEADER("Referer").VALUE(0).REGEX_SELECT(re~(?<=https://foobar.com/).*(?=/morepath)~) -seems to  incorrectly identify the instance of  "somepath" in the GET request and not the one in the Referer header

 

When my expressions got a little too complex, the highlight tool in the evaluator selected wrong. But building up the expression elements one at a time it was mostly getting the right elements in the referer header. Your regex wigged it out big time though.

 

Here's the thing, since the elemental tests worked as expected, then its reasonable to expect teh policy engine is going to do this right, but the evaluator is being quirky.  Basically, ever since they migrated the evaluator from the java code to html 5 it has had some issues. 12.0 was very bad and 12.1 fixed many of the display issues, but a few still remain.  So it does a good job of allowing us to test most expressions, but a few still get messed up.  I think if you moved to a live test, it would probably work especially with the less complex expression.  

 

  • Like 2
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...