Jump to content
Welcome to our new Citrix community!

convert query to parameter using netscaler


Dany Demers

Recommended Posts

Hi, we have a specific need to pass settings from a short URL to be setttings of a long URL but I'm not certain this can be achieved via rewrite policy (or other preferred method). I'm already using rewrite with "replace ALL" to convert short stuff to long stuff and keep whatever is sent after but never converted trying to parse some parameter from the URL to send it to the backend using positional parameters.

 

The 3 settings that I need to parse are the following :

<storeNumber> = 3 digit number

<lang> = 2 digit language code

<searchValue> = variable length string

 

 

Here is an explanation of the need:

 

Customer would send this url:

https://productfinder.company.ca/?store=<storeNumber>&lang=<lang>&search=<searchValue>

 

Netscaler would have to parse it to:

http://backend/solr/solr_geoloc/select?fl=name:name_<lang>,category:category_<lang>,img:img_<lang>,brand,map:store_<storeNumber>_map,format:format_<lang>&q=((brand:*<searchValue>*)^5%20OR%20(name_<lang>:*<searchValue>*)^1)%20AND%20store_<storeNumber>_present:true

Link to comment
Share on other sites

If you are doing multiple rewrites, you might have issues depending on what else is being rewritten; but a URL rewrite can be done separate from the body rewrite as long as policies are bound with NEXT. If you are rewriting the URL in the body that you are already rewriting than you may have issues.  If you have multiple url rewrite actions to perform, you may have to use a url transform instead.

 

A responder policy can redirect. If needed.  BUT - since you are also changing host portions of urls, you probably will need a URL transform...but let me get you the query example first as just a basic expression example.  But I think with the host changes AND the existing replace all...I don't think rewrite alone will work.  However, your URL transform might have issues here to.

 

So just a path/query manipulation to kick things off:

Example 1: simplified:

ORIGINAL:  /?store=<storeNumber>&lang=<lang>&search=<searchValue>

TO:  /solr/solr_geoloc/select?fl=name:name_<lang>,category:category_<lang>,img:img_<lang>,brand,map:store_<storeNumber>_map,format:format_<lang>&q=((brand:*<searchValue>*)^5%20OR%20(name_<lang>:*<searchValue>*)^1)%20AND%20store_<storeNumber>_present:true

 

rewrite action:

add rewrite action rw_act_demo1 replace http.REQ.URL.PATH_AND_QUERY "\"/sol/solr_geoloc/select\?fl=name:name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",category:category_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",img:img_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",brand,map:store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_map,format:format_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \"&q=((brand:*\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^5%20OR%20(name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \":\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^1)%20AND%20store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_present:true\""
 

NOTE: CLI version above ("?") must be escaped as \? to copy and paste via CLI; won't harm GUI entry (FYI)

VIA GUI rewrite expression would be this:

"/sol/solr_geoloc/select?fl=name:name_" + http.REQ.URL.QUERY.VALUE("lang") + ",category:category_" + http.REQ.URL.QUERY.VALUE("lang") + ",img:img_" + http.REQ.URL.QUERY.VALUE("lang") + ",brand,map:store_" + http.REQ.URL.QUERY.VALUE("store") + "_map,format:format_" + http.REQ.URL.QUERY.VALUE("lang") + "&q=((brand:*" + http.REQ.URL.QUERY.VALUE("search") + "*)^5%20OR%20(name_" + http.REQ.URL.QUERY.VALUE("lang") + ":" + http.REQ.URL.QUERY.VALUE("search") + "*)^1)%20AND%20store_" + http.REQ.URL.QUERY.VALUE("store") + "_present:true"

 

 

So via Expression evaluator:

GET /?store=123&lang=eng&search=testme HTTP/1.1

Did this:
Result as string: /sol/solr_geoloc/select?fl=name:name_eng,category:category_eng,img:img_eng,brand,map:store_123_map,format:format_eng&q=((brand:*testme*)^5%20OR%20(name_eng:testme*)^1)%20AND%20store_123_present:true
Evaluation time in nanoseconds: 2949

 

NOTE: kind of long evaluation time though... 2 ms.

 

Example 2: using a URL transform to do all of this and host changes too

NOW:  REQUEST URL TRANSFORM FROM:  (grrr copy/paste lost your colors so I reapplied them but they may be different now)

https://productfinder.company.ca/?store=<storeNumber>&lang=<lang>&search=<searchValue>

 

http://backend.company.com/solr/solr_geoloc/select?fl=name:name_<lang>,category:category_<lang>,img:img_<lang>,brand,map:store_<storeNumber>_map,format:format_<lang>&q=((brand:*<searchValue>*)^5%20OR%20(name_<lang>:*<searchValue>*)^1)%20AND%20store_<storeNumber>_present:true

 

BUT you can't have more than 5 backreferences in a single URL transform expression... this is ugly.

I only attempted a request time URL Transform and was having issues with the response time pattern back into the client side. If you know IF the store is a only digits of a certain length or range (or a specific pattern), the lang is a 2/3/4 digit language pattern and search is probably open ended, then the response time transform MIGHT be do-able.  

I also can't test this transform easily as I don't have a way to "intercept" the the request out to server to see if it working at all.  But this might get you started.

 

If your earlier rewrite is preventing from doing the transform, then you might have to have traffic hit lb vserver 1 (on adc1) and do one set of rewrites and then send request to lb vserver 2 on a different adc and do a second level...but your scenario is complex (if it makes you feel better).

 

 

add transform profile urlt_prof_demo2
add transform action urlt_act_demo1_fullurls urlt_prof_demo2 20
set transform action urlt_act_demo1_fullurls -priority 20 -reqUrlFrom "^https://(productfinder[.]company[.]ca)/\\?store=([^&]*)&lang=([^&]*)&search=([^&]*)$" -reqUrlInto "http://backend.company.com/solr/solr_geoloc/select\\?fl=name:name_$3,category:category_$3,img:img_$3,brand,map:store_$2_map,format:format_$3&q=((brand:*$4*)^5%20OR%20(name_$3:*$4*)^1)%20AND%20store_$2_present:true"
 

# NOTE I didn't create the URLTransform policy to reference the profile or bind it yet.

 

=====

See if any of this helps at all.

 

  • Like 1
Link to comment
Share on other sites

wow that is a huge job on your part, I'm trying to decipher all of it but that's way more complex than anything I've ever done before in the Netscaler, the host change isn't really necessarey, it was just a reference that I'm sending the query to an internal server which name is different but it doesn't proces the url based on hostname, it simply accept the request as is. The important part is the "variable value". I still have a hard time understanding how you parse the value but I will give this a shot and see what happens and get back to you. As for the lenght of the settings I guess it was only valid for the url transfom path?

Link to comment
Share on other sites

OK - if you don't have to do a host change - then the REWRITE policy is the BEST way to do this and skip the url transform which is all sorts of regex based.  

(The rewrite was easy, just did it in the gui and let the config spit it out...the URL Transform was not straightforward)

 

Let's see if I can help match things up for you:

So just a path/query manipulation to kick things off:

Example 1: simplified:

ORIGINAL:  /?store=<storeNumber>&lang=<lang>&search=<searchValue>

TO:  /solr/solr_geoloc/select?fl=name:name_<lang>,category:category_<lang>,img:img_<lang>,brand,map:store_<storeNumber>_map,format:format_<lang>&q=((brand:*<searchValue>*)^5%20OR%20(name_<lang>:*<searchValue>*)^1)%20AND%20store_<storeNumber>_present:true

 

http.req.url.query.value("store")   will match the value of the query parameter store

http.req.url.query.value("lang")   will match the value of the query parameter lang

http.req.url.query.value("search")   will match the value of the query parameter search

 

So in the REWRITE action, it was just about taking your current http.req.url.path_and_query (path and query elements) and rewriting them into the new output and inserting the retrieved values into their new positions:

/solr/solr_geoloc/select?fl=name:name_<lang>,category:category_<lang>,img:img_<lang>,brand,map:store_<storeNumber>_map,format:format_<lang>&q=((brand:*<searchValue>*)^5%20OR%20(name_<lang>:*<searchValue>*)^1)%20AND%20store_<storeNumber>_present:true

 

So the rewrite action uses literal text in black for the static strings and then we insert at all the appropriate positions the store, lang, and search values with the expression syntax.

 

The rewrite action targets the original path_and_query and replaces it with the new one and ignores the host.

So your get /<somepath/query> will be replaced with the get <newpath/query>

 

Just tweak the policy to trigger only when this is needed.

 

rewrite action:

add rewrite action rw_act_demo1 replace http.REQ.URL.PATH_AND_QUERY "\"/sol/solr_geoloc/select\?fl=name:name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",category:category_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",img:img_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",brand,map:store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_map,format:format_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \"&q=((brand:*\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^5%20OR%20(name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \":\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^1)%20AND%20store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_present:true\""

 

If that helps at all.

 

 

 

  • Like 1
Link to comment
Share on other sites

Hi, I'm trying your config and it's really close when I choose the evaluate request from my Netscaler (13.0-47.24) there is the addition of character at unusal place that I don't understand and to me your http.REQ.URL.QUERY.VALUE looks identical but the Netscaler add some equal sign (=) at different spot in the url and that doesn't make sense in my brain since the original request sent to the Netscaler (via built-in evaluation tool) all 3 parameter have the equal sign to them and the lang is parsed correctly but the store number and the search value arent at least not on all value in the end result:

 

Query this:

GET /?store=413&lang=en&search=mango/ HTTP/1.1

 

Return this:

 

GET /solr/solr_geoloc/select?fl=name:name_en,category:category_en,img:img_en,brand,map:store=_413_map,format:format_en&lq=((brand:*mang=o/*)^5%20OR%20(name_en&search=:mango/*)^1)%20AND%20store_413_present:true HTTP/1.1

 

 

while the expected result are this:

 

/solr/solr_geoloc/select?fl=name:name_en,category:category_en,img:img_en,brand,map:store_413_map,format:format_en&q=((brand:*mango*)^5%20OR%20(name_en:*mango*)^1)%20AND%20store_413_present:true HTTP/1.1

 

 

it's easier to compare when you copy/paste everything into notepad to see side by side (actually up and down but you get my point) the difference between the expected and the actual result. I tried googling to find the syntax of http.REQ.URL.QUERY.VALUE to see if I could correct it by myselft but that's not a well documented feature anywhere that I could find and also the fact that some random text get added to the query simply amaze me as your action doesn't even have them

Link to comment
Share on other sites

So 1) the evaluator sometimes freaks out on patterns/strings that the real system would handle and there were a lot of html encodings that could be affecting this and 2) the evaluator has a few bugs in how it tries to display the markup...so the trick is is this the evaulator or a problem in the rewrite. Usually though things the evaluator mangles will not impact real traffic.

 

In your example above you have a vertical pipe |q in a position where we had an ampersand "&q"

And that might have been me interpreting your original request wrong OR an issue in my expression OR an issue in the evaluator...

 

In a few cases, if the evaluator is weird, you may have to test live (using an alternate vip or test environment where you can make sure its not impacting production and observe it there. Proxies or web logs can help you confirm y our client-side vs. server side (or use a responder policy to "echo" the text in a message body for quick "live testing" on a test environment.)

 

My last recommendation, if the below doesn't help, is to build the rewrite in "sections" and see at what point the evaluator chokes on it.  Then you can see if its the evaluator or real behavior.  

 

So look at this area first:

[a] [target output]

/solr/solr_geoloc/select?fl=name:name_en,category:category_en,img:img_en,brand,map:store_413_map,format:format_en&q=((brand:*mango*)^5%20OR%20(name_en:*mango*)^1)%20AND%20store_413_present:true

 

[evaulator result]

GET /solr/solr_geoloc/select?fl=name:name_en,category:category_en,img:img_en,brand,map:store=_413_map,format:format_en&lq=((brand:*mang=o/*)^5%20OR%20(name_en&search=:mango/*)^1)%20AND%20store_413_present:true HTTP/1.1

 

[c] expression area that should affect this

One thought we had to escape the "?" to insert via cli. You might compare the show ns runningconfig | grep rw_act_demo1 output to the command you inserted and see if any other characters changed on insert that may need to be escaped to be entered via the CLI OR edit in the GUI since you won't have to deal with all the escaped quotes.

 

add rewrite action rw_act_demo1 replace http.REQ.URL.PATH_AND_QUERY "\"/sol/solr_geoloc/select\?fl=name:name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",category:category_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",img:img_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",brand,map:store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_map,format:format_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \"&q=((brand:*\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^5%20OR%20(name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \":\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^1)%20AND%20store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_present:true\""

 

GREEN:  There is a  problem in the brand,map:store_<store> insertion.    The action I have is store_ but what you see in evaluator is store=_

1) So, does your policy have an "=" or an "_" in it.

 

PURPLE:  This is another change from expression to evaluator

We have &q and you end up with &|q

========

Both of these I think are the evaluator messing with the output...but its hard to test.  Because there are some character formats that could be converted that we aren't aware of.

 

This would be easier to "show" you have to do than to "test"

Basically, create a copy of your action where you can work in pieces in the evaulator until you find out if it breaks or not.

Example:  

Start the evaulator with the sample request with this type of pattern:

GET /?store=413&lang=en&search=mango/ HTTP/1.1

 

And then build the rewrite action in the evaulator in sections to see if the evaulator breaks. If it does, just use the evaluator to confirm that things are mostly right and then go to a real url to see if it works. You can also use the audit policy to "audit the result" in syslog if needed.

 

[1] test first

"\"/sol/solr_geoloc/select\?fl=name:name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",category:category_\" + http.REQ.URL.QUERY.VALUE(\"lang\") 

[2] test next

"\"/sol/solr_geoloc/select\?fl=stuff + \",category:category_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \",img:img_\"

[3] then test:

"\"/sol/solr_geoloc/select\?fl=stuff + http.REQ.URL.QUERY.VALUE(\"lang\") + \",brand,map:store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_map,format:format_\"

 

[4] and so on:

+ http.REQ.URL.QUERY.VALUE(\"lang\") + \"&q=((brand:*\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^5%20OR%20(name_\" + http.REQ.URL.QUERY.VALUE(\"lang\") + \":\" + http.REQ.URL.QUERY.VALUE(\"search\") + \"*)^1)%20AND%20store_\" + http.REQ.URL.QUERY.VALUE(\"store\") + \"_present:true\""

 

That way you can test the parsing in pieces and then isolate if the expression is the problem or the evaluator.

 

  • Like 2
Link to comment
Share on other sites

After some testing seem that you were right in the fact the I can't trust the evaluator (lesson learned), I builded a dummy IIS server to send request to it (it didn't matter that IIS replied with 404 not found) and used the Netscaler built-in trace tool to watch the traffic sent to the this server and the result were near perfect, it only missed an asterix before the second <brandname> that I was able to insert easily based on your original query. That always amaze me all the things you can do with a Netscaler that I would have never thought possible.

Thanks alot!

 

On 6/25/2020 at 6:57 PM, Rhonda Rowland1709152125 said:

So 1) the evaluator sometimes freaks out on patterns/strings that the real system would handle and there were a lot of html encodings that could be affecting this and 2) the evaluator has a few bugs in how it tries to display the markup...so the trick is is this the evaulator or a problem in the rewrite. Usually though things the evaluator mangles will not impact real traffic.

 

In your example above you have a vertical pipe |q in a position where we had an ampersand "&q"

And that might have been me interpreting your original request wrong OR an issue in my expression OR an issue in the evaluator...

 

In a few cases, if the evaluator is weird, you may have to test live (using an alternate vip or test environment where you can make sure its not impacting production and observe it there. Proxies or web logs can help you confirm y our client-side vs. server side (or use a responder policy to "echo" the text in a message body for quick "live testing" on a test environment.)

Edited by Dany Demers
typo
Link to comment
Share on other sites

GLAD it worked and it was just the evaluator. Usually the evaluator is great...but really complex replacement strings and anything that gets near regex it tends to "misinterpret" the output and you get weird things. So I usually assume its likely to be the evaluator and not the expression.  (The HTML 5 client just inserts things in weird positions when it gets really complex.)

 

The evaulator is great for simple url/header insertions and manipulations and to help you confirm your expression does what you want. But for this, the live test confirmed all.

  • Like 1
Link to comment
Share on other sites

One more question for my personal knowledge related to query, how does the netscaler knows the query value? I get that we say use the value of the query "Lang" but is it the "equal" sign that tell him that its the value of the query?

 

Would it also work if the original URL was like "/lang?en" instead of "/lang=en"

Link to comment
Share on other sites

1 hour ago, Dany Demers said:

how does the netscaler knows the query value

Your mostly right in the statement after this; but since this is your inherent question, i'll answer below.

 

 

1 hour ago, Dany Demers said:

Would it also work if the original URL was like "/lang?en" instead of "/lang=en"

NOPE (to this statement) - and here's what the syntax is doing.

 

Here's what you need to know about how the epressions parse a URL (though you can find this in some of my other posts on policies lately). And the appexpert section has a section on the syntax.

 

Let's take a sample URL:

Syntax Example 1:

https://demo.company.com/dir/subdir/myscript.cgi?a1=b1&a2=b2&lang=en&other=something

Typically, this then becaome a REQUEST in the structure of:

Get /dir/subdir/myscript.cgi?a1=b1&a2=b2&lang=en&other=something HTTP1/1

Host:  demo.company.com

<other headers>

 

These expressions, match the following elements of the URL:

http.req.url   targets the "target" of the action so usually does NOT include the HOST portion of the URL:  /dir/subdir/myscript.cgi?a1=b1&a2=b2&lang=en&other=something

Then you can break down the url into elements:

http.req.url.path   IS    /dir/subdir/myscript.cgi

http.req.url.query IS  a1=b1&a2=b2&lang=en&other=something

http.req.url.path_and_query /dir/subdir/myscript.cgi?a1=b1&a2=b2&lang=en&other=something

And use host header, to get FQDN: 

http.req.header("host")  IS  demo.company.com

 

Syntax Example 2:

So, now when we evaluate elements of URLs, paths, and query parameters, sometimes you just want to see if an element contains or equals a certain phrase. Other times, we want to grab an element and parse this into our new output like redirect urls and rewrite statements.

Same URL, but let's look at QUERY parameters now:

https://demo.company.com/dir/subdir/myscript.cgi?a1=b1&a2=b2&lang=en&other=something

 

http.req.url.query IS  a1=b1&a2=b2&lang=en&other=something    # the query portion of the URL...

 

The query is made up of name=value pairs separated by ampersands, so the expression syntax can look at this as a string, or we can pull apart its elements:

 

http.req.url.query.name(<int>)   will treat the query parameter as an array of name/value pairs (zero-based) and allow you pull back the parameter names (left side of "=" signs) based on numerical position.

http.req.url.query.name(0):  is a1

http.req.url.query.name(2):  is lang

http.req.url.query.name(4):  is undefined as the highest index is 3 in this example (0-3 for 4 elements)

 

http.req.url.query.value(<int>)   will treat the query parameter as an array of name/value pairs (zero-based) and allow you pull back the parameter VALUES (right side of "=" signs) based on numerical position.

http.req.url.query.value(0):  is b1

http.req.url.query.value(2):  is en

http.req.url.query.value(4):  is undefined as the highest index is 3 in this example (0-3 for 4 elements)

 

Positions are annoying and we can also extract values, based on the parameter name, regardless of position. Expressions use ampersands and equals signs to parse the different name/value pairs, so we don't have to worry about them with this particular operators like you would if we just did a contains/eq operator.

http.req.url.query.value("<parameter name>")  will extract the value of that parameter.

http.req.url.query.value("a1")  IS   b1

http.req.url.query.value("a2")   IS  b2

http.req.url.query.value("lang")  IS  en

http.req.url.query.value("other")   IS something

 

Allowing you to pull part elements of the request to build your new outputs...

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • Like 1
Link to comment
Share on other sites

Thank you very much for the clear explanation and the extra information. I'll keep this in my documentation for future reference and will go to bed less dummy tonight :6_smile:

I know the value of time and appreciate that you took yours to help me in this.

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...