Jump to content
Welcome to our new Citrix community!

Auto Update whitelist on a responder using an Api json url


Recommended Posts

I need to have a whitelist that will updated in daily base from this https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json api. So then i will use responder to allow the traffic to a Lb only for this whitelist. I think to add Data set but i dont know how to auto update the white list from the above oracle api url. Any help?

 

thank you.

Link to comment
Share on other sites

Auto-updates would be much easier to do via a HTTP Callout. You could then use whatever infrastructure you want to update the callout whitelist file/database whatever and the adc config just queries the callout.    You should also use callouts over pattern sets/data sets if the lists get too long to evaluate effectively.

 

So, are you looking for the Nitro API to update the responder policy?  

Will the values you're updating be individual ips, cidr notation subnets, or other?

Data Sets will still probably be easier than direct expression management, but also how big a list are you going for?

 

 

Link to comment
Share on other sites

14 hours ago, Rhonda Rowland1709152125 said:

Auto-updates would be much easier to do via a HTTP Callout. You could then use whatever infrastructure you want to update the callout whitelist file/database whatever and the adc config just queries the callout.    You should also use callouts over pattern sets/data sets if the lists get too long to evaluate effectively.

 

So, are you looking for the Nitro API to update the responder policy?  

Will the values you're updating be individual ips, cidr notation subnets, or other?

Data Sets will still probably be easier than direct expression management, but also how big a list are you going for?

 

 

thank you for your answer. Yes i am lookin http callout also, but i dont know how to configure it. I read that it Need a server with http callout agent how to do this on a public server .

If you have directions please give them to me .

 

the point is that We must have the updated list of ips from this url https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json every day.

 

and especially for "region": "eu-frankfurt-1".

 

I know also about data sets. I am trying to figure out how to add all those url on data set automatically , how to add it from a file.

 

thank you,

Link to comment
Share on other sites

I'll try to answer end of day - if no one else gets back to you.

But use an outside script to get data into the callout server.  And that's the only external thing you have to worry about.

Then the callout is the easiest way to do this as once the callout is built you won't need to update the adc config.  

 

So, lets' tackle smaller problems first. Such, as how a callout works and then maybe a patternset/dataset example.  

Then you can figure out how to get the data from the URL into a format you can use.  

 

The *tricky* part is still that last piece, get data from <SOURCE> (aka the URL) into a <THING YOU CAN USE>.  I'm starting with the <THING YOU CAN USE> examples first and then we'll figure out how to solve the <SOURCE> piece.

 

Also, I looked at your source url, trying to do cidr subnet ranges in patternsets is TRICKY. So while I will construct a basic patternset example so you can see how they work, I'll link to a previous forum where we showed this hard.  Your BEST BET really is to define a mechanism that gets the <source> list into the CALLOU and then the ADC jsut has to invoke the callout. Then all the hard parts of getting data into a file/database or other to process is a callout problem and NOT an ADC problem.  Trying to take the raw data directly to an adc expression or patternset would be fussy.

 

Example 1:  Expression Basics:  Responder Policies (brief)

A RESPONDER policy can be used to REDIRECT/DROP/RESET traffic of interest to prevent access to the resource.  Usually, your looking for what you DON'T WANT (blacklist) to take action. So when, using a WHITELIST, your responder is based on traffic NOT (Whitelist).  We just have several ways to build this filter.

 

Suitable whitelist expressions so RESPONDER DROPS traffic NOT on Whitelist:

[1a]  !client.ip.src.eq(10.10.10.1)          Result:  Allows 10.10.10.1; blocks all others.

[1b] For compound expressions remember !(A||B) is equivalent logically to (!A && !B); !(A&&B) is equivalent to (!A || !B)

So, if you had a list of allowed ips:  (10.10.10.1, 10.10.20.2, 10.10.30.3) and you want to allow (not hit on these); but block everything else:

Your !<whitelist> expression for responder would be:  !( client.ip.src.eq(10.10.10.1) || client.ip.src.eq(10.10.20.2) || client.ip.src.eq(10.10.30.3) )

You need the responder to ONLY evaluate to TRUE when IP is !<WHITELIST>.  So, if IP is not in whitelist, <whitelist> is false and !False is true.

So, if you come from any whitelisted IP the <whitelist> is TRUE; threfore !<whitelist> is FALSE and responder will not hit. 

 

So, going forward, whether your responder expressions is based on a single IP, or a hardcoded IP your expression is !( <whitelist> )  and uses OR clauses with a whitelist determination it will be one of these types of formats:

!( 10.10.10.1 )   --> anything other 10.10.10.1 is blocked; ip is allowed.

!( 10.10.10.1 || 10.10.20.2 || 10.10.30.3 ) --> 10.10.10.1, 20.2, 30.3 is allowed; all other source ips blocked.

!( client.ip.src.typecast_text_t.contains_any(<ps_whitelist>) )

!( sys.http_callout("hc_whitelist")

 

 

Example 2:  Pattern Sets/Data Sets.

A pattern set (and data sets) are index tables.  So once you know what values you want you can use the _any() operators: equals_any, contains_any, startswith_any, endswith_any, etc....

So, they allow you to build a pattern set of values and the _any operators are OR clauses across the contents.  

Pattern Sets work for strings; data sets work for numerical data types (like IPS etc...) BUT you have to convert your numerical object to a string to use a equals_any, or contains_any operator to do a comparison.

 

If this was just IPs, then a patternset of the three values above 10.10.10.1, 10.10.20.2, 10.10.30.3 in a ps_whitelist could be evaluated like:

client.ip.src.typecast_text_t.contains_any("ps_whitelist")

 

If you had a list of subnets (masks applied to range), then a list of subnets as 10.10.10.0, 10.10.20.0, 10.20.30.0 could be compared if an IP is in the whitelist subnet range as:

client.ip.src.subnet(24).typecast_text_t.contains_any("ps_whitelist")

 

While we can write a client.ip.src.in_subnet("10.30.20.0/24") expression. This is one thing you can't easily do with pattern set of cidr notations in the format of: 10.10.10.0/24, 10.10.20.0/20, 10.10.30.0/22.  The In_subnet("") operator doesn't have an _any variant to use a list of cidr subnets.  

 

Since your source list is a mixture of <network>/<subnet masks>, you would either need to convert them to a list of all the /24 networks in one whitelist, and all the /20 in another to get the pattern sets to work. (And you still need to populate/build all the patternsets.)

Here's an example thread, where I show several examples for someone else (It also shows actual syntax for constructing patternsets):

https://discussions.citrix.com/topic/409145-netscaler-policy-assistance/

 

 

If this was a list you built once and were done, then a couple pattern sets/data sets might work. But I don't think it would be easy to do even if using an API to generate the data was easy, which is why the callout is still the most flexible.

 

NEXT:  I'm going to post the Callouts in a separate post.

 

 

 

 

 

 

 

 

 

Link to comment
Share on other sites

Example 3:  HTTP Callouts:

So, a HTTP Callout is basically a utility you run an external entity (web server), that you can write a script/agent to do whatever you want against a database/file or whatever (or even the URL if needed). You then make the Callout able to receive requests form the ADC has a HTTP REQUEST and generate an HTTP RESPONSE.

So for the ADC, the HTTP Callout acts a web-based remote function call, where you use the ADC to invoke a function (web page/script) and pass data inputs via the request url, query parameters, and headers and get function return result back as a HTTP RESPONSE either in body content or response headers.

 

On the ADC, you build the HTTP Callout invocation by structuring the HTTP REQUEST to call the page and "send" data to the remote server and identify the function output based on how to parse the HTTP RESPONSE.  

 

The external sever runs the agent which is the web page ot interact with. And it can be used to evaluate is the IP you send it on the WHITELST (or not) and return a result to ADC.  

If !<whitelist> adc can DROP traffic; else no action.

 

This means you just need a separate regular web script to extract the URl and populate a database or file for the callout to use. And the ADC and the Callout agent, don't need to change. You shift getting the ip list and processing the values to the callout server and whatever scripts you want to write to do it.

 

So the ADC has a sample blacklist example and a script to evaluate if the client.ip.src is in the blacklist and how to build the agent to take a query parameter and execute against he database. You still need to write the piece that gets the source and puts in a format to evaluate.   But an external script/application could deal with fetching the data, and doing the cidr comparisons as is, or converting them to a format - much easier than trying to get the ADC to do all that.

 

https://docs.citrix.com/en-us/citrix-adc/current-release/appexpert/http-callout/use-case-filtering-clients-ip-blacklist.html

This example says you have an database of <blacklist ips> on this mysql server in the backend. (Getting data into database is the thing you still have to solve, but first things first).

So the sample web page runs on the remote web server (not the ADC) and you can run a request against it like so:

 

Example 3a:

Note this is example is a <blacklist> example, but whitelist is just the IPs you want to allow based on the previous post logic.

 

http://demo1.domain.com/cgi-bin/checkip.cgi?cip=<ip1>

If you were to set up this script as the page name /checkip.cgi on a webserver (demo1) with perl enabled for cgi-bin (etc), then it takes an IP in the CIP parameter, compares against its local MYSQL database and returns result IP Matched (blacklist) or IP Failed (not on blacklist)

 

Build callout hc_blacklist:

You then build the Callout on the ADC to make the HTTP Rquest to the page /checkip.cgi, by passing parameter CIP with client.ip.src.

You then define the return response of the callout value as http.res.body(10000).contains("IP Matched") and return type as Boolen.

I would start with the default database query and a hard coded list of test ips, so you can just see the callout work.

 

So, your responder expression to invoke callout is now:

sys.http_callout("hc_blacklist")

For a blacklist example, IF callout returns TRUE (IP is on blacklist) --> then trigger policy.

 

If you made the list the whitelist of ALLOWED IPS, and again ahve the callout return TRUE (on whitelist); FALSE (not on whitelist), then your callout could be:

!sys.http_callout("hc_whitelist")

Result: if IP is not on whitelist, callout returns FALSE, threefore !whitelist is TRUE and responder DROPs traffic;

If IP is on whitelist, callout returns TRUE, so !whitelist is FALSE and responder drops traffic.

 

Example 3b:  

Next update the callout database to contain the type of data your getting from the remote url, like IP/cidr notations. And decided if you want the remote logic to try to do an in subnet x.x.x.x/yy comparison. Or when you seed the data in the database it applies mask and converts all cidr to x.x.x.0 style subnets.

 

Then figure out how to update the callout script to compre if client ip a.a.a.a is in the subent/mask  x.x.x.x/yy or in subent x.x.x.0 style comparisons.   How you do this up to you whether you want to use perl/ruby/python or any scripting language. ADC will give the IP, the remote agent compares against list of available values and says YES or NO.

 

Example 3c:

Once you know what format you want your data in, then you can solve the how do I get the data from the source URL into the database or file for the callout to evaluate.

 

But by using a callout you can use any service/api/scripting engine to solve this problem. The ADC only cares about if a call checkclient.cgi with a parameter, do I get a result back.

 

Summary:

Example Callout agent (the script on the remote server the ADC calls to spit out a result). This particular script is based on perl, takes a source ip in the cip query paramter and spits out a string in the http response. 

https://docs.citrix.com/en-us/citrix-adc/current-release/appexpert/http-callout/use-case-filtering-clients-ip-blacklist.html

https://docs.citrix.com/en-us/citrix-adc/current-release/appexpert/http-callout/configuring-http-callouts.html

 

You will still have to figure out how to get the data from <SOURCE URL> to the callout agent and in a format you can compare. But you have more script/tools/options on a remote server than trying to do this directly to the ADC.  You also have to figure how to right the agent to do the comparison giving the data format  the source engine gives you.

 

But the callout on the ADC to call this script and process the result is fairly straightforward.  Once you get the agent script built and web-based, just interact with it in a browser and look at the URL path and query in the request, the headers if needed, and response body/headers output. That's what you use to build the callout function on the ADC.

 

 

Finally:

Be sure your callout destination is load balanced by the ADC or you have a vserver and backup vserver. If the callout is down the callout expression is automatically down. So if no callout, do you want your whitelist to fail so no data is passed. For large volumes of traffic you also don't want the callout to bottleneck.

 

Link to comment
Share on other sites

On 3/14/2022 at 7:02 PM, Rhonda Rowland1709152125 said:

Auto-updates would be much easier to do via a HTTP Callout. You could then use whatever infrastructure you want to update the callout whitelist file/database whatever and the adc config just queries the callout.    You should also use callouts over pattern sets/data sets if the lists get too long to evaluate effectively.

 

So, are you looking for the Nitro API to update the responder policy?  

Will the values you're updating be individual ips, cidr notation subnets, or other?

Data Sets will still probably be easier than direct expression management, but also how big a list are you going for?

 

 

thank you for your answer.  I DONT KNOW WHAT TO SAY! you did a very good analysis . I'm still drying to figure it out. I think a trick to take a part the responder to 3-4 responders so with Nitro api to update the responders. I dont know if this will work. Other thing i am considering is the use of Waf. The callout is easy as you said on ADC part but i need effort from other people (programmers) to do the other  parts... i dont know yet how we will proceed. 

Link to comment
Share on other sites

The reason I prefer the CALLOUT to do this is because to get the ADC to 1) get the file into a usable form and 2) compare a source IP given the network/cidr notation used with the policy syntax alone on the ADC would be hard.

 

Nitro API isn't easier and constantly changing policies is still challenging. To be honest, I don't think there is a good way to process the list you get from the website because its in cidr notation with just the policy expressions on the ADC.  If it was all one subnet mask, we could probably do it but that long of a file would be inefficient to do a contains any, but with the different masks, there's just not an easy way to do this.  

 

IF it was a finite/fixed list you could hardcode some stuff.

 

Here's some coding examples for the types of things you need, but if not your skill set, you would need some help to get the callout servers "backend" to do the work for you.  

===

The "CALLOUT Agent" (web page on web server) compares the IP from the ADC against the whitelist.

The "CALLOUT" on the ADC is the expression you need to invoke the call to the "CALLOUT Agent" and should return the yes/no (whitelist/blacklist response, depending on how you construct).

 

So, separately, once you see how the Callout can work for a simple IP or network comparison.  Then you need to figure out how to manage the "WHITELIST CONTENTS"  from the <SOURCE file> you are using.   By getting this part off the ADC and into a LAMP server you have a wider range of scripting tools to use.  Without more info, hard to narrow down specific recommendations, but a few things that may help:

 

Here's a PERL example of taking a JSON file and converting/inserting it into a database:  https://codeburst.io/convert-raw-json-data-into-an-sql-database-using-sqlite-and-perl-489c921b8868.  

None of this is ADC-based; its all on the lamp server.

1. Construct a job to fetch the source file (once a day or whatever interval you need)

2. Use a perl script like the above example to convert to a database and right a job to drop old and create new; or once you get the values scan and append in database.  Decide if you want to use the "x.x.x.x/yy" notation and the comparison command will have to compare a test ip "cip" against the cidr as an in network query. Or once you get the json data you might be able to make the database as both network and cidr values.  Or convert cidr masks to regular netmasks, so you can use a perl net::subnet query.

3. Then  A perl command to compare an ip to cidr range, might be like this:  https://perlmaven.com/is-this-ip-in-the-given-subnet or https://metacpan.org/pod/Net::Subnet (this one uses CIDR)

 

4. Update CALLOUT_AGENT script with commands needed to compare IP against the whitelists extracted from the source file given the format you are now using.  

 

The ADC Callout would only need to be tweaked if you change the way you call the agent script, pass in the client ip, or return the response.

 

Link to comment
Share on other sites

18 hours ago, Rhonda Rowland1709152125 said:

The reason I prefer the CALLOUT to do this is because to get the ADC to 1) get the file into a usable form and 2) compare a source IP given the network/cidr notation used with the policy syntax alone on the ADC would be hard.

 

Nitro API isn't easier and constantly changing policies is still challenging. To be honest, I don't think there is a good way to process the list you get from the website because its in cidr notation with just the policy expressions on the ADC.  If it was all one subnet mask, we could probably do it but that long of a file would be inefficient to do a contains any, but with the different masks, there's just not an easy way to do this.  

 

IF it was a finite/fixed list you could hardcode some stuff.

 

Here's some coding examples for the types of things you need, but if not your skill set, you would need some help to get the callout servers "backend" to do the work for you.  

===

The "CALLOUT Agent" (web page on web server) compares the IP from the ADC against the whitelist.

The "CALLOUT" on the ADC is the expression you need to invoke the call to the "CALLOUT Agent" and should return the yes/no (whitelist/blacklist response, depending on how you construct).

 

So, separately, once you see how the Callout can work for a simple IP or network comparison.  Then you need to figure out how to manage the "WHITELIST CONTENTS"  from the <SOURCE file> you are using.   By getting this part off the ADC and into a LAMP server you have a wider range of scripting tools to use.  Without more info, hard to narrow down specific recommendations, but a few things that may help:

 

Here's a PERL example of taking a JSON file and converting/inserting it into a database:  https://codeburst.io/convert-raw-json-data-into-an-sql-database-using-sqlite-and-perl-489c921b8868.  

None of this is ADC-based; its all on the lamp server.

1. Construct a job to fetch the source file (once a day or whatever interval you need)

2. Use a perl script like the above example to convert to a database and right a job to drop old and create new; or once you get the values scan and append in database.  Decide if you want to use the "x.x.x.x/yy" notation and the comparison command will have to compare a test ip "cip" against the cidr as an in network query. Or once you get the json data you might be able to make the database as both network and cidr values.  Or convert cidr masks to regular netmasks, so you can use a perl net::subnet query.

3. Then  A perl command to compare an ip to cidr range, might be like this:  https://perlmaven.com/is-this-ip-in-the-given-subnet or https://metacpan.org/pod/Net::Subnet (this one uses CIDR)

 

4. Update CALLOUT_AGENT script with commands needed to compare IP against the whitelists extracted from the source file given the format you are now using.  

 

The ADC Callout would only need to be tweaked if you change the way you call the agent script, pass in the client ip, or return the response.

 

thank you for you effort in this. I think is a good scenario to play with. For now i did a conversion of json to csv manually and imported to locations. Then with responder check this location csv file and drop or allow the connection. The tricky part is to convert this json to csv and with the format needed from NS automatically but this is not my responsibility. This will be done from programmer and then with the Nitro api he will update the csv . I will try also the above scenario you mention with callout.

 

thank you again for your help. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...