Error applying mitigation steps for Log4j issue on ADC

[Sabine Ludewig1709156713]

Hi there

We don't have Web Application Firewall activated, but Web Logging. And I think that's why a scan found our system to be vulnerable, although Citrix marks ADC as not affected.

So, I'm trying to implement mitigation steps on ADC VPX 13.0 as described in https://support.citrix.com/article/CTX335705.

 

add policy patset patset_cve_2021_44228 

bind policy patset patset_cve_2021_44228 ldap 

bind policy patset patset_cve_2021_44228 http 

bind policy patset patset_cve_2021_44228 https 

bind policy patset patset_cve_2021_44228 ldaps 

bind policy patset patset_cve_2021_44228 rmi 

bind policy patset patset_cve_2021_44228 dns 

add responder policy mitigate_exploit_cve_2021_44228 q^HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE. SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228")^ DROP 

bind responder global mitigate_exploit_cve_2021_44228 100
 

But when running the 'add responder policy'  command, it returns with ERROR: Action does not exist
We don't have Web Application Firewall activated, is that the reason why? Or is there anything wrong with the command published by Citrix?

 

Thanks a lot

[Rhonda Rowland1709152125]

It works fine on my instance.  Be sure your source doesn't have any smartquotes or line breaks in it.  However, I found one space in the original output that might trip up on a version if its being picky and removed it below. My version took it, but different versions might see it as an expression break.  Which version of the firmware are you on?

1 hour ago, Sabine Ludewig1709156713 said:

add responder policy mitigate_exploit_cve_2021_44228 q^HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE. SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228")^ DROP

There's a space between this one text_mode. and the set_text_mode in purple above.  That *might* be where it interpreted it wrong.  But my system took it no problem.

 

Here's a copy from a running config after import which converts a few characters to quotes but is still valid (without the space above). NOTE: This is a single line command (requiring all patternsets referenced to be created first).

 

add responder policy mitigate_exploit_cve_2021_44228 "HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR(\"${\").BEFORE_STR(\"}\").CONTAINS(\"${\") || HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS(\"${: }/+\").AFTER_STR(\"jndi\").CONTAINS_ANY(\"patset_cve_2021_44228\") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR(\"${\").BEFORE_STR(\"}\").CONTAINS(\"${\") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS(\"${: }/+\").AFTER_STR(\"jndi\").CONTAINS_ANY(\"patset_cve_2021_44228\")" DROP

[Sabine Ludewig1709156713]

14 hours ago, Rhonda Rowland1709152125 said:

It works fine on my instance.  Be sure your source doesn't have any smartquotes or line breaks in it.  However, I found one space in the original output that might trip up on a version if its being picky and removed it below. My version took it, but different versions might see it as an expression break.  Which version of the firmware are you on?

There's a space between this one text_mode. and the set_text_mode in purple above.  That *might* be where it interpreted it wrong.  But my system took it no problem.

 

Here's a copy from a running config after import which converts a few characters to quotes but is still valid (without the space above). NOTE: This is a single line command (requiring all patternsets referenced to be created first).

 

add responder policy mitigate_exploit_cve_2021_44228 "HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR(\"${\").BEFORE_STR(\"}\").CONTAINS(\"${\") || HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS(\"${: }/+\").AFTER_STR(\"jndi\").CONTAINS_ANY(\"patset_cve_2021_44228\") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR(\"${\").BEFORE_STR(\"}\").CONTAINS(\"${\") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS(\"${: }/+\").AFTER_STR(\"jndi\").CONTAINS_ANY(\"patset_cve_2021_44228\")" DROP

 

Thanks for you reply.

I fixed the spaces, but no luck. Sure my code doesn't have any line breaks, but what do you mean by smartquotes? I only use ", as copied from Citrix's website.

I'm wondering if I need a responder action, too, which isn't mentioned in the Citrix code. Sorry I'm not very familiar with these sort of things.

I'm on ADC 13.0.82.45 VPX

[Oliver Schuhmacher1709158889]

I have the same behavior whe i want to create a Responder-Policy on my VPX. The message is "Action does not exist". Is the same error if i create the Responder-Policy in command line or in the GUI.
Mabey ist that a problem only on a VPX. I found that article from Citrix "Rewrite 'ACTION DROP' is not working as expexted on NetScaler VPX" (https://support.citrix.com/article/CTX204349).

[Sabine Ludewig1709156713]

17 minutes ago, Oliver Schuhmacher1709158889 said:

I have the same behavior whe i want to create a Responder-Policy on my VPX. The message is "Action does not exist". Is the same error if i create the Responder-Policy in command line or in the GUI.
Mabey ist that a problem only on a VPX. I found that article from Citrix "Rewrite 'ACTION DROP' is not working as expexted on NetScaler VPX" (https://support.citrix.com/article/CTX204349).

It says that the behavior is different on VPX,  it should be possible to add the responder policy, though.

I have a case open with Citrix, but response yet

[Oliver Schuhmacher1709158889]

11 minutes ago, Sabine Ludewig1709156713 said:

It says that the behavior is different on VPX,  it should be possible to add the responder policy, though.

I have a case open with Citrix, but response yet

OK, I'm curious to hear what citrix has to say about this.

[Rhonda Rowland1709152125]

I ran it on a 13.0.58.x build, because it was what was in front of me (also vpx) and it had no issue. So it sounds version specific. 

You could try it in the GUI and see if the GUI takes it and then compare the cli in case it is a firmware specific thing.  The expression is written as q^<stuff>^  the q is quoting the start and end of the expression using a character not in the expression (^, carat). So in the GUI, you can omit the q^ and ^ and just use the <stuff> in between. 

[Sabine Ludewig1709156713]

2 hours ago, Rhonda Rowland1709152125 said:

I ran it on a 13.0.58.x build, because it was what was in front of me (also vpx) and it had no issue. So it sounds version specific. 

You could try it in the GUI and see if the GUI takes it and then compare the cli in case it is a firmware specific thing.  The expression is written as q^<stuff>^  the q is quoting the start and end of the expression using a character not in the expression (^, carat). So in the GUI, you can omit the q^ and ^ and just use the <stuff> in between. 

Thanks for the advice, but even in the GUI I receive "Action Does Not Exist", so I think it's a version thing

And it doesn't matter wether I change the Undefined result action to "Drop" or keep the default "Global Undefined result Action"

[Rhonda Rowland1709152125]

Definitely a build bug and needs to be reported to support.  

You could use a RESET if you had to.

If this content is WEB based, you can possibly use an appfw policy to run the appfw_DROP action/profile for this expression in place of responder as a work around.   This would be one of the built in appfw profiles (pre-defined drop) and doesn't do the rest of the appfw profile inspection.  But would still require appfw feature to be licensed/enabled.

[Oliver Schuhmacher1709158889]

22 hours ago, Rhonda Rowland1709152125 said:

Definitely a build bug and needs to be reported to support.  

You could use a RESET if you had to.

If this content is WEB based, you can possibly use an appfw policy to run the appfw_DROP action/profile for this expression in place of responder as a work around.   This would be one of the built in appfw profiles (pre-defined drop) and doesn't do the rest of the appfw profile inspection.  But would still require appfw feature to be licensed/enabled.

I updatet my VPX to Version 13.1 12.50 in order to avoid an build bug. But unfortunatly if i create a Responder-Policy with the action DROP or RESET i get still the message "Action does not exist". If i choose NOOP as action so i able to create the Responer Policy without any problems. This is really crazy.

[Toon Lauwaert]

We have the same problem, Action does not exist. Running  13.1 12.50 on a VPX. 

So what is the shell commands I have to run to have ths active?

[Rhonda Rowland1709152125]

The commands above in CLI in the original post should work from the cli.  I don't know why you are get an action does not exist message.

[Shahzad Siddique 2]

Hi guys,

   after applying Log4j mitigation steps using responder policy, we were observed false positive behavior were legitimate traffic is getting blocked. Is there any way to get a Logs for binded responder policy getting hits.

 

trying using policy based logging, https://support.citrix.com/article/CTX125466. But see no log in tail -f /log/ns.log.

 

please help in achieving the same, thanks in advance.

[Sabine Ludewig1709156713]

Eventually received an answer from Citrix support

It seems if you don't have DoS Protection and AppQoE licensed, the DROP action is missing and cannot be created.

So Citrix provided us with a differend approach using a 'Forbidden' responder action:
 

add policy patset patset_cve_2021_44228 

bind policy patset patset_cve_2021_44228 ldap 

bind policy patset patset_cve_2021_44228 http 

bind policy patset patset_cve_2021_44228 https 

bind policy patset patset_cve_2021_44228 ldaps 

bind policy patset patset_cve_2021_44228 rmi 

bind policy patset patset_cve_2021_44228 dns 

 

add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" 

 

add responder policy mitigate_exploit_cve_2021_44228 q^HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.AFTER_STR("${").BEFORE_STR("}").CONTAINS("${") || HTTP.REQ.BODY(8192).SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.SET_TEXT_MODE(IGNORECASE).STRIP_CHARS("${: }/+").AFTER_STR("jndi").CONTAINS_ANY("patset_cve_2021_44228")^ respondwith403 

 

bind responder global mitigate_exploit_cve_2021_44228 100
 

[Rhonda Rowland1709152125]

On 12/29/2021 at 12:10 AM, Shahzad Siddique 2 said:

trying using policy based logging, https://support.citrix.com/article/CTX125466. But see no log in tail -f /log/ns.log.

Shahzad - when you enabled policy based logging, did you 1) assign a log action to your responder policy AND enable logging of "user configurable messages" in the syslog global audit parameters (for local logging). The log policy audit action will then be generated when the responder policy is true, but its only captured in syslog if the user configurable messages is also included in the syslog logging.  Global syslog parameters are under System > Auditing:: right-hand pane for Syslog parameters.  If you are looking at a remote log, verify the actual syslog policy action for your alternate destination has this.  Also, be sure your log alert levels includes the log level you set in your audit action associated with your responder policy.

(sorry for much delayed response)

[Ricardo Jaime1709158738]

11 minutes ago, Rhonda Rowland1709152125 said:

Shahzad - when you enabled policy based logging, did you 1) assign a log action to your responder policy AND enable logging of "user configurable messages" in the syslog global audit parameters (for local logging). The log policy audit action will then be generated when the responder policy is true, but its only captured in syslog if the user configurable messages is also included in the syslog logging.  Global syslog parameters are under System > Auditing:: right-hand pane for Syslog parameters.  If you are looking at a remote log, verify the actual syslog policy action for your alternate destination has this.  Also, be sure your log alert levels includes the log level you set in your audit action associated with your responder policy.

(sorry for much delayed response)

Is there a way to log the whole HTTP Request? (headers, body, etc) I have configured an audit messageaction with HTTP.REQ.FULL_HEADER and HTTP.REQ.BODY(8192) but I'm still not seeing why the policy is blocking certain traffic, I can't identify any of the patterns on the log data, so I would assume I'm missing a part of the HTTP request

[Rhonda Rowland1709152125]

Go back to your policy expression and try to do things like decoding the output:  hTTP.REQ.FULL_HEADER.SET_TEXT_MODE(URLENCODED).DECODE_USING_TEXT_MODE.

 

OR write separate policies for the individual OR clauses with separate logging actions so that you can see which expression is being triggered on.  Such as one for the header checks and a second policy for the body checks.

Compare your policy expression and/or patternsets to see if there is an error in what's on your ADC compared to what you expected it to look for.  

Try to simplify the policy and logging action until you get the result you needed.