Jump to content
Welcome to our new Citrix community!

Mitigation Steps for CVE-2019-19781


Recommended Posts

I went through this article

https://support.citrix.com/article/CTX267679

 

We have one NS in our primary datacenter and one in our DR. We do not have a HA or Cluster. I am not sure whether it is a Cluster which is in the DR site. My question is

1. How would I find if it is in the cluster?

2. The below steps is shown for Standalone system. Assuming our is a standalone system, where do I run the first CLI command? It says run it on the CLI interface of the appliance. So if I login through Putty to apply which IP is this?

 

The following configuration changes serve as a mitigation to the aforementioned vulnerability.

Standalone System

Run the following commands from the command line interface of the appliance to create a responder action and policy: (should this be applied to SNIP ip address or? )???????
 enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config 


Ensure that the changes apply to the management interfaces as well. From the command line interface, please run the following commands. (This will be the Management Interface which is the NSIP)

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot

Link to comment
Share on other sites

1) To determine if you are standalone, HA pair, or Cluster participant:

show ha node or  in gui: system > HA > node

If only node 0 is present, you are not in an HA pair.  If node 0 and node 1 are shown, then you are participating in an HA pair.

 

If you are in a cluster, try show cluster or in GUI go tot system > cluster.

Or show ns ip and see if a CLIP is identified.

 

If neither HA pair or cluster, you are in a standalone configuration.

 

2) If you are in an HA pair, yo will run the responder policy cli command on the PRIMARY member of the pair (setting will then propagat to partner secondary netscaler).  In a CLUSTER you would connect to the CLIP which would run the command on the authoritative member of the cluster (aka the Cluster Coordinator).

 

For the napimgr command this is applied via shell and therefore has to be done per NetScaler as it will not propagat to the secondary member(s) of an ha pair or cluster and must be done per netscaler.  The export to the /nsconfig/rc.netscaler file ensures the command persists between reboots.

 

3) Regarding connecting to your CLI.

When you connect to any management enabled IP via putty (ssh) you engage with the CLI.  This means either use the NSIP of the current primary NetScaler or a management enabled SNIP (will only be active on the primary NS).  But for this command, I would go NSIP on each NetScaler you need to administer.

 

Once in the CLI to run the responder policy and then save ns config to retain settings.  This only needs to be on primary member of ha pair, ClIP of cluster, or each nsip of standalone netscalers.

Enter the shell command to drop to bsd shell to run the nsapimgr commands as listed. This needs to be done per netscaler.

 

  • Like 1
Link to comment
Share on other sites

I have an HA pair of VPX appliances and followed the instructions and after the reboot the appliances had an invalid license and lost all certificate information.  Anyone else experience that?

 

Dealing with virtual netscalers and licensing is such a pain in the ***.  There is zero reason that the appliances should delete all the certificates when the license is invalid.  Sure, not allow you to use them, but seriously, why completely delete all the certificates.

 

That makes the recovery so much more complicated then simply applying a new license and rebooting.

Link to comment
Share on other sites

On 12/27/2019 at 12:43 PM, Rhonda Rowland1709152125 said:

1) To determine if you are standalone, HA pair, or Cluster participant:

show ha node or  in gui: system > HA > node

If only node 0 is present, you are not in an HA pair.  If node 0 and node 1 are shown, then you are participating in an HA pair.

 

If you are in a cluster, try show cluster or in GUI go tot system > cluster.

Or show ns ip and see if a CLIP is identified.

 

If neither HA pair or cluster, you are in a standalone configuration.

 

2) If you are in an HA pair, yo will run the responder policy cli command on the PRIMARY member of the pair (setting will then propagat to partner secondary netscaler).  In a CLUSTER you would connect to the CLIP which would run the command on the authoritative member of the cluster (aka the Cluster Coordinator).

 

For the napimgr command this is applied via shell and therefore has to be done per NetScaler as it will not propagat to the secondary member(s) of an ha pair or cluster and must be done per netscaler.  The export to the /nsconfig/rc.netscaler file ensures the command persists between reboots.

 

3) Regarding connecting to your CLI.

When you connect to any management enabled IP via putty (ssh) you engage with the CLI.  This means either use the NSIP of the current primary NetScaler or a management enabled SNIP (will only be active on the primary NS).  But for this command, I would go NSIP on each NetScaler you need to administer.

 

Once in the CLI to run the responder policy and then save ns config to retain settings.  This only needs to be on primary member of ha pair, ClIP of cluster, or each nsip of standalone netscalers.

Enter the shell command to drop to bsd shell to run the nsapimgr commands as listed. This needs to be done per netscaler.

 

 

Thanks Rhonda, that was great info from you. Appreciate your answer

Link to comment
Share on other sites

Hello,

 

  Can anyone tell what the policy expression does exactly ? from my understanding :

 

HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") ==> if the url contains /vpns/

!CLIENT.SSLVPN.IS_SSLVPN ==> if the client is not using ssl vpn traffic ?

HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\") ==> if the url contains any text after the / until the next / ?

 

thanks for your help

 

Regards

 

 

Link to comment
Share on other sites

Just a clarifying question....this is basically disabling the Netscaler Gateway or "Portal" as our users call it? If so, how do external users who do not have a dedicated VPN access to our network, get their apps?

 

And does this need to be applied to all Netscalers or just DMZ netscalers hosting the Gateway?

Edited by jparks943
Forgot question
Link to comment
Share on other sites

On 1/2/2020 at 11:43 AM, Joseph Parks said:

Just a clarifying question....this is basically disabling the Netscaler Gateway or "Portal" as our users call it? If so, how do external users who do not have a dedicated VPN access to our network, get their apps?

 

And does this need to be applied to all Netscalers or just DMZ netscalers hosting the Gateway?

 Is there any update on this? What is the fix recommending? Is it restricting users to access the URL only via VPN? Our customers that access the app via citrix gateway are all external customers, and will not have VPN access. 

 

Link to comment
Share on other sites

It is not disabling the gateway portal page if the connection is recognized as an sslvpn connection.  If you are using the ICA Proxy connection (via StoreFront), there should be no disallowed reference to the /vpns/ path, so external access to citrix resources in ICA Proxy (non vpn mode) should not be affected.

The policy is blocking access to the /vpns/ directory if not a sslvpn connection or if an invalid reference to a directory browse up /../ is received.

 

The policy as implemented in the recommended article would prevent violations from any entry point vpn vserver (other) or management ips.

 

 

Link to comment
Share on other sites

On 12/30/2019 at 8:00 PM, Neil Heppenstall1709161579 said:

Can anyone confirm whether this affects NS9.3: Build 68.3.cl


I hope that's trolling and not a serious question. It was basically all versions of the netscaler/gateway still supported, they (Citrix) will not supply any notifications on such things happening on unsupported platforms. It would thus be safer to assume it is, as its got lots of bugs, security holes, performance issues etc that have since been patched.

read some  of the articles out there about the issue, this is a new discovery of a code issue that has been there since prior to all the newest supported versions so assume the worst.

personally I'd rather put my hand in a blender than support 9.3 again, and would seriously recommend upgrading. I don't even want to know how you keep that thing licensed

Regards

Link to comment
Share on other sites

On 12/28/2019 at 2:46 PM, Mark Hodges said:

I have an HA pair of VPX appliances and followed the instructions and after the reboot the appliances had an invalid license and lost all certificate information.  Anyone else experience that?

 

We've just had exactly that with one of our pairs in AWS. Looking back through the logs it was saying the "platform license" had expired and all config had basically been reverted.. :(

Link to comment
Share on other sites

On 1/10/2020 at 9:04 PM, Andrew Rogers said:

 

We've just had exactly that with one of our pairs in AWS. Looking back through the logs it was saying the "platform license" had expired and all config had basically been reverted.. :(


On device boot the licence file is checked, and its the only time, if you have not been keeping it up to date its a nasty surprise. Try using a licence server or make sure you keep applying updated lic files as required. its terrible to loose your features and bandwidth suddenly with no notice. Dropping to an 'express' licence can ****

On the other hand if you followed the mitigation steps and didn't blindly reboot you would have been OK until it rebooted next. the two steps prior to the reboot were to apply to the running config and the saved config. the reboot just lets you verify its applied, but should not be required Citrix always just put it there. 

NOTE:  even accelerating the licence procurement process can take a couple of days to get one. If you talk to Citrix nicely they might be ok with you reapplying a temporary licence to a production device if they know a purchase order has been sorted out. but then having more than 20 can give you some pull  a smaller operator might not have. but its always worth trying, they can be very reasonable about production systems.

  • Like 1
Link to comment
Share on other sites

Patching and mitigation will be probably not enough. You have to re-check and control all your appliances.
In our case, we found some compromised appliance, we decide to restore instances (or re-image instances) from 1st week of december before the CVE-2019-19781 publication, implement the mitigation proposed by Citrix, revoke/renew certificates + reset of all passwords involved with NetScaler + reset of all administrative accounts with priviledges.. Recontrol everything after remediation.

Read carefully these both articles for the verification steps and other recommandations

https://www.poppelgaard.com/cve-2019-19781-what-you-should-know-and-how-to-fix-your-citrix-adc-access-gateway

Read also this one, not so funny :

https://www.fireeye.com/blog/threat-research/2020/01/vigilante-deploying-mitigation-for-citrix-netscaler-vulnerability-while-maintaining-backdoor.html

 

Link to comment
Share on other sites

On 12/29/2019 at 1:16 AM, Mark Hodges said:

I have an HA pair of VPX appliances and followed the instructions and after the reboot the appliances had an invalid license and lost all certificate information.  Anyone else experience that?

 

Dealing with virtual netscalers and licensing is such a pain in the ***.  There is zero reason that the appliances should delete all the certificates when the license is invalid.  Sure, not allow you to use them, but seriously, why completely delete all the certificates.

 

That makes the recovery so much more complicated then simply applying a new license and rebooting.


The other possible issue is when you created the VPX's is you did not use a fixed MAC address. This is used for licencing so when the 'device ID' changes  this does to match.....

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...