Jump to content
Welcome to our new Citrix community!

How to configure LB to perform mutual TLS? Only a specific client cert presenter is allowed access, not any cert signed by trusted CA's.


Greg Hodgins

Recommended Posts

My own searches for Netscaler mutual TLS or client certificate authentication, as well as many forum posts lead to this article. https://support.citrix.com/article/CTX205823

 

However, our use case (and I suspect others - especially given some questioning where they install the client cert on the NS) is to restrict access to an https API to the presenter of a specific certificate.  I would argue the typical use case for client certificate "authentication" is not to just determine the client is presenting a certificate that can be proven valid or "authentic", but restrict access to all but a specific client certificate holder - which is still proven valid/authenticate through PKI.

 

This has recently caused considerable confusion and issues with a given implementation where our security team wishes the added layer of client certificate authentication on a externally exposed VIP in our DMZ - on top of FW IP restrictions.

 

1. Is there an article that describes what I think would be termed mutual TLS?  The require is , only a specific certificate presenter would be allowed access to a load balancer virtual server?  The way I understand the above article, anyone who presents a client cert signed by the installed trusted CA's would gain access.  And a well known public CA like the one our partner used would be used by many thousands of users - and could be used by anyone.

 

Also for further clarification.... the same article above references this article in relation to device certificates vs. client certificates. https://docs.citrix.com/en-us/netscaler-gateway/12/install/certificate-management/using-device-certificates.html

 

Our use case is a partner application making https POSTs to an API.  There is no user interaction as is alluded to in several Citrix articles I've read in searching this subject.

 

2. In the above scenario we are still talking a client certificate vs. a device certificate, correct?

 

3. Several of the articles also describe the above scenarios specifically for protecting web servers and user access.  Perhaps being too critical I look at a web server as something a user interacts with, much like the articles imply.  However, once again, we're talking an application exposing an API via https.  Assuming this is still covered by the articles describing the protection of web servers, perhaps they should be updated so as to be more precise and subject to less doubt as to whether they are applicable to a http applications.  Of course, if if an https API is configured differently that a web server, then am interested in the articles specific to https applications.

 

Thanks much.

 

Greg

Link to comment
Share on other sites

Hi Siddharthas.  Did you read the requirement?  What you describe is what has been done.  It does not meet the requirement. 

 

Another more correct or better description of the requirement might be that we want to use the client certificate, or some attribute(s) of the validated certificate to "authorize" access to the VIP.

 

Thanks.

Link to comment
Share on other sites

Here are some expressions that you can use in a responder policy to parse the client cert.

https://docs.citrix.com/en-us/netscaler/12/appexpert/policies-and-expressions/ns-pi-ae-parse-ssl-certs-wrapper-con.html

 

AAA can also be leveraged to do auth based on cert i.e. pull the subject name from cert and validate using a LDAP.

 

PS: If you can confirm what exactly you are looking for that would enable better responses 

Link to comment
Share on other sites

On 7/30/2019 at 10:58 PM, Siddhartha Sarmah said:

Here are some expressions that you can use in a responder policy to parse the client cert.

https://docs.citrix.com/en-us/netscaler/12/appexpert/policies-and-expressions/ns-pi-ae-parse-ssl-certs-wrapper-con.html

 

AAA can also be leveraged to do auth based on cert i.e. pull the subject name from cert and validate using a LDAP.

 

PS: If you can confirm what exactly you are looking for that would enable better responses 

 

I'm not sure how else to articulate it.  We only want the presenter of a validated (by the installed trusted CA) "single specific certificate" to have access to the LB.  If not inspecting something in the validated cert, I would think installing the client cert on the Netscaler for the purposes of matching would have been possible or even typical.

 

We have it implemented now using a virtual AAA server and an authentication policy with an expression match for the HEXADECIMAL value of the serial number lie you show above.

 

Now that issue is that when we enable said policy there are serious issues.  With just client authentication enabled (and valid cert from the CA is good) we see good response times on the client side (60 ms) and the keep-alive and connection re-use looks good/healthy/effective.  As soon as we enable said serial number check policy it works, BUT response times go to 250-300 ms, the connection re-use is broken (see curl excerpt below), our throughput is restricted well below requirements AND there are memory errors on the box.

 

We have a case open with Citrix currently looking at the memory issues.  Whether the memory errors or below dead connection issues are a cause or effect of something else is TBD.  If can provide you the case if you like.

 

 

Example curl with multiple requests that normally respects connection re-use, but breaks with AAA auth policy enabled.

* Connection #0 to host hostname.com left intact

* Re-using existing connection! (#0) with host hostname.com

* Connected to hostname.com (xxx.xxx.xxx) port 443 (#0)

> HEAD / HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: hostname.com

> Accept: */*

* Empty reply from server

* Connection #0 to host hostname.com left intact

curl: (52) Empty reply from server

* Connection #0 seems to be dead!

* Closing connection #0

* About to connect() to hostname.com port 443 (#0)

*   Trying xxx.xxx.xxx.xxx... connected

* Connected to hostname.com (xxx.xxx.xxx.xxx) port 443 (#0)

Link to comment
Share on other sites

  • 2 weeks later...
On 8/4/2019 at 10:48 AM, Greg Hodgins said:

 

I'm not sure how else to articulate it.  We only want the presenter of a validated (by the installed trusted CA) "single specific certificate" to have access to the LB.  If not inspecting something in the validated cert, I would think installing the client cert on the Netscaler for the purposes of matching would have been possible or even typical.

 

We have it implemented now using a virtual AAA server and an authentication policy with an expression match for the HEXADECIMAL value of the serial number lie you show above.

 

Now that issue is that when we enable said policy there are serious issues.  With just client authentication enabled (and valid cert from the CA is good) we see good response times on the client side (60 ms) and the keep-alive and connection re-use looks good/healthy/effective.  As soon as we enable said serial number check policy it works, BUT response times go to 250-300 ms, the connection re-use is broken (see curl excerpt below), our throughput is restricted well below requirements AND there are memory errors on the box.

 

We have a case open with Citrix currently looking at the memory issues.  Whether the memory errors or below dead connection issues are a cause or effect of something else is TBD.  If can provide you the case if you like.

 

 

Example curl with multiple requests that normally respects connection re-use, but breaks with AAA auth policy enabled.

* Connection #0 to host hostname.com left intact

* Re-using existing connection! (#0) with host hostname.com

* Connected to hostname.com (xxx.xxx.xxx) port 443 (#0)

> HEAD / HTTP/1.1

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: hostname.com

> Accept: */*

* Empty reply from server

* Connection #0 to host hostname.com left intact

curl: (52) Empty reply from server

* Connection #0 seems to be dead!

* Closing connection #0

* About to connect() to hostname.com port 443 (#0)

*   Trying xxx.xxx.xxx.xxx... connected

* Connected to hostname.com (xxx.xxx.xxx.xxx) port 443 (#0)

 

We got to the bottom of our issue.  There is an obscure global setting called drop_aaatm_nocookie_conn that controls how TCP connections are handled when a cookie is not present in the request. 

 

According to Citrix default value is 1. "That implies close client side tcp connection if there is no cookie on previously authentication tcp connection.
0 results in 403 and a tcp connection close, not a silent drop".  Both of these resulted in unhealthy connection states for which I don't have a precise understanding - and am not pursuing.  However, it was not just a performance penalty of a full TLS handshake on each request - things got really messed up with client connections sitting in syn_sent state.

 

Modifying the setting to 2 allows the authenticated connection to continue - "connection re-use", and all is healthy.  We are going to test by sending a dummy cookie too, but so far testing has shown setting the value to 2 alleviates the problem.

 

The client (a vendor partner application), simply uses curl to post messages read from a queue to our load balanced service. It is not a web browser that appears to be so widely assumed in terms of use cases.  It does not have any need for a cookie and therefore is not presenting one.  With a drop_aaatm_nocookie_conn value of 0 or 1 the connections were dropped and the client/server connections become broken/unhealthy.  Throughput is almost nil.  40 RPS will drop and go up and down from 0 - 8 with extended duration of 0.

 

I'll also note there is a counter for observing the number of these connections.

 

nsconmsg120 -K newnslog -d finalstatswt0 | grep -i aaa_auth_succ **
34311       0            734141 aaa_auth_succ
         3       0            729662 aaatm_no_cookie_on_authn_conn

 

I hope this might help someone else implementing client certificate authentication with a AAA authentication policy that requires a specific cert be presented.

Link to comment
Share on other sites

  • 1 year later...

Hello everybody, a question, can i use a selfsigned certificate for client auth? (Previously installing the selfsigned as a CA certificate on netscaler?)

I cant make it work, i got Reason "Certificate signature verification failed" 

 

Sep  8 12:33:38 <local0.debug> 10.128.50.71 08/09/2020:15:33:38 GMT dmz-coe 0-PPE-1 : default SSLLOG SSL_HANDSHAKE_FAILURE 1758485 0 :  SPCBId 42585409 - ClientIP 192.168.210.10 - ClientPort 64073 - VserverServiceIP 192.168.3.136 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - CLIENT_AUTHENTICATION_FAILED - SerialNumber "D0748A1785E89BEA" - Reason "Certificate signature verification failed" 
Sep  8 12:33:38 <local0.debug> 10.128.50.71 08/09/2020:15:33:38 GMT dmz-coe 0-PPE-1 : default SSLLOG SSL_HANDSHAKE_ISSUERNAME 1758486 0 :  SPCBId 42585409 - IssuerName " CN=test-sandbox" 
Sep  8 12:33:38 <local0.debug> 10.128.50.71 08/09/2020:15:33:38 GMT dmz-coe 0-PPE-1 : default SSLLOG SSL_HANDSHAKE_SUBJECTNAME 1758487 0 :  SPCBId 42585409 - SubjectName " CN=test-sandbox"

 

Thanks!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...