Jump to content
Welcome to our new Citrix community!

NetScaler 11.1 huge memory and client session peak caused by VPN connections


Mark Nickolai 2

Recommended Posts

Hello,

 

first of all some flavour indroduction of the scenario:

 

due to COVID-19 we moved a lot of people recently to our client SSL VPN, which is based on the NetScaler Gateway funciton in the NetScaler ADC (version 11.1-63.15 of our MPX5650 appliance.

According to the datasheet up to 5000 users are supported.

Currently we have around 900 users in parallel connected, and dispite some performance issues for the users it seems to work quiet well.

 

Or at least could be...

 

We had 2 times already an issue where there was a huge tcp client session peak along with a huge memory peak probably caused by connected vpn clients.

The peaks lead to kind of soft crash of the appliance, where all the people where kicked out of vpn, webui was not accessible anymore.

The state was fixed by removing all tcp sessions.

 

So what I want to know now, is: How to prevent this situation?

I know for lb vservers there are tons of possibilities to to prevent situation like this, while the Internet is not so responsive regarding hints how to prevent session surges in VPN traffic itself.

 

I get the feeling that it is not by design to prevent a session overload in the ssl vpn tunnel connection itself.

 

Any recommondation would be highly appreciated.

 

Best regards

 

Mark

Link to comment
Share on other sites

Hi Mark,

 

do you already used the Reporting Tab in your ADC WebUI to check the memory / cpu load vs. Bandwidth consumption vs. concurrent aaa sessions? Maybe the main problem comes from another sector? Is your MPX direct attached to the internet or is all traffic routed through firewalls, too?

Link to comment
Share on other sites

Hi Julian,

 

we monitor the usage via snmp with a separate monitoring system.

 

I saw that the peaks have no relation to the concurrent users amount.

 

Is there a good way to monitor bandwidth consumption via report? What would you suggest to look for?

 

Our internet traffic is routed through a firewall appliance in a first step. Meanwhile we added there traffic shapers for maximum TCP sessions, which seem to work.

I just wonder if there is no mechanism in the netscaler itself, like there are a lot of mechanisms to protect LB Vservers.

 

 

 

 

Link to comment
Share on other sites

  • 4 weeks later...

Hi Scott,

 

currently we assume, that our ssl vpn clients opening tons of connections, for whatever reason.

 

We figured out that 1 ssl session from a client towards netscaler gateway equals 1 intranet application connection towards vpn.

 

So we created DOS protections in our Firewall appliance in front of NetScaler, to reduce the maximum amount of sessions per source IP.

In addition we set up a maximum tcp session towards the external IPs of NetScaler in general.

 

 

 

Link to comment
Share on other sites

8 hours ago, Mark Nickolai 2 said:

Hi Scott,

 

currently we assume, that our ssl vpn clients opening tons of connections, for whatever reason.

 

We figured out that 1 ssl session from a client towards netscaler gateway equals 1 intranet application connection towards vpn.

 

So we created DOS protections in our Firewall appliance in front of NetScaler, to reduce the maximum amount of sessions per source IP.

In addition we set up a maximum tcp session towards the external IPs of NetScaler in general.

 

 

 

Hi Mark,

I'm wondering how long you've had this mitigation in place and if you've had the issue occur again? 

I'm also interested if numbers have decreased in VPN since the COVID-19 has flattened? 
I ask the last question because our experience seems to be that as we approach 1000 users this issue happens everyday or multiple times a day.  If we send 25% of our staff to our DR VPN solution this issue went almost a month without occurring.  The presumed "fix" was 11.1.64.11 but as soon as we directed the 25% to connect to production VPN this reared it's ugly face again.

Here is a graph of our connections skyrocketing before a failure
 

TCP_Jump.JPG

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...