Jump to content
Welcome to our new Citrix community!

TCP MSS anomaly on Netscaler 11.1-52.13

Recommended Posts

I've been troubleshooting a bizarre problem in which VPN connected users are unable to reach an internal HTTPS website via Netscaler.


It looks like a bug in Netscaler's TCP implemntation.  From looking at traces, the following is happening when the client initiates the HTTPS handshake:

  1. Client sends TCP SYN with MSS=1327 (determined by client-side VPN virtual adapter)
  2. Netscaler sends TCP SYN+ACK with MSS=1330
  3. TCP handshake completes succesfully and Client Hello is received by Netscaler
  4. Netscaler responds with certificate chain, forming multiple segments of MSS 1330 with DF bit set
  5. These are too big for the VPN tunnel, and cannot be fragmented, so the Cisco ASA (VPN concentrator) responds to the Netscaler with ICMP type 3,4 (fragmentation needed) and Next Hop MTU=1327
  6. Despite receiving ICMP 3,4 messages, Netscaler continues sending oversized segments which client does not receive.
  7. Client connection hangs and eventually times out

From reading this document - noting it is intended for Telco customers and carrying the caveat of requiring Citrix support to change the values - it would appear that Netscaler uses only 8 MSS values which is required to support its implementation of SYN cookies.  When establishing a TCP connection, the send MSS used by the Netscaler will be chosen from this table either by matching or rounding down to the next value.


The preset MSS table values in 11.1-52.13 are




(presumably the first 8 exist to provide Jumbo Frames support if enabled - otherwise only the latter 8 are used)


Based on my troubleshooting, if the MSS value in the SYN lies between 1322 and 1329 then for some reason, instead of rounding down to MSS=1220, the Netscaler instead rounds UP to the value MSS=1330 and then uses that value as the send MSS for the connection despite it being greater than the client's advertised MSS.


I have tried changing TCP parameters, disabling SYN Cookies and also disabling/enabling Path MTU Discovery to no avail.


Eventually I changed the MSS table as per the above document (on a non-production device) and as soon as I did this the issue went away and the connection send MSS was negotiated down to the nearest value.


I'm posting this incase anyone else has the same problem or encounters it in the future - however I'd also like some clarification from Citrix on whether we should be changing the MSS table or whether there are some risks or side-effects to be aware of.


Its worth noting that on the customer WAN I'm working on presently, there are a multitude of different tunnelling protocols in use and the MSS of connections is generally in the 1300-1400 range.  The default MSS table provides no granularity in this range.  As an experiment I have implemented a custom table which matches all the common tunnel MTU's seen on this particular network and as a result the TCP connections are being matched precisely instead of the Netscaler using an MSS which is 5-10% less than would be possible.



  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...

I have done some further testing and identified that this issue appeared in 11.1-52.13 and has persisted in 53.11.  Earlier 11.1 releases and 11.0, 10.x aren't affected.


Its easy to reproduce the problem if you set your client side interface MTU to between 1362 and 1369 then observe Wireshark whilst initiating a TCP connection which is likely to result in full-size frames being returned by the Netscaler, such as a HTTPS handshake.


When the client sends a TCP SYN with MSS between 1322 and 1329 the Netscaler chooses the MSS value 1330 from its table and then replies with packets which exceed the client MSS, showing as frames of size 1384 in Wireshark.  When reproducing this with Wireshark and a custom interface MTU the "oversized" packets are recieved by Windows, but if its the network path rather than the client interface MTU which lies in this range then the reply packet from the Netscaler is instead dropped by the intermediate router, the Netscaler ignores any resulting ICMP 3,4 packet and the connection times out.


Anyone thinking of rolling these later 11.1 versions should pay attention to this issue as it will affect client connections over VPNs where the path MTU lies in the 1320-1330 range.

Link to comment
Share on other sites

Neil  ,

  Your observation is correct . If the MSS table is edited as per the document you have pointed , this issue will disappear. This can be used as a workaround . Yes tuning is supported  . 


nsapimgr_wr.sh -ys mss_table=9176,9156,8192,7168,6144,4196,3072,2048,1460,1400,1330,1212,956,536,384,128


To make the setting persist across reboots , as mentioned in the document it should be added to /nsconfig/rc.netscaler





  • Like 2
Link to comment
Share on other sites

Hi Neil,


Yes it is supported to write 16 standard-Ethernet values instead of using Jumbo. There is an example in the same page as well as how to make this change permanent across reboots. For your clarity:


nsapimgr_wr.sh -ys mss_table=1460,1440,1420,1400,1380,1360,1340,1320,1300,1280,1260,1212,956,536,384,128


Values can be modified accordingly based on your needs. Aforementioned example is just for reference purposes.




  • Like 2
Link to comment
Share on other sites

  • 1 month later...
  • 3 years later...

Neil, thanks for posting this we are experiencing the exact issue with marakei vpn from our offsite network. 


Our appliance version -  NetScaler NS11.1: Build 52.13.nc,


Couple of questions for Citrix/Neil

  1. is there a downtime for setting the mss table? 
    • #nsapimgr -ys mss_table=1460,1440,1420,1400,1380,1360,1340,1320,1300,1280,1260,1212,956,536,384,128
  2. It appears, turning off DF bit could also have solved the issue. Any reason not to go with this? [https://support.citrix.com/article/CTX230693]
    • nsapimgr -ys drop_df_flag=1
  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...