Jump to content
Welcome to our new Citrix community!

HA broke after upgrade NS13.0 76.29.nc


Recommended Posts

Hi,

 

 

Having a weird problem. After the upgrade to NS13.0 76.29.nc (from NS13.0 58.32.nc) the two nodes do not want to talk to each other within the HA, meaning no HA sync & no config replication.

The secondary node floods the logs with this:

Mar  7 12:17:36 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>
Mar  7 12:17:39 <local0.alert> <current-node-IP> 03/07/2021:12:17:39 GMT <current-node-hostname> 0-PPE-0 : default EVENT STATECHANGE 587 0 :  Device "self node <current-node-IP>" - State "SYNC Failure - Save remote config failed"
Mar  7 12:17:40 <local0.info> <current-node-hostname> nssync: Send HA File sync to netsvc
Mar  7 12:17:40 <local0.err> <current-node-hostname> nsconf: nsnetssl_connect: SSL_connect failed for <primary-node-ip>:3008: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Mar  7 12:17:40 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>
Mar  7 12:17:43 <local0.alert> <current-node-IP> 03/07/2021:12:17:43 GMT <current-node-hostname> 0-PPE-0 : default EVENT STATECHANGE 588 0 :  Device "self node <current-node-IP>" - State "SYNC Failure - Save remote config failed"
Mar  7 12:17:44 <local0.info> <current-node-hostname> nssync: Send HA File sync to netsvc
Mar  7 12:17:44 <local0.err> <current-node-hostname> nsconf: nsnetssl_connect: SSL_connect failed for <primary-node-ip>:3008: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Mar  7 12:17:44 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>
Mar  7 12:17:47 <local0.alert> <current-node-IP> 03/07/2021:12:17:47 GMT <current-node-hostname> 0-PPE-0 : default EVENT STATECHANGE 589 0 :  Device "self node <current-node-IP>" - State "SYNC Failure - Save remote config failed"
Mar  7 12:17:48 <local0.info> <current-node-hostname> nssync: Send HA File sync to netsvc
Mar  7 12:17:48 <local0.err> <current-node-hostname> nsconf: nsnetssl_connect: SSL_connect failed for <primary-node-ip>:3008: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Mar  7 12:17:48 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>
Mar  7 12:17:51 <local0.alert> <current-node-IP> 03/07/2021:12:17:51 GMT <current-node-hostname> 0-PPE-0 : default EVENT STATECHANGE 590 0 :  Device "self node <current-node-IP>" - State "SYNC Failure - Save remote config failed"
Mar  7 12:17:52 <local0.info> <current-node-hostname> nssync: Send HA File sync to netsvc
Mar  7 12:17:52 <local0.err> <current-node-hostname> nsconf: nsnetssl_connect: SSL_connect failed for <primary-node-ip>:3008: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Mar  7 12:17:52 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>
Mar  7 12:17:55 <local0.alert> <current-node-IP> 03/07/2021:12:17:55 GMT <current-node-hostname> 0-PPE-0 : default EVENT STATECHANGE 591 0 :  Device "self node <current-node-IP>" - State "SYNC Failure - Save remote config failed"
Mar  7 12:17:56 <local0.info> <current-node-hostname> nssync: Send HA File sync to netsvc
Mar  7 12:17:56 <local0.err> <current-node-hostname> nsconf: nsnetssl_connect: SSL_connect failed for <primary-node-ip>:3008: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Mar  7 12:17:56 <local0.err> <current-node-hostname> nsconf: failed to connect to the host <primary-node-ip>

 

I've rebuild one of them, add it to the pair - nothing.

Had to restore it from backup to get the configuration. After the restore, same thing.

I then tried with yet again a new machine, copied the certs & ran the config commands in CLI to restore (so, manually - not from backup). Ran into some problems as it would not take the LDAP and RADIUS actions creation from command line, but added those manually. After all that, HA status is still the same.

I've tried re-importing the ns-server-certificate.

I've tried re-binding it to the internal services.

No joy.

 

 

I've now shutdown the one that was not rebuilt, keeping this new one as primary.

Before the shutdown, i've rebooted that one forcing a failover. All that it accomplished was that the "new" secondary node now puts out the same error (ofc, different IPs and hostnames - they're reversed).

 

 

Could not find any info on what that error is about, or how to tackle it.

Any ideas ?

 

 

 

 

 

BR,

Sergiu K.

 

 

 

Link to comment
Share on other sites

Make sure that TLS 1.2 is enabled on the  nsrpcs-127.0.0.1-3008 internal service (Traffic management - Load balancing - Services - Internal services.

 

It's was forced in an earlier v13 firmware (around .60 if i recall right) that it should run secure. The problem is that ADC's that has been updated a lot, might still run SSLv3 and TLS 1.0 only

  • Like 6
Link to comment
Share on other sites

Asked around before doing my own upgrade and it appears from 13.0.64.35 TLS 1.0 and TLS 1.1 are depreciated and HA communication fails if TLS 1.2 is disabled. TLS 1.2 is disabled on the firmware version by default, so if it were never enabled for HA it will fail. 

TLS 1.2 needs to be enabled for internal services or will get the errors you are seeing. 

Link to comment
Share on other sites

  • 2 weeks later...
  • 7 months later...

I am testing new ADC VPX 13.1 xxx  HA in my LAB, Found the exact same error.  It seems my existing VPX before adding another one to join HA has some Internal service SSL certificate and SSL setting and Cipher Groups change from the default with some hardening.   I finally fixed the error by changing these settings back to default manually on both Unit.

Fall back to ns-server-certificate, SSL settings to back to sslv3, TLS 1.0, TLS1.1 TLS 1.2, Cipher Group back to DEFAULT     for now.  After doing this manually on both UNITs. The HA status come back to normal again. Do it all for the internal 127 443  SSL TCP 3008   3009 everything

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...