SSL/TLS Handshake deeper dive

[Chad Buser]

Hi I'm trying to understand and looking for suggestions on why the Client Key Exchange did not take place in this connection. I have a powershell script (forcing TLS 1.2) where I connect to a set of ADCs to do stuff. Here is the error I get from the script:

System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.

I'd like to see something more in Wireshark as to why the Client Key Exchange did not take place if possible. Both ADCs agree on the same Cipher however, the successful ADC 'ns-server-certificate' connects with SHA1 and a 2048 key size whereas the other ADC fails with SHA256 and a 512 key size. I know how we got in this position and how to fix it. However, If I look at the wireshark trace, I see how the successful connection progresses from:

Client Hello

Server Hello

Certificate, Server Hello Done

Client Key Exchange, Change CIpher Spec, Encrypted

Change CIpher Spec, Encrypted Handshake Message

Application Data

vs, the unsuccessful connection:

Client Hello

Server Hello

Certificate, Server Hello Done

However, it seems that Wireshark just reports on the connection with no real errors.

If I run the nstrace with 'capsslkeys ENABLED' will this give me the output I'm looking for or is it unrealistic to assume Wireshark is going to give me any output that matches the script error?

Best

Chad

[Marcelo Oguma de Souza]

Perhaps the ciphers offered by the server side are not compatible with the client side? Can you compare the list offered in both cases, looking at the Wireshark traces?

[Chad Buser]

Yes, mentioned above but let me expand, both traces show the same 26 Ciphers offered by the ADCs and the Client agrees on TLS_RSA_WITH_AES_256_CBC_SHA.

[Marcelo Oguma de Souza]

Then I suppose your client does not accept a 512bit key size. You can create a new self-signed certificate with a 2048bit key in the NetScaler.

[Chad Buser]

That is most likely going to be the fix but not the question I'm asking here.

[Subhojit Goswami]

Hi @Chad Buser​,

This needs further analysis, please raise a support ticket.

[Steven Wright]

Hi Chad,

>If I run the nstrace with 'capsslkeys ENABLED' will this give me the output I'm looking for or is it unrealistic to assume Wireshark is going to give me any output that matches the script error?

I don't believe nstrace set to capture SSL keys will give you the response you're looking for. The option 'capsslkeys ENABLED' will cause nstrace to record the pre-shared master key received during the handshake phase "Client Key Exchange, Change Cipher Spec, Encrypted," but that phase doesn't appear to have been reached yet. Since the client (the PowerShell script) stops immediately after the server exchange, it's likely that it didn't accept the server's response, and therefore, there isn't data on the wire for Wireshark to capture.

As Marcelo commented, one of the most likely reasons that the client stopped was that it didn't accept a 512-bit key. However, in your situation, I would enable System.Net tracing to create a log of the network communication, including the TLS handshake process. I believe this will help faster than reviewing WireShark, which is unlikely to give the output you need.

[Morten Kallesøe]

Hi Chad,

i guess your client is not accepting any of the ciphers. and if the powershell is not returning an error, how would you possibly see this in a trace?

Remember, packets dont lie!

[Chad Buser]

Thank you Steven. I was hoping Wireshark would tell me why it didn't accept/rejected the connection between the NS and the Windows Server but no dice. Something like 'insufficient key size' would have been nice but the only errors that were reported were from the script and Event Viewer which were the same and vague. I'll have to check out the tracing you suggested.