When debugging TLS problems I got various return codes. I’m collecting them here, so I can find them next time I have a problem.
I’d be happy to add to any problems and solutions you find, please let me know.
TLS Handshake failure
Alert 40
Wireshark produced
- Alert Message
- Level: Fatal (2)
- Description: Handshake Failure (40)
Looking in the CTRACE I got
No SSL V3 cipher specs enabled for TLS V1.3
See tls-1-3-everything-possibly-needed-know. This has
just five recommended cipher suites:
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256
- TLS_AES_128_GCM_SHA256
- TLS_AES_128_CCM_8_SHA256
- TLS_AES_128_CCM_SHA256
Alert 51
With TLS 1.3, A certificate like
SUBJECTSDN(CN('10.1.1.2') -
O('NISTEC256') -
OU('SSS')) -
ALTNAME(IP(10.1.1.2))-
NISTECC -
KEYUSAGE( HANDSHAKE ) -
SIZE(256 ) -
SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -
WITHLABEL('NISTEC256')
Failed. But changing it to SIZE(512) worked. Strange, because size 512 is supposed to be supported.
Debug details
From the CTRACE
ICSF service failure: CSFPPKS retCode = 0x8, rsnCode = 0x2b00
S0W1 MESSAGE 00000004 10:25:45.006617 SSL_ERROR
Job TCPIP Process 0001003B Thread 00000003 crypto_sign_data
crypto_ec_sign_data() failed: Error 0x03353084
S0W1 MESSAGE 00000004 10:25:45.006883 SSL_ERROR
Job TCPIP Process 0001003B Thread 00000003 construct_tls13_certificate_verify_message
Unable to generate certificate verify message: Error 0x03353084
S0W1 MESSAGE 00000004 10:25:45.007124 SSL_ERROR
Job TCPIP Process 0001003B Thread 00000003 send_tls13_alert
Sent TLS 1.3 alert 51 to ::ffff:10.1.0.2.43416.
in z/OS Unix the command
grep 03353084 /usr/incl/gsk
gave
/usr/include/gskcms.h:#define CMSERR_ICSF_SERVICE_FAILURE 0x03353084
The ICSF API points to return codes. 2B00 (11008) says
The public or private key values are not valid (for example, the modulus or an exponent is zero or the exponent is even) or the key could not have created the signature (for example, the modulus value is less than the signature value). In any case, the key cannot be used to verify the signature.
Changing to
Policy agent
...
ServerCertificateLabel NISTECC521
...
RACDCERT ID(START1) GENCERT -
SUBJECTSDN(CN('10.1.1.2') -
O('NISTECC256') -
OU('SSS')) -
ALTNAME(IP(10.1.1.2))-
NISTECC -
KEYUSAGE(HANDSHAKE ) -
SIZE(256) -
SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -
WITHLABEL('NISTECC256')
worked.
I needed to do F CPAGENT,REFRESH to pickup the change. I needed to refresh the policy agent, because I was using TN3270, which uses AT-TLS.
Session just ends with no alert
Looking at the CTRACE output I got
S0W1 MESSAGE 00000004 12:52:55.333904 SSL_ERROR
Job TCPIP Process 0201001E Thread 00000001 crypto_chacha_encrypt_ctx
ICSF service failure: CSFPSKE retCode = 0x8, rsnCode = 0xbfe
S0W1 MESSAGE 00000004 12:52:55.334123 SSL_ERROR
Job TCPIP Process 0201001E Thread 00000001 crypto_chacha_encrypt_ctx
The algorithm or key size is not supported by ICSF FIPS
S0W1 MESSAGE 00000004 12:52:55.334355 SSL_ERROR
Job TCPIP Process 0201001E Thread 00000001 gsk_encrypt_tls13_record
ChaCha20 Encryption failed: Error 0x0335308f
The return code 0xbfe is
The PKCS #11 algorithm, mode, or keysize is not approved for ICSF FIPS 140-2. This reason code can be returned for PKCS #11 clear key requests when ICSF is in a FIPS 140-2 mode or 140-3,HYBRID mode. To see how 8/BFE(3070) can be returned when the ICSF FIPSMODE is 140-3,HYBRID, see ‘Requiring FIPS 140-2 algorithm checking from select z/OS PKCS #11 applications’ in z/OS Cryptographic Services ICSF Writing PKCS #11 Applications.
FIPS was incorrectly specified. For example FIPS-140 with TLS 1.3
ColinPaice 