Or, My TLS connection just dies during the handshake – because of FIPS!

I was working with John M. on a problem connecting a client machine to talk to z/OS TN3270, and this identified some “interesting” holes.

• The root cause is that on z/OS 3.1 and earlier AT-TLS does not support FIPS with TLS 1.3.
• There is support in z/OS 3.2 for FIPS 140-3.
• The cards in ICSF need to be configured for FIPS. If they are not configured, the sessions will fail with a trace entry in the CTRACE output saying “FIPS not supported” or some other vague message.
• You can use the operator command D ICSF,CARDS to display the status.
• You can use the ISPF panels.
  • In ISPF option 6 type the command @ICSF. This displays the ICSF main panel.
  • Option 1 COPROCESSOR MGMT
  • It displays your co-processors.
  • Use the S line command on the co-processors
  • If you get a message like FIPS Compliance Mode          : NOT SUPPORTED. You need to reconfigure your co-processors.
• To configured FIPS, it is a destructive reset, and all master keys will be reset. This needs to be carefully planned.

Steps to solving the problem

You can use tools like Wireshark to display the traffic, and sometimes see why a TLS handshake fails.

Many of the problems I experienced were due to configuration problems on z/OS. I got a CTRACE trace on z/OS, see GSK trace and TCPIP and this usually allowed me to fix the problem.

Alert (40)

Alert Message:Level: Fatal (2): Description: Handshake Failure (40)

I used the gsksrvr ctrace to find that I did not have any TLS 1.3 certificates in my configuration.

Alert (51)

With TLS 1.3, A certificate like

SUBJECTSDN(CN('10.1.1.2') - 
           O('NISTEC256') - 
           OU('SSS')) - 
 ALTNAME(IP(10.1.1.2))-                        
 NISTECC - 
 KEYUSAGE(   HANDSHAKE     )  - 
 SIZE(256 ) - 
 SIGNWITH (CERTAUTH LABEL('DOCZOSCA')) -       
 WITHLABEL('NISTEC256') 

Failed. But changing it to SIZE(512) worked. Even though size 256 is supported.

Using TLS 1.3, the handshake to TN3270 failed with no reason.

I tracked down some problems due to FIPS being enabled.

FIPS standards establish requirements for ensuring computer security and interoperability, and are intended for cases in which suitable industry standards do not already exist.

I think of FIPS as taking the existing standards and making them a bit more secure. For example not allowing some cipher suites. Not allowing certificates with small keys.

Enabling FIPS properly does not look easy. For example the documentation says it requires that load modules are cryptographically signed, so code authorised programs can check they have not been changed. Under the covers I believe that when IBM ships a module, it calculates the hash of the code, then encrypts the hash, and stores the encrypted has within the loadmodule. At runtime you use IBM’s public key to decrypt this value; does the same hash on the module, and compares this.

Once this has been done, you can add statements to the ICSF configuration, such as FIPSMODE(YES,FAIL(YES)).

This says use FIPS, and if any checking fails – fail the request.

In z/OS 3.2 there is FIPS support for TLS 1.3 see option FIPSMODE(140-3,INDICATE,FAIL(fail-option))

Not all configurations are supported

The TLS 1.3 ciipher suites, ChaCha20 and ChaCha20-Poly1305 are not supported by FIPS. You need to use cipher suites, configured with AES-GCM or AES-CCM.

I ran my test using FIPS

I could see in Wireshark that there was the TLS 1.3 trace

• ClientHello request going to the server
• ServerHello coming from the server
• Change Cipher spec coming from the server
• and nothing. No Alert message.

I found an entry in the z/OS 2.5 documentation.

The FIPS 140-2 standard does not define support for TLSv1.3 or the new cipher suites defined for it. Enabling both the TLSv1.3 protocol and FIPS support results in an error.

When my request failed I got CTRACE entries like

S0W1      MESSAGE   00000004  12:52:55.333904  SSL_ERROR                                  
  Job TCPIP     Process 0201001E  Thread 00000001  crypto_chacha_encrypt_ctx              
  ICSF service failure: CSFPSKE retCode = 0x8, rsnCode = 0xbfe                            
                                                                                          
S0W1      MESSAGE   00000004  12:52:55.334123  SSL_ERROR                                  
  Job TCPIP     Process 0201001E  Thread 00000001  crypto_chacha_encrypt_ctx              
  The algorithm or key size is not supported by ICSF FIPS                                 
                                                                                          
S0W1      MESSAGE   00000004  12:52:55.334355  SSL_ERROR                                  
  Job TCPIP     Process 0201001E  Thread 00000001  gsk_encrypt_tls13_record               
  ChaCha20 Encryption failed: Error 0x0335308f 

Where the return code 0xbfe is

The PKCS #11 algorithm, mode, or keysize is not approved for ICSF FIPS 140-2. This reason code can be returned for PKCS #11 clear key requests when ICSF is in a FIPS 140-2 mode or 140-3,HYBRID mode. To see how 8/BFE(3070) can be returned when the ICSF FIPSMODE is 140-3,HYBRID, see ‘Requiring FIPS 140-2 algorithm checking from select z/OS PKCS #11 applications’ in z/OS Cryptographic Services ICSF Writing PKCS #11 Applications.

May the FIPS code is badly implemented, by not producing an alert message such as “FIPS processing problem”, but some security products to not display error information, because it makes it easier to break in!