How to restrict what certificates and algorithms clients can use to connect to java web servers

As part of your regular housekeeping you want to limit connections to your web server from weak keys and algorithms.   Making changes to the TLS configuration could be dangerous, as there is no “warning mode” or statistics to tell you if weak algorithms etc are being used.  You have to make a change and be prepared to have problems.

In this posting I’ll explain how to do it, then explain some of the details behind it.

How to restrict what certificates and algorithms can be used by web servers and java programs doing TLS.

One way which does not work.

The jvm.options file provided by mqweb includes commented out

-Djdk.tls.disabledAlgorithms=… and  -Djdk.tls.disabledAlgorithms=…..

These is the wrong way of specifying information, as you do it via the java.security file, not -D… .

Create an mqweb specific private disabled algorithm file

Java uses a java.security file to define security properties.

On my Ubuntu, this file if in /usr/lib/jvm/…/jre/lib/security/java.security  .

Create a file mqweb.java.security.  It can go anywhere – you pass the name using a java system property.

Copy  from the java.security file to your file, the lines with  with jdk.tls.disabledAlgorithms=..  and jdk.certpath.disabledAlgorithms=… . 

On my system, the lines are (but your security people may have changed them – if so,  you might want to talk to them before making any changes)

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024,     EC keySize < 224, 3DES_EDE_CBC, anon, NULL

jdk.certpath.disabledAlgorithms=jdk.certpath.disabledAlgorithms=MD2, MD5,    SHA1 jdkCA & usage TLSServer,    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224

The jvm.options file provided by IBM has

-Djdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC, DESede, EC keySize < 224, SHA1 jdkCA & usage TLSServer

So you may want to add this as in your override file ( without the -D), so add “, SHA1 jdkCA & usage TLSServer” to  jdk.certpath.disabledAlgorithms .

Tell mqweb to use this file

Create a java system property in the mqweb jvm.options file

“-Djava.security.properties=/home/colinpaice/eclipse-workspace-C/sslJava/bin/serverdisabled.properties”

Restart your web server.  You have not changed anything – just copied some definitions into an mqweb specific file, so it should work as before.

Limit what can be used

I set up several certificates with combination of RSA and Elliptic Curves, varying keysize, signatures;  and signed with CAs with RSA, and Elliptic Curve, and different signatures.

For example RSA4096,SHA256withECDSA,/EC256,SHA384with ECDSA means

• RSA4096 certificate is RSA with a key size of 4096
• SHA256withECDSA signed with this
  
• /EC256 the CA has a public key of EC 256
  
• SHA384with ECDSA and the CA was signed with this
  

I then specified different options in the servers’ file, and recorded if they TLS connection worked or not; if not – why not.

certpath: RSA keySize <= 2048

Server EC407,   SHA256withRSA,   /RSA4096,   SHA512withRSA

• ✅RSA4096,SHA256withECDSA,   /EC256,SW=SHA384with ECDSA
• ❌RSA2048, SHA256withRSA,      /RSA4096,/SHA512withRSA
• ✅ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
• ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: RSA keySize <= 4096

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

• ❌RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ❌ RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
• ❌EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
• ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath:EC keySize <= 256

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

• ❌  RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ✅ RSA2048,SHA256withRSA,       /RSA4096,  SHA512withRSA
• ✅ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
• ❌ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

tls:EC keySize <= 256

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

• ✅ RSA4096,SHA256withECDSA,   /EC256,       SHA384with ECDSA
• ✅ RSA2048,SHA256withRSA,        /RSA4096,  SHA512withRSA
• ✅ EC407,      SHA256withRSA,       /RSA4096, SHA512withRSA
• ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: SHA256withRSA

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

• ✅ RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ❌RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
• ❌ EC407,      SHA256withRSA,    /RSA4096, SHA512withRSA
• ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

tls:SHA256withRSA

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

• ✅ RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ❌RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
• ❌ EC407,      SHA256withRSA,    /RSA4096, SHA512withRSA
• ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

either: RSA

Server RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA

All requests failed due to the server’s RSA.   Only 18 out of 50 cipher suites were available.  Server reported javax.net.ssl.SSLHandshakeException: no cipher suites in common

• ❌RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ❌ RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
• ❌ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
• ❌EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: RSA keySize == 4096

Server RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA

This was a surprise as I did not think this would work!

• ❌RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
• ❌ RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
• ❌EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
• ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

Summary of overriding.

You can specify restrictions in the server’s jdk.certpath.disabledAlgorithms and jdk.certpath.disabledAlgorithms. The restrictions apply to the how the certificate has been signed and the CA certificate.

You should check that the server’s certificate is not affected.

More details and what happens under the covers

The section below may be too much information, unless you are trying to work out why something is not working.

In theory jdk.tls.disabledAlgorithms and jdk.certpath.disabledAlgorithms are used for different areas of checking – reading certificates from key files, and what is passed during the handshake – but this does not seem to be true.  I found that it was best to put restrictions on both lines.

A certificate is of type RSA, EC, or DSA.

A certificate is signed for example Signature Algorithm: SHA256withECDSA.   This comes from the CA which signed it, message digest SHA256, and the CA is an Elliptic Curve.  See How do I create a certificate with Elliptic Curve (or RSA).

Signature Algorithms: is a combination of Hash Algorithm and Signature Type.   There are 6 hash algorithms: md5, sha1, sha224, sha256, sha384, sha512, and three types:  rsa, dsa, ecdsa.    These can be combined to to give 14 combinations of Signature Algorithms used in TLSv1.2

You can use java.security to control what TLS does.  On my Ubuntu this file /usr/lib/jvm/java-8-oracle/jre/lib/security/java.security.

This includes

• jdk.certpath.disabledAlgorithms: Algorithm restrictions for certification path (CertPath) processing:  In some environments, certain algorithms or key lengths may be undesirable for certification path building and validation. For example, “MD2” is generally no longer considered to be a secure hash algorithm. This section describes the mechanism for disabling algorithms based on algorithm name
  and/or key length. This includes algorithms used in certificates, as well as revocation information such as CRLs and signed OCSP Responses.
• jdk.tls.disabledAlgorithms: Algorithm restrictions for Secure Socket Layer/Transport Layer Security  (SSL/TLS) processing.  In some environments, certain algorithms or key lengths may be undesirable when using SSL/TLS. This section describes the mechanism for disabling algorithms during SSL/TLS security parameters negotiation, including protocol version negotiation, cipher suites selection, peer authentication and key exchange mechanisms.

I found you get better diagnostics if you put the restrictions on both statements.

The TLS Handshake (relating to java.security)

Server starts  up

• I had 50 available cipher suites
• Using -Djdk.tls.server.cipherSuites=…,… you can specify a comma separated list of  which cipher suites you want make available.  I recommend you do not specify this and use the defaults.
• Using jdk.tls.disabledAlgorithm you can specify which handshake information is not allowed.  For example
  •  Any of the following would stop cipher suite TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 from being used
    • java.security.tls = AES_256_CBC
    • java.security.tls= SHA384 – this loses 4 certificate TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 etc
    • java.security.tls=TLS_ECDHE_ECDSA
  • Elliptic curves names Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}.  Specifying EC keySize <= 521 would only allow {sect571k1, sect571r1} to be used
• The server builds a supported list of cipher suites, signature algorithm, and elliptic curve names

Client starts up

• As with the server, the client builds up a list of supported cipher suites, signature algorithms, and supported Elliptic curve names.
• The client sends “ClientHello” and the list to the server.

Server processes

• The server takes this list and iterates over it  to find the first acceptable certificate ( or for wlp, if <ssl … serverKeyAlias=”…” />, then the specified aliases  is used )
  • if the cipher suite name is like  TLS_AAA_BBB_WITH…   AAA must match the servers certificate type ( RSA, Elliptic Curve, DSA)
  • the signature algorithm (BBB).   This is the algorithm for encrypting the payload, and the algorithm for calculating the hash of the payload
  • if the certificate is EC,  check the  elliptic curve name is valid.  A  server’s certificate created with openssl ecparam -name prime256v1, would be blocked if EC keySize <= 256 was specified in the client resulting.
• If no certificate was found in the trust store which passed all of the checks, it throws javax.net.ssl.SSLHandshakeException: no cipher suites in common , and closes the connection
• The server sends “ServerHello” and the server’s public key to the client.
• The server sends the types of certificate it will accept.   This is typically RSA, DSA, and EC
• The server sends down the Elliptic Curve names it will accept, if present  – I dont think it is used on the java client
• The server uses the jdk.certpath.disabledAlgorithm to filter the list of Signature Algorithms, and sends this filtered list to the client.
• The server extracts the CAs and self signed certificates from the trust store and sends them down to the client.
• The server sends “ServerHelloDone”, saying over to you to respond.

Client processing:

• Checks the server’s certificate is valid, including
  • Checks the public certificate of the servers CA chain is allowed according to the client’s jdk.certpath.disabledAlgorithm..  So jdk.certpath.disabledAlgorithm =…, SHA256withECDSA would not allow a  server’s certificate with Signature algorithm:SHA256withECDSA .
    
• The client takes the list of certificate types ( RSA, DSA, EC), and CAs and iterates over the keystore and selects the records where
  • the certificate type in the list
  • the signature algorithm is in the list
  • the certificate signed by one of the CA’s in the list
• Displays this list for the end user to select from.  It looks like the most recently added certificate is first in the list.
• The client sends “Finished” and  the selected certificate to the server

Server processing

• The server checks the certificate and any imbeded CA certificates from the client matches.
  • Checks the signature algorithm
  • Checks the constraints, for example RSA keySize < 2048
  • Checks the certificate and CA are valid

End of handshake.

This is a useful link for describing the java.security parameters.

This specification  describes the handshake, with the “ClientHello” etc.