How to restrict what certificates and algorithms clients can use to connect to java web servers

As part of your regular housekeeping you want to limit connections to your web server from weak keys and algorithms.   Making changes to the TLS configuration could be dangerous, as there is no “warning mode” or statistics to tell you if weak algorithms etc are being used.  You have to make a change and be prepared to have problems.

In this posting I’ll explain how to do it, then explain some of the details behind it.

How to restrict what certificates and algorithms can be used by web servers and java programs doing TLS.

One way which does not work.

The jvm.options file provided by mqweb includes commented out

-Djdk.tls.disabledAlgorithms=… and  -Djdk.tls.disabledAlgorithms=…..

These is the wrong way of specifying information, as you do it via the java.security file, not -D… .

Create an mqweb specific private disabled algorithm file

Java uses a java.security file to define security properties.

On my Ubuntu, this file if in /usr/lib/jvm/…/jre/lib/security/java.security  .

Create a file mqweb.java.security.  It can go anywhere – you pass the name using a java system property.

Copy  from the java.security file to your file, the lines with  with jdk.tls.disabledAlgorithms=..  and jdk.certpath.disabledAlgorithms=… . 

On my system, the lines are (but your security people may have changed them – if so,  you might want to talk to them before making any changes)

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024,     EC keySize < 224, 3DES_EDE_CBC, anon, NULL

jdk.certpath.disabledAlgorithms=jdk.certpath.disabledAlgorithms=MD2, MD5,    SHA1 jdkCA & usage TLSServer,    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224

The jvm.options file provided by IBM has

-Djdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC, DESede, EC keySize < 224, SHA1 jdkCA & usage TLSServer

So you may want to add this as in your override file ( without the -D), so add “, SHA1 jdkCA & usage TLSServer” to  jdk.certpath.disabledAlgorithms .

Tell mqweb to use this file

Create a java system property in the mqweb jvm.options file

“-Djava.security.properties=/home/colinpaice/eclipse-workspace-C/sslJava/bin/serverdisabled.properties”

Restart your web server.  You have not changed anything – just copied some definitions into an mqweb specific file, so it should work as before.

Limit what can be used

I set up several certificates with combination of RSA and Elliptic Curves, varying keysize, signatures;  and signed with CAs with RSA, and Elliptic Curve, and different signatures.

For example RSA4096,SHA256withECDSA,/EC256,SHA384with ECDSA means

  • RSA4096 certificate is RSA with a key size of 4096
  • SHA256withECDSA signed with this
  • /EC256 the CA has a public key of EC 256
  • SHA384with ECDSA and the CA was signed with this

I then specified different options in the servers’ file, and recorded if they TLS connection worked or not; if not – why not.

certpath: RSA keySize <= 2048

Server EC407,   SHA256withRSA,   /RSA4096,   SHA512withRSA

  • ✅RSA4096,SHA256withECDSA,   /EC256,SW=SHA384with ECDSA
  • RSA2048, SHA256withRSA,      /RSA4096,/SHA512withRSA
  • ✅ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
  • ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: RSA keySize <= 4096

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

  • RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
  • ❌EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
  • ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath:EC keySize <= 256

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

  • ❌  RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • ✅ RSA2048,SHA256withRSA,       /RSA4096,  SHA512withRSA
  • ✅ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
  • ❌ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

tls:EC keySize <= 256

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

  • ✅ RSA4096,SHA256withECDSA,   /EC256,       SHA384with ECDSA
  • ✅ RSA2048,SHA256withRSA,        /RSA4096,  SHA512withRSA
  • ✅ EC407,      SHA256withRSA,       /RSA4096, SHA512withRSA
  • ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: SHA256withRSA

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

  • ✅ RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • ❌RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
  • ❌ EC407,      SHA256withRSA,    /RSA4096, SHA512withRSA
  • ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

tls:SHA256withRSA

Server EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA

  • ✅ RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • ❌RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
  • ❌ EC407,      SHA256withRSA,    /RSA4096, SHA512withRSA
  • ✅ EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

either: RSA

Server RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA

All requests failed due to the server’s RSA.   Only 18 out of 50 cipher suites were available.  Server reported javax.net.ssl.SSLHandshakeException: no cipher suites in common

  • ❌RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • ❌ RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
  • ❌ EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
  • ❌EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

certpath: RSA keySize == 4096

Server RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA

This was a surprise as I did not think this would work!

  • RSA4096,SHA256withECDSA, /EC256,       SHA384with ECDSA
  • RSA2048,SHA256withRSA,     /RSA4096,  SHA512withRSA
  • ❌EC407,      SHA256withRSA,      /RSA4096, SHA512withRSA
  • ✅EC384,       SHA256withECDSA, /EC256,      SHA384withECDSA

Summary of overriding.

You can specify restrictions in the server’s jdk.certpath.disabledAlgorithms and jdk.certpath.disabledAlgorithms. The restrictions apply to the how the certificate has been signed and the CA certificate.

You should check that the server’s certificate is not affected.

More details and what happens under the covers

The section below may be too much information, unless you are trying to work out why something is not working.

In theory jdk.tls.disabledAlgorithms and jdk.certpath.disabledAlgorithms are used for different areas of checking – reading certificates from key files, and what is passed during the handshake – but this does not seem to be true.  I found that it was best to put restrictions on both lines.

A certificate is of type RSA, EC, or DSA.

A certificate is signed for example Signature Algorithm: SHA256withECDSA.   This comes from the CA which signed it, message digest SHA256, and the CA is an Elliptic Curve.  See How do I create a certificate with Elliptic Curve (or RSA).

Signature Algorithms: is a combination of Hash Algorithm and Signature Type.   There are 6 hash algorithms: md5, sha1, sha224, sha256, sha384, sha512, and three types:  rsa, dsa, ecdsa.    These can be combined to to give 14 combinations of Signature Algorithms used in TLSv1.2

You can use java.security to control what TLS does.  On my Ubuntu this file /usr/lib/jvm/java-8-oracle/jre/lib/security/java.security.

This includes

  • jdk.certpath.disabledAlgorithms: Algorithm restrictions for certification path (CertPath) processing:  In some environments, certain algorithms or key lengths may be undesirable for certification path building and validation. For example, “MD2” is generally no longer considered to be a secure hash algorithm. This section describes the mechanism for disabling algorithms based on algorithm name
    and/or key length. This includes algorithms used in certificates, as well as revocation information such as CRLs and signed OCSP Responses.
  • jdk.tls.disabledAlgorithms: Algorithm restrictions for Secure Socket Layer/Transport Layer Security  (SSL/TLS) processing.  In some environments, certain algorithms or key lengths may be undesirable when using SSL/TLS. This section describes the mechanism for disabling algorithms during SSL/TLS security parameters negotiation, including protocol version negotiation, cipher suites selection, peer authentication and key exchange mechanisms.

I found you get better diagnostics if you put the restrictions on both statements.

The TLS Handshake (relating to java.security)

Server starts  up

  • I had 50 available cipher suites
  • Using -Djdk.tls.server.cipherSuites=…,… you can specify a comma separated list of  which cipher suites you want make available.  I recommend you do not specify this and use the defaults.
  • Using jdk.tls.disabledAlgorithm you can specify which handshake information is not allowed.  For example
    •  Any of the following would stop cipher suite TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 from being used
      • java.security.tls = AES_256_CBC
      • java.security.tls= SHA384 – this loses 4 certificate TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 etc
      • java.security.tls=TLS_ECDHE_ECDSA
    • Elliptic curves names Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}.  Specifying EC keySize <= 521 would only allow {sect571k1, sect571r1} to be used
  • The server builds a supported list of cipher suites, signature algorithm, and elliptic curve names

Client starts up

  • As with the server, the client builds up a list of supported cipher suites, signature algorithms, and supported Elliptic curve names.
  • The client sends “ClientHello” and the list to the server.

Server processes

  • The server takes this list and iterates over it  to find the first acceptable certificate ( or for wlp, if <ssl … serverKeyAlias=”…” />, then the specified aliases  is used )
    • if the cipher suite name is like  TLS_AAA_BBB_WITH…   AAA must match the servers certificate type ( RSA, Elliptic Curve, DSA)
    • the signature algorithm (BBB).   This is the algorithm for encrypting the payload, and the algorithm for calculating the hash of the payload
    • if the certificate is EC,  check the  elliptic curve name is valid.  A  server’s certificate created with openssl ecparam -name prime256v1, would be blocked if EC keySize <= 256 was specified in the client resulting.
  • If no certificate was found in the trust store which passed all of the checks, it throws javax.net.ssl.SSLHandshakeException: no cipher suites in common , and closes the connection
  • The server sends “ServerHello” and the server’s public key to the client.
  • The server sends the types of certificate it will accept.   This is typically RSA, DSA, and EC
  • The server sends down the Elliptic Curve names it will accept, if present  – I dont think it is used on the java client
  • The server uses the jdk.certpath.disabledAlgorithm to filter the list of Signature Algorithms, and sends this filtered list to the client.
  • The server extracts the CAs and self signed certificates from the trust store and sends them down to the client.
  • The server sends “ServerHelloDone”, saying over to you to respond.

Client processing:

  • Checks the server’s certificate is valid, including
    • Checks the public certificate of the servers CA chain is allowed according to the client’s jdk.certpath.disabledAlgorithm..  So jdk.certpath.disabledAlgorithm =…, SHA256withECDSA would not allow a  server’s certificate with Signature algorithm:SHA256withECDSA .
  • The client takes the list of certificate types ( RSA, DSA, EC), and CAs and iterates over the keystore and selects the records where
    • the certificate type in the list
    • the signature algorithm is in the list
    • the certificate signed by one of the CA’s in the list
  • Displays this list for the end user to select from.  It looks like the most recently added certificate is first in the list.
  • The client sends “Finished” and  the selected certificate to the server

Server processing

  • The server checks the certificate and any imbeded CA certificates from the client matches.
    • Checks the signature algorithm
    • Checks the constraints, for example RSA keySize < 2048
    • Checks the certificate and CA are valid

End of handshake.

This is a useful link for describing the java.security parameters.

This specification  describes the handshake, with the “ClientHello” etc.

Using java -Djavax.net.debug=… to examine data flows, including TLS

Jan 2021. This page gets many hits. Please can you tell me if the information below is what you were looking for, or if you were looking for more information, such as understanding the trace flow, and finding problems.   Please send me an email or post a comment below with information on what you were looking for, and I’ll see what I can do.

There are different levels you can get from the -Djavax.net.debug option.  You can display its options using -Djavax.net.debug=help .

Ive added a section on -Djava.security.debug=… which looks at keystore access.

Ive also added a examples of what each trace option give you.

I’ve also put up a post which describes the trace, and has an annotated output (from both ends), showing common problems and possible solutions.

With -Djavax.net.debug you get

all            turn on all debugging
ssl            turn on ssl debugging

The following can be used with ssl:

    record       enable per-record tracing
    handshake    print each handshake message
    keygen       print key generation data
    session      print session activity
    defaultctx   print default SSL initialization
    sslctx       print SSLContext tracing
    sessioncache print session cache tracing
    keymanager   print key manager tracing
    trustmanager print trust manager tracing
    pluggability print pluggability tracing

    handshake debugging can be widened with:
    data         hex dump of each handshake message
    verbose      verbose handshake message printing

    record debugging can be widened with:
    plaintext    hex dump of record plaintext
    packet       print raw SSL/TLS packets

and your program exits.

You can use

-Djavax.net.debug=ssl:record or -Djavax.net.debug=ssl:handshake

To display specific levels of detail.

To specify more than one option use -Djavax.net.debug=ssl:keymanager:record

-Djava.security.debug=

Out of all the options below, I found certpath,provider provided the most useful information for seeing what was happening with certificates and the handshake.  For example

certpath: X509CertSelector.match(SN: 1a
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Subject: CN=SERVER, OU=SSS, O=ZZZZ)
certpath: X509CertSelector.match: subject DNs don’t match
certpath: ForwardBuilder.getMatchingCACerts: found 0 CA certs
certpath: SunCertPathBuilder.depthFirstSearchForward(): certs.size=0
certpath: PKIXCertPathValidator.engineValidate()…
certpath: AdaptableX509CertSelector.match: subject key IDs don’t match. Expected: [] Cert’s: []
certpath: NO – don’t try this trustedCert
certpath: X509CertSelector.match(SN: 0
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP)
certpath: X509CertSelector.match returning: true
certpath: YES – try this trustedCert
certpath: anchor.getTrustedCert().getSubjectX500Principal() = CN=TEMP4Certification Authority, OU=TEST, O=TEMP
certpath: AlgorithmChecker.contains: SHA256withRSA
Provider: MessageDigest.SHA-256 algorithm from: SUN

I grepped with pkcs12: and got

pkcs12: Loading PKCS#7 encryptedData (PBEWithSHA1AndRC2_40 iterations: 2048)
pkcs12: Loading PKCS#7 data
pkcs12: Checking keystore integrity (HmacPBESHA1 iterations: 2048)
pkcs12: PKCS12KeyStore load: private key count: 1. secret key count: 0. certificate count: 1
pkcs12: Retrieved a protected private key at alias ‘ss’ (PBEWithSHA1AndDESede iterations: 2048)
pkcs12: Retrieved a 1-certificate chain at alias ‘ss’

The options…


all              turn on all debugging - this produces millions of lines of
                 output for one handshake! 
access           print all checkPermission results 
certpath        PKIX CertPathBuilder and CertPathValidator debugging 
combiner         SubjectDomainCombiner debugging 
gssloginconfig   GSS LoginConfigImpl debugging 
configfile       JAAS ConfigFile loading 
configparser     JAAS ConfigFile parsing 
jar              jar verification 
logincontext     login context results 
jca              JCA engine class debugging 
policy           loading and granting 
provider        security provider debugging 
pkcs11           PKCS11 session manager debugging 
pkcs11keystore   PKCS11 KeyStore debugging 
sunpkcs11        SunPKCS11 provider debugging 
scl              permissions SecureClassLoader assigns 
ts               timestamping 

The following can be used with access: 

stack            include stack trace 
domain           dump all domains in context 
failure          before throwing exception, dump stack and domain that 
                 didn't have permission 
thread           include the thread name 

The following can be used with stack and domain: 

permission=<classname> 
                only dump output if specified permission is being checked 
codebase=<URL>  only dump output if specified codebase  is being checked 
permname=<name> only dump output if the specified name matches the 
                permission getName() 
permactions=<actions>  only dump output if the specified actions 
                matches the permission getActions() 

The following can be used with provider: 

engine=<engines> only dump output for the specified list of JCA engines.  
                Supported values: 
                   Cipher, KeyAgreement, KeyGenerator, 
                   KeyPairGenerator, KeyStore, Mac, 
                   MessageDigest, SecureRandom, Signature. 

Note: Separate multiple options with a comma  

-Djavax.net.debug=ssl:record gave

records doing read and write
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 294
SSLSocketInputRecord.java:214|READ: TLSv1.2 handshake, length = 2948

-Djavax.net.debug=ssl:handshake gave

All of the sections.

-Djavax.net.debug=ssl:handshake:verbose gave

the same as ssl:handshake +

Signature algorithm, ed448, is not supported by the underlying providers
SignatureScheme.java:373|Ignore unsupported signature scheme: ed25519
SignatureScheme.java:373|Ignore unsupported signature scheme: ed448
SignatureScheme.java:393|Ignore disabled signature scheme: rsa_md5

-Djavax.net.debug=ssl:keygen gave

No records.

-Djavax.net.debug=ssl:session gave just

Session initialized:  Session(1611922238363|TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)

-Djavax.net.debug=ssl:defaultctx gave

No records.

-Djavax.net.debug=ssl:sslctx gave

javax.net.ssl|DEBUG|01|main|2021-01-29 12:29:48.847 GMT|X509TrustManagerImpl.java:79|adding as trusted certificates (
  "certificate" : {
      "subject"            : "CN=DANCA1, OU=DAN, O=DAN",
...
  "certificate" : {
    "subject"            : "CN=SS, O=SSS, C=GB",
... 
  "certificate" : {
    "subject"            : "CN=ecec, O=cpwebuser, C=GB",
...
System property jdk.tls.client.cipherSuites is set to 'null'
System property jdk.tls.server.cipherSuites is set to 'null'
jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
Ignore unsupported cipher suite: TLS_AES_256_GCM_SHA384
found key for : ecec (
  "certificate" : {
       "subject"            : "CN=ecec, O=cpwebuser, C=GB",
 
    "subject"            : "CN=SSCA256, OU=CA, O=SSS, C=GB",
  ...
found key for : mqweb (
  "certificate" : {...   
trigger seeding of SecureRandom
done seeding of SecureRandom
...
Produced ClientHello handshake message (
"ClientHello": {
...
Consuming ECDH ServerKeyExchange handshake message (
"ECDH ServerKeyExchange": {
  "parameters": {
    "named group": "secp256r1"
    "ecdh public": {
      0000: 04 5F 30 8E 64 45 55 C0   
      0010: 55 23 1B 69 CA FF 78 8D  
      0020: B5 59 72 BA B4 95 8A 4C  
      0030: 2B 37 55 3D F2 A5 3D 71  
      0040: 34                                               
    },
  },

-Djavax.net.debug=ssl:sessioncache gave

No records.

-Djavax.net.debug=ssl:keymanager gave

found key for : ecec (
“certificate” : {
“version”            : “v3”,

found key for : mqweb (
“certificate” : {

-Djavax.net.debug=ssl:trustmanager gave

adding as trusted certificates (
“certificate” : {

“subject” : “CN=mqweb8, O=cpwebuser, C=GB”,

“certificate” : {

“subject” : “CN=DANCA1, OU=DAN, O=DAN”,

Found trusted certificate (
“certificate” : {

“issuer” : “CN=DANCA1, OU=DAN, O=DAN”,

“subject” : “CN=DANCA1, OU=DAN, O=DAN”,

ss private or public key is not of EC algorithm

Where adding as …  is what the local trust store found, and found trusted is what as sent from the server as part of the certificateRequest.

-Djavax.net.debug=ssl:pluggability

No records.