How to stop weak people from accessing your web server (liberty) and mqweb

In the ongoing security battle arena, standards of protection are improving.  What may have been an acceptable cipher suite last year may be less acceptable today.  For many years a RSA certificate was considered the best.  Now certificates using Elliptic Curves are considered superior as they require smaller keys for the same level of protection, are harder to break, and use less CPU when they are used.

Google wants to have digital certificates renewed every year rather than the two years it tends to be today.  (From what I have read Certificate Authorities certificates can be around for years).

Your security team should be updating the user certificates regularly, so this should not be your problem.  You need to review your servers and do regular “housekeeping” to bring them inline with current best practice.  It is easy to forget a service which just sits there and runs, and so is a weak entry point to your environment.

The house keeping tasks:

  1. Ensuring your Certificate Authority certificates use current options: type of certificate, key size etc.
  2. Ensure the server’s certificate uses current options
  3. You restrict what certificates can be used to connect to the server.  For example do not allow RSA certificate with RSA with a small key size.

Renewing certificates

In some enterprises you have one certificate which identifies you, and then have controls within each server which manage which users are allowed to connect.  In other enterprises you may have a general certificate, plus a certificate for “monitoring” which has been signed by the “monitoring CA”.  This makes it easier to control access – if your certificate does not have the “monitoring CA” you do not get into the server.

If you are an enterprise you are likely to have a process which automatically renews individual’s certificates.  This is likely to be “go on-site, plug into the ethernet, ( so people cannot intercept your wireless), get  a new certificate”.   It is harder to renew your server’s certificates.

You will need steps along the lines of

  • Strengthen your CA if required
    • Generate a new, stronger, CA certificate;   for example using Elliptic Curves instead of RSA
    • Every machine needs to install the new CA certificate in the trust stores/browsers, alongside the old CA certificate.  You need a period when both CA certificates are available to allow for a transition period, as you need to change both ends of the TLS connection.
    • You can now deploy new individual certificates with their CA, to your users which go into their key store. Users connecting to a server with the new certificates will be validated with the  server’s new CA. Users which have certificates with the old CA will continue to use the old CA.
  • You need a new certificate for your web server.  If this is using the new CA, then all end users will need to have the new CA in their trust store, so the web server certificate can be validated.  If you are not using a new CA, then the existing CA can be used with no change.
  • Once all the users have their new certificate, you can remove the old CA from the server and from the user’s trust stores/browsers.  If a user tries to use the old certificate, if will fail to verify as the CA is no longer present.

Restricting what certificates and cipher suites can be used.

You can configure java security to restrict the certificates and cipher suites are used.  For example enforce no RSA certificate, or RSA certificate with a key sizes < 2048 cannot connect to the web server.  You configure java.security.  See here.

(Not) Using TLS 1.3

The TLS standards are being improved, for example TLS 1.3 protocol has improved algorithms, which are faster and more secure than in TLS 1.2.  The handshake has been improved, and weak algorithms have been dropped.   Most browsers already support TLS v1.3.

In theory this is a simple matter of changing the protocol parameter from TLSv1.2 to TLSv1.3 and it will all magically work.  This support is in java release 11.  Unfortunately the java used by mqweb is not at this level (it it java version 8) so cannot support it.   This means you have to make the changes yourself.

Change it, pray, answer the phone, and backout the change.

In an ideal world your server would produce a report of which certificates were used, with the Distinguished Name(the owner), what algorithms are used, and what CAs are used.  You make a note of all those which need attention, get the certificates upgraded, and when all the certificates are good – you can enable the stronger functions and it will all work first time.

I have not been able to get this list from my servers.

What I expect will happen as you enable the stronger functions, is that people who are not ready, will stop working, and will report problems.   You turn off the stronger functions, fix the machines which had problems and repeat until the phone stops ringing.
Some changes, for example changing java -D…. options in the jvm.options will require a server restart.  Other changes within the mqwebuser.xml, may get picked up periodically (and so avoids the need to restart the server), but you may need to restart the server.  On my laptop it takes over 10 seconds to stop and restart the server.

This upgrade process is very disruptive and does not provide a highly available server your enterprise needs.  Think how many change requests you will need to submit before it all works!  The best solution is to start with a very secure web server.

Using java -Djavax.net.debug=… to examine data flows, including TLS

Jan 2021. This page gets many hits. Please can you tell me if the information below is what you were looking for, or if you were looking for more information, such as understanding the trace flow, and finding problems.   Please send me an email or post a comment below with information on what you were looking for, and I’ll see what I can do.

There are different levels you can get from the -Djavax.net.debug option.  You can display its options using -Djavax.net.debug=help .

Ive added a section on -Djava.security.debug=… which looks at keystore access.

Ive also added a examples of what each trace option give you.

I’ve also put up a post which describes the trace, and has an annotated output (from both ends), showing common problems and possible solutions.

With -Djavax.net.debug you get

all turn on all debugging
ssl turn on ssl debugging

The following can be used with ssl:
  record       enable per-record tracing
  handshake    print each handshake message
  keygen       print key generation data
  session      print session activity
  defaultctx   print default SSL initialization
  sslctx       print SSLContext tracing
  sessioncache print session cache tracing
  keymanager   print key manager tracing
  trustmanager print trust manager tracing
  pluggability print pluggability tracing

handshake debugging can be widened with:
  data         hex dump of each handshake message
  verbose      verbose handshake message printing

record debugging can be widened with:
  plaintext    hex dump of record plaintext
  packet       print raw SSL/TLS packets

and your program exits.

To display specific levels of detail you can use

-Djavax.net.debug=ssl:record or -Djavax.net.debug=ssl:handshake

To specify more than one option use

-Djavax.net.debug=ssl:keymanager:record

-Djava.security.debug=

Out of all the options below, I found certpath,provider provided the most useful information for seeing what was happening with certificates and the handshake.  For example

certpath: X509CertSelector.match(SN: 1a
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Subject: CN=SERVER, OU=SSS, O=ZZZZ)
certpath: X509CertSelector.match: subject DNs don’t match
certpath: ForwardBuilder.getMatchingCACerts: found 0 CA certs
certpath: SunCertPathBuilder.depthFirstSearchForward(): certs.size=0
certpath: PKIXCertPathValidator.engineValidate()…
certpath: AdaptableX509CertSelector.match: subject key IDs don’t match. Expected: [] Cert’s: []
certpath: NO – don’t try this trustedCert
certpath: X509CertSelector.match(SN: 0
Issuer: CN=TEMP4Certification Authority, OU=TEST, O=TEMP
Subject: CN=TEMP4Certification Authority, OU=TEST, O=TEMP)
certpath: X509CertSelector.match returning: true
certpath: YES – try this trustedCert
certpath: anchor.getTrustedCert().getSubjectX500Principal() = CN=TEMP4Certification Authority, OU=TEST, O=TEMP
certpath: AlgorithmChecker.contains: SHA256withRSA
Provider: MessageDigest.SHA-256 algorithm from: SUN

I grepped with pkcs12: and got

pkcs12: Loading PKCS#7 encryptedData (PBEWithSHA1AndRC2_40 iterations: 2048)
pkcs12: Loading PKCS#7 data
pkcs12: Checking keystore integrity (HmacPBESHA1 iterations: 2048)
pkcs12: PKCS12KeyStore load: private key count: 1. secret key count: 0. certificate count: 1
pkcs12: Retrieved a protected private key at alias ‘ss’ (PBEWithSHA1AndDESede iterations: 2048)
pkcs12: Retrieved a 1-certificate chain at alias ‘ss’

The options…

all              turn on all debugging - this produces millions of lines of
                 output for one handshake! 
access           print all checkPermission results 
certpath        PKIX CertPathBuilder and CertPathValidator debugging 
combiner         SubjectDomainCombiner debugging 
gssloginconfig   GSS LoginConfigImpl debugging 
configfile       JAAS ConfigFile loading 
configparser     JAAS ConfigFile parsing 
jar              jar verification 
logincontext     login context results 
jca              JCA engine class debugging 
policy           loading and granting 
provider        security provider debugging 
pkcs11           PKCS11 session manager debugging 
pkcs11keystore   PKCS11 KeyStore debugging 
sunpkcs11        SunPKCS11 provider debugging 
scl              permissions SecureClassLoader assigns 
ts               timestamping

The following can be used with access:

stack            include stack trace 
domain           dump all domains in context 
failure          before throwing exception, dump stack and domain that 
                 didn't have permission 
thread           include the thread name 

The following can be used with stack and domain:

permission=<classname> 
                only dump output if specified permission is being checked 
codebase=<URL>  only dump output if specified codebase  is being checked 
permname=<name> only dump output if the specified name matches the 
                permission getName() 
permactions=<actions>  only dump output if the specified actions 
                matches the permission getActions() 

The following can be used with provider: 

engine=<engines> only dump output for the specified list of JCA engines.  
                Supported values: 
                   Cipher, KeyAgreement, KeyGenerator, 
                   KeyPairGenerator, KeyStore, Mac, 
                   MessageDigest, SecureRandom, Signature. 

Note: Separate multiple options with a comma

-Djavax.net.debug=ssl:record

gave

records doing read and write
...
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 294
SSLSocketInputRecord.java:214|READ: TLSv1.2 handshake, length = 2948
...

-Djavax.net.debug=ssl:handshake

gave all of the sections.

-Djavax.net.debug=ssl:handshake:verbose

gave the same as ssl:handshake +

Signature algorithm, ed448, is not supported by the underlying providers
SignatureScheme.java:373|Ignore unsupported signature scheme: ed25519
SignatureScheme.java:373|Ignore unsupported signature scheme: ed448
SignatureScheme.java:393|Ignore disabled signature scheme: rsa_md5

-Djavax.net.debug=ssl:keygen

gave: No records.

-Djavax.net.debug=ssl:session

gave just

Session initialized:  Session(1611922238363|TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)

Djavax.net.debug=ssl:defaultctx

gave No records.

-Djavax.net.debug=ssl:sslctx

gave

javax.net.ssl|DEBUG|01|main|2021-01-29 12:29:48.847 GMT|X509TrustManagerImpl.java:79|adding as trusted certificates (
  "certificate" : {
      "subject"            : "CN=DANCA1, OU=DAN, O=DAN",
...
  "certificate" : {
    "subject"            : "CN=SS, O=SSS, C=GB",
... 
  "certificate" : {
    "subject"            : "CN=ecec, O=cpwebuser, C=GB",
...
System property jdk.tls.client.cipherSuites is set to 'null'
System property jdk.tls.server.cipherSuites is set to 'null'
jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
Ignore unsupported cipher suite: TLS_AES_256_GCM_SHA384
found key for : ecec (
  "certificate" : {
       "subject"            : "CN=ecec, O=cpwebuser, C=GB",
 
    "subject"            : "CN=SSCA256, OU=CA, O=SSS, C=GB",
  ...
found key for : mqweb (
  "certificate" : {...   
trigger seeding of SecureRandom
done seeding of SecureRandom
...
Produced ClientHello handshake message (
"ClientHello": {
...
Consuming ECDH ServerKeyExchange handshake message (
"ECDH ServerKeyExchange": {
  "parameters": {
    "named group": "secp256r1"
    "ecdh public": {
      0000: 04 5F 30 8E 64 45 55 C0   
      0010: 55 23 1B 69 CA FF 78 8D  
      0020: B5 59 72 BA B4 95 8A 4C  
      0030: 2B 37 55 3D F2 A5 3D 71  
      0040: 34                                               
    },
  },

-Djavax.net.debug=ssl:sessioncache

gave: No records.

-Djavax.net.debug=ssl:keymanager

gave

found key for : ecec (<br>"certificate" : {
"version": "v3",
...

found key for : mqweb (
"certificate" : {
...

-Djavax.net.debug=ssl:trustmanager

gave

adding as trusted certificates</strong> (
"certificate" : {
...
"subject" : "CN=mqweb8, O=cpwebuser, C=GB",
...
"certificate" : {
...
subject" : "CN=DANCA1, OU=DAN, O=DAN",
...
Found trusted certificate</strong> (
"certificate" : {
...
"issuer" : "CN=DANCA1, OU=DAN, O=DAN",
...
"subject" : "CN=DANCA1, OU=DAN, O=DAN",
...
ss private or public key is not of EC algorithm
<p>Where <em>adding as</em> ...&nbsp; is what the local trust store found, and <em>found trusted </em> is what as sent from the server as part of the certificateRequest.</p>

Where adding as… ; is what the local trust store found, and Found trusted certificate is what as sent from the server as part of the certificateRequest.

-Djavax.net.debug=ssl:pluggability

No records.

How do I create a certificate with Elliptic Curve (or RSA)

Why would I want to use Elliptic Curve?

Some ciphers are considered stronger than others.  For example certificates with Elliptic Curve algorithms are now considered better than using the well known RSA.    They are more secure and use less resources.  Over time certificates with Elliptic Curves may become the norm.  See here.

If you change to use a different algorithm you need to make sure that both ends of the TLS connection support it.   If a cipher spec beginning with TLS_ECDHE is the only cipher spec available, it may not work with certificates with RSA.

When you create a certificate you first create the private key, and then make the public certificate.  You can sometimes combine this into one operation.

April 2021 – I had added some information on using strkmqikr, runmqakm and runmqckm not working.

To make a private key using Elliptic Curve

Use

openssl genpkey -out $name.key.pem -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -aes256 -pass file:password.file

where

  • $name – I create the certificate in a shell script.  As the name of the certificate is used in many places – it is best to use a shell variable to hold the short certificate name.
  • -algorithm EC says this is an Elliptic Curve
  •  P-256  is the Elliptic Curve definition to use.   This is a popular key;  it has a key length of 256.  It is also known as prime256v1.
  • -aes256 -pass file:password.file says encrypt the private key using the aes 256 cipher spec (there are others available) – and use the password in the file. You need this when doing working with private key and public certificate, for example creating the certificate request.  If you do not specify -aes256 (or equivilant)  etc the private key is not encrypted, and so could be used if stolen.   This is not used during TLS handshakes.

Or (the old syntax )

openssl ecparam -name prime256v1 -genkey -noout -out $name.key.pem …

You then create the request and get the request signed (this is common to all requests)

name=”eccert”
openssl req -config xxx.config -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy  -md sha256 -cert ca2.pem -keyfile ca2.key.pem -out $name.pem -in $name.csr  -extensions clientServer

The command openssl x509 -in eccert.pem -text -noout|less displays the certificate and gives

Subject Public Key Info:
  Public Key Algorithm: id-ecPublicKey
    Public-Key: (256 bit)
     pub:
       04:...
       ce:60:63:03:84
     ASN1 OID: prime256v1
     NIST CURVE: P-256

During the TLS handshake, this can be processed by CipherSpecs TLS_EC*, such as TLS_ECDH… and TLS_ECDHE…

If you use openssl ecparam -name secp521r1  this gives Public Key Algorithm: id-ecPublicKey Public-Key: (521 bit)

To make a private key using RSA

Use

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out $name.key.pem -aes256 -pass pass:password

Or (the old syntax)

openssl genrsa -out $name.key.pem 4096 -aes256 -pass pass:password

where

  • rsa_keygen_bits:4096  – is the size of the key to use.
  • -aes256 -pass pass:password says encrypt the private key  using the aes 256 cipher spec (there are others available) – the password is password.  You need this when doing working with private key and public certificate.  This is not used during TLS handshakes.

You make the request and get it signed (the statements below are the same as for the EC certificate)

name=”rsa”
ca=”ca2″
openssl req -config xxx.config -new -key $name.key.pem -out $name.csr -outform PEM -subj “/C=GB/O=cpwebuser/CN=”$name -passin file:password.file -passout file:password.file
openssl ca -config openssl-ca-user.cnf -policy signing_policy  -md sha256 -cert $ca.pem -keyfile $c2.key.pem -out $name.pem -in $name.csr  -extensions clientServer

The command openssl x509 -in rsa.pem -text -noout|less displays the certificate and gives

Subject Public Key Info:
  Public Key Algorithm: rsaEncryption
   RSA Public-Key: (4096 bit)
    Modulus:
      00:d0:88:d2:d0:86:34:82:bb:1a:7b:a0:6d:37:fd:
      ... 
     1e:3d:31
    Exponent: 65537 (0x10001)

During the TLS handshake, this can be processed by CipherSpecs TLS_RSA*.

Changing the Signature Algorithm:

With a java program, you can use java.security to limit which Security Algorithms are allowed during the handshake, and so prevent certificates from being used.

As part of the TLS handshake there is a conversation about the encryption of the certificate.  For example listing an RSA certificate gives

Signature Algorithm: sha256WithRSAEncryption

You can change this by using

openssl ca … -md sha384

This gives

Signature Algorithm: sha384WithRSAEncryption

For an Elliptic Curve certificate this was

Signature Algorithm: ecdsa-with-SHA256  with the default -md (sha256) or Signature Algorithm: ecdsa-with-SHA384 ( when -md sha384 is specified)

Storing the certificate

I used a script to generate my certificate.  In this script I had

  • openssl x509 -in $name.pem -text -noout|less to display the certificate, and check the options
  • openssl pkcs12 -export -inkey $name.key.pem -in $name.pem -out $name.p12 -CAfile ca256.pem -chain -name $name -passout file:password.file -passin file:password.file to create the *.p12 file with the certificate and CA chain, so it can be used by java, and curl etc
  • certutil -D $sql -n $name remove the certificate from the Chrome browser keystore.  Where sql=”-d sql:/home/colinpaice/snap/chromium/current/.pki/nssdb”
  • pk12util -i $name.p12 $sql -W password to add the  .p12 created above into the Chromium keystore (along with its CA chain)

Using the certificate

For my java programs I used the certificate keystore sssks.p12 with -Djavax.net.ssl.keyStore=/home/colinpaice/ssl/ssl2/sssks.p12 -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.keyStoreType=pkcs12 or for mqwebuser.xml  <keyStore id=”defaultKeyStore” location=”/home/colinpaice/ssl/sssks.p12″ type=”pkcs12″ password=”password”/>

How to use strmqikm, runmqakm and runmqckm.

I had a question about doing this with cms repositories used by MQ.

For a long time I could not see how to create an certificate using Elliptic curve. It was well hidden (inplain sight) but I eventually found it.

Using runmqakm

If you specify the signing algorithm (-sig_alg) with (you can use either value)

  • SHA224WithECDSA | EC_ecdsa_with_SHA224
  • SHA256WithECDSA | EC_ecdsa_with_SHA256
  • SHA384WithECDSA | EC_ecdsa_with_SHA384
  • SHA512WithECDSA | EC_ecdsa_with_SHA512

then the certificate uses an Elliptic Curve. For example

runmqckm -cert -create -sig_alg EC_ecdsa_with_SHA224 -db zzclient.kdb -dn “CN=EC,O=EC” -label EC2 -stashed

then display it using

runmqakm -cert -details -db zzclient.kdb -label EC -stashed

It gave

Public Key Type : EC_ecPublicKey (1.2.840.10045.2.1), Parameters: namedCurve: EC_NamedCurve_secp256r1 (1.2.840.10045.3.1.7)
Signature Algorithm : EC_ecdsa_with_SHA224 (1.2.840.10045.4.3.1)

You can specify -size 224|256|384|512. To get the appropriate EC size. This appears in the name, for example -size 384 gave EC_NamedCurve_secp384r1.

The signing algorithm also has a size, for example EC_ecdsa_with_SHA224 uses a key size of 224.

Using strmqikr

With strmqikm, select the Signature Algorithm first from the pull down, such as SHA512WithECDA, then select the key size. The list of key size values changes depending on the Signature Algorithm.

Runmqckm does not work

Note runmqckm does not support Elliptic curves. Use runmqakm instead.

Should I specify which cipher suites my web browser should use – or the opposite?

I was investigating how to upgrade the certificate used by my mqweb server from RSA to the new, improved, Elliptic Curve, and wondered how to make it most secure.
At first I thought the answer to the question was yes, then I changed my mind to no, then I changed to yes, and now I think you should do something else!

The short answer is you should not specify which cipher specs to use, but you may considering saying which ones not to use by overriding java.security features.

SSL and TLS Deployment Best Practices covers many good topics.

The first part of the TLS handshake

  • The client sends a list of the cipher suites it supports to the server
  • The serve has its own list (which you can influence)
  • The server takes each cipher suite in turn from the client list, and selects the first one which is in the server’s list and matches the server’s certificate
  • If the server is using an RSA certificate, then cipher suites with TLS_…RSA_WITH… are used. With Elliptic Curve certificates then TLS_….ECDSA_WITH… are used.

When using Chrome to talk to my java server the certificate suites sent up were

  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, strong
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 strong
  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 very strong
  • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384  strong
  • SSL_RSA_WITH_3DES_EDE_CBC_SHA weak

These go from strong to weak.  Certificates with 3DES or _SHA are considered too weak to use.

My java server had the following cipher suites

  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  • TLS_RSA_WITH_AES_256_CBC_SHA256
  • TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
  • TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
  • TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
  • TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
  • ….

And the final cipher suite chosen was

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 – top of the list from the client and 29th in the list in the server – not the strongest certificate in the list which was a surprise.

Specifying a server cipher suite.

You can specify the list of cipher suites used by the server using the java property -Djdk.tls.server.cipherSuites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,…
So yes;  you can specify a list of cipher specs, and the order, and so put the “strongest first”.

I then asked myself is the default one TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 any better than the  “best one” TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384?

Bulk data encryption.

Looking on the internet, there is a general feeling that there is no significant difference between AES_128 and AES_256.  If quantum computers become a reality, then there may be benefits of AES_256, but then the whole of the encryption world will have changed.

AES…GCM is better than AES…CBC.  The difference between them is not so clear.  GCM is considered better than CBC, it can run on pipeline processors, but may use more CPU overall.  Both have weaknesses, but different weaknesses.

Hashing algorithm

I looked at the hashing algorithm, again there was not much difference between SHA256, SHA 384 and SHA512.
I did some evaluation on the use of SHA256 and SHA512 on my laptop using the command openssl speed sha256 sha512. This hashes different sized buffers and calculates bytes processed per second.

type    16 bytes   64 bytes  256 bytes 1024 bytes 16384 bytes
sha256 70348.86k 158677.42k 297724.76k 370884.61k 400878.25k
sha512 49354.44k 195907.78k 341598.21k 507424.77k 597409.79k

So we can see with very small buffers SHA 256 could do 70 MB/second, and SHA512 could only do 49 MB/second, but with bigger buffers SHA256 did 40MB/second, and SHA512 did 59 MB/second

It looks like there is no significant difference between them.

Note: when they upgraded SHA256 to SHA512 they improved the algorithm as well as the number of iterations. This applies to other algorithms as well.

Overall  the default TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 is as good as the “best one” TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, and so there was no advantage in specifying a cipher suite.

Other advantages of not specifiying a cipher suite

Last year you may have specified the then best cipher suite.   This year there are new, better, cipher suites which are the default.  If you specified the cipher suite you will use last year’s suite.

When your web server supports TLSv1.3 ( dependent on java 11) it will use a different set of cipher suites.  By not specifying a cipher suite name, the migration will be easier.  Today browsers already support TLS v1.3 and send up a mixture of cipher suites for TLSv1.2 and TLS v1.3 as part of the TLS handshake.

Do not specify the cipher suite – specify what you do not want.

You can specify security information to java using a java.security file.  See here.  You can override this file using a system parameter like -Djava.security.properties=/home/colinpaice/eclipse-workspace-C/sslJava/bin/serverdisabled.properties.

In this file you can specify

jdk.tls.disabledAlgorithms=…
jdk.certpath.disabledAlgorithms= …

You specify what do you not want.

The defaults are

  • jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
  • jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024,EC keySize < 224, 3DES_EDE_CBC

You could specify RSA keySize < 2048, so any cipher suites and certificate using an RSA key size of under 2048 would not be allowed.

Whoops my certificate has expired – what do I need to do?

The amount of work depends on what has expired.

  • If the root CA has expired then you need to reissue all certificate which include the root CA, and update trust stores
  • If the enterprise CA has expired you need to renew all certificates signed by the enterprise CA, and update trust stores with the new certificate
  • If your personal certificate has expired, you need to renew that certificate.

The amount of work also depends on the size of your enterprise.

I’ll list the work need if the certificate is going to expire next week – as this is more work than if it has already expired

If your personal certificate is going to expire

You need to

  • renew it, create a new private key and certificate request
  • or extend the date of the existing certificate  – by re-signing it
  • update the keystores which include it.  This could be those used by java programs ( .p12, or .jks) as well as any browser keystore (.nssdb)

To renew your personal certificate

You can just recreate it.  If you are considering to use a stronger algorithms, such as Elliptic Curves, bear in mind that the servers need to be able to support what you specify.

Depending on how your openssl has been configured, if you have unique_subject = yes you may need to use openssl revoke… to remove the old personal certificate from the configuration.   This is to ensure there is only one certificate with the same distinguished name, and online checking of certificate validity (Certificate Revocation Lists and Online Status Certificate Protocol) can be used.  If you have unique_subject = no, you can recreate the certificate without revoking it.

  • openssl genpkey  to generate a new private key
  • openssl req to create a request

You do not need to recreate it – you can just sign it again.

You then need to do normal process of update the users of the certificate.

  • openssl ca to sign the request
  • openssl pkcs12 -export to create the .p12 file
  • You may want to deploy it only when the machine is attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • certutil -D $sql -n $shortName to remove the certificate from the browser’s keystore
  • pk12util -i $shortName.p12 $sql -W password to add the certificate to the browser’s keystore
  • /opt/mqm/bin/runmqckm -cert -delete  to remove a certificate from a keystore (.jks)
  • /opt/mqm/bin/runmqckm -cert -import to add a certificate to a keystore(.jks)
  • restart the servers to pick up the new certificate

Rather than deleting the certificate from the keystore and adding the new one with the same short name,  you may want to make a copy of the old certificate from the keystore, or simply backup the whole keystore – as it should not change once frequently.  If there are problems, restore from the backup.

To renew your enterprise certificate

This requires a lot more work than updating a personal certificate.

Just like the personal certificate you can recreate it, or just sign it again.

You need to phase in the new certificate as it may take days or weeks to deploy it successfully.

The client’s trust store needs the CA certificate matching the CA used by the server’s certificate.

If you have two servers, and change servers to use the new CA certificate, the client’s trust store content will change over time.

  • The old CA certificate – before any work is done
  • Both old and new CA certificates during the migration period.  The server using the old CA will use the old CA in the client.   The server which has been migrated to use the new CA will use the new CA in the client.
  • Just the new CA certificate.  Once the migration has finished. The old certificate can be removed when all of the servers have been migrated, or the certificate has expired.

A key store and a trust store can have certificate with the same distinguished names, but with a different short name or alias name.  My enterprise is called SSS, and the CA for my enterprise is CASSS… .  My trust store has

  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2016 and
  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2020 and

When you maintain a keystore you use the short name, for my example CASSS2016 or CASSS2020.

During the TLS handshake, when the server’s certificate is sent to the client, it is checked against all of the trusted CA certificates in the keystore, so having two certificates with the same DN does not matter.

The steps to deploy your new CA are

  • Get the new CA.  Either create a new CA using the latest algorithms and cipher suites (the better solution), or have the old certificate resigned.
  • For each user and server
    • Deploy the CA certificate into the trust stores with a new name – eg CASSS2020
  • For each user and server
    • Renew or recreate each personal and server certificate and sign it with the new CA
    • Deploy it securely to each user and server and replace the old personal certificate
    • You may want to deploy the private key only when attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • For the server create java.security rules to disable weaker algorithms and cipher specs.   It is easy to undo this change.
    • This should identify any user not using the latest cipher suites and certificates.  These rules can be relaxed while problems are fixed.
  • For each user and server
    • Make a copy of the old CA certificate prior to deletion
    • Delete the old CA certificate from the trust stores eg CASSS
    • After a validation period delete the copy of the old certificate

Start all over again!

You can see how much work you need to do when you renew your enterprise CA, so make sure you  renew it in plenty of time – months rather than days.

mqweb – what’s the difference between MQWebAdmin* and MQWebUser roles?

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.

Thanks to Gwydion for this enlightenment.

mqweb – who did what to what, when, and how long did it take?

You can provide an audit trail of the http requests coming into your server.  This is described in the base liberty document, and works in mqweb.

Within the httpEndoint tag you can add

<httpEndpoint host=”${httpHost}” httpPort=”${httpPort}” httpsPort=”${httpsPort}” id=”defaultHttpEndpoint”>
<httpOptions removeServerHeader=”false”/>
<accessLogging enabled=”true” filePath=”${server.output.dir}/logs/http_access.log”
logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U X:%{X}W r:”%r” s:%s’
maxFileSize=20

maxFiles=0
/>
</httpEndpoint>

See here for information on the accessLogging,  here for the syntax of the <accessLogging…> and here for the logFormat format options.

From the logFormat page

%a
Remote IP address
%A
Local IP address
%b
Response size in bytes excluding headers
%B
Response size in bytes excluding headers. 0 is printed instead of – if no value is found.
%{CookieName}C
The request cookie specified within the brackets, or if the brackets are not included, prints all of the request cookies.
%D
The elapsed time of the request – millisecond accuracy, microsecond precision
%h
Remote host
%{HeaderName}i
HeaderName header value from the request
%m
Request method
%{HeaderName}o
HeaderName header value from the response
%q
Output the query string with any password escaped
%r
First line of the request
%{R}W
Service time of the request from the moment the request is received until the first set of bytes of the response is sent – millisecond accuracy, microsecond precision
%s
Status code of the response
%t
NCSA format of the start time of the request
%(t)W
The current time when the message to the access log is queued to be logged in normal NCSA format
%u
Remote user according to the WebSphere Application Server specific $WSRU header
%U
URL Path, not including the query string
%{X}W
Cross Component Tracing (XCT) Context ID

So

logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U  r:”%r” s:%s’

gave me

  • a:127.0.0.1 remote IP address
  • A:127.0.0.1 local host
  • D:225960 duration in millisecond
  • h:127.0.0.1 remote host
  • HeaderHost:127.0.0.1:9443 the Host header
  • HeaderOrigin:- the Origin header (missing in this case)
  • m:GET request method
  • R:186930 Service time of the request from the moment the request is received until the first set of bytes of the response is sent
  • t:[27/Feb/2020:17:04:27 +0000] NCSA format of the start time of the request
  • u:colinpaice remote user
  • U:/ibmmq/console url path
  • r:”GET /ibmmq/console HTTP/1.1″ First line of the request
  • s:302 Status code of the response

There are other log formatting options available, I picked those I thought were most useful.

Note when using the MQConsole from a browser, the interface is chatty. I had 20 request to refresh one window.

Other ways of formatting the data

  • I separated each field with a ‘,’ and could read it into a spread sheet.
  • You could configure your log format string to produce the output in JSON format, to make it easier to post process.

Is mqweb secure enough? Depends on what you mean by secure enough.

When doing research into the MQ Console and REST interface I was some times surprised at how “insecure” the mqweb was, but with a bit more thought it seemed more reasonable.

One of the questions I had was, is authentication with certificates better than using userid and password?  See here.

What do I mean by secure?

You can buy a safe with a combination lock to secure your valuables.    It will not be 100% secure.   It will have a statement like “This safe will provide protection from fire for up to 1 hour”.  The number of combinations is limited, and so it will take someone some hours to try them all.   They may be lucky and guess it first go!  An insecure safe would have one combination wheel.  The more wheels, the longer it should take to find the combination.

Digital certificates are considered a good way of providing “secure” communication.  However if you are careless with your private information, then other people have easy access to your secrets.  The private information for some encryption types is based on multiplying two very large prime numbers together.  The theory being it will take you a long time to factorize  the product .  If I was a government, I would take the “book of prime numbers” and go through each number, and multiply it be each other number in the book, and generate a reverse dictionary of factor1 * factor 2 -> public key.    You need a lot of computing power as the numbers are very large, but computing power is cheap, and you only need to do it once.  Take the public key, and look up the factors.  Of course it is not that simple, you might need a dictionary for each encryption type, as well as a lot of CPU power.  These certificates give some security, but are not totally secure.

You can have locks on your house, but if you leave the doors unlocked – it is easy to break in.  People often lock the front door, but leave the back door or a window open.

Security is down to how long it will take someone to access the information.  All components have to be secure; you cannot be “half secure”.

What do I want to secure?

This is where it starts to get murky.  Some of the HTTP protocol protection depends on “state change”, so changing a field in a database is a state change, creating an object is a state change, but browsing a record is not a state change.

With the MQ REST API an HTTP GET maps to a browse;  an HTTP POST is a change or put;  and an HTTP DELETE is a destructive get.  With the standard HTTP protection POST and DELETE have more protection than GET.

I consider being able to browse a queue which has financial transaction information, as insecure.  You can obtain bank account details, which could be used in a different sort of attack.   In mqweb, the protection for browse is weaker than for POST and DELETE.

Surprise number 1 – which certificate should the browser use?

My first surprise when using digital certificates for identification was how the browser decides which certificate to use.

I said here

If client authentication is used, the certificates are read from the server’s trust store. The DN of the self signed certificates and the DN of the CA certificates are extracted and sent down to the browser.

The browser takes this list and checks through the “user” certificates in its key store, for those self signed certificates with a matching DN, or those certificates which have been signed by the CA.

The list of matching certificates is displayed for the user to select from.

From then on, communication to that https://url:port will use that certificate.  If you create a new tab within the browser instance, and work with the same site, the selected certificate will be used – without prompting.

You can only change the certificate by restarting the browser and selecting a different certificate.  You cannot logoff.

Surprise number 2 – it is easy to browse the queue with no additional security.

In an HTML page,  I used the html link <a href=”https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message “> direct link https ></a> .

If I had already accessed the server then this displayed the content of the next message on the queue, with no further security checks.

If I opened another tab within the browser, displayed the page and clicked on the link, I also got the next message content displayed.

I used a file on my laptop with the link in it – and it displayed the message content.  It was easy to create a web page in an editor and so display the content of the queue.

The CSRF checks are not applicable – as these only occur when the request is a change of state, POST, DELETE etc.

Using Curl, and changing the GET request to a DELETE I received (as expected)

{“error”: [{
  “action”: “Resubmit the request with the correct CSRF header.”,
  “completionCode”: 0,
  “explanation”: “The REST API request cannot be completed because the CSRF header was omitted.”,
  “message”: “MQWB0100E: The CSRF header ‘ibm-mq-rest-csrf-token’ was omitted from the request.”,
  “msgId”: “MQWB0100E”,
  “reasonCode”: 0,
  “type”: “rest”
 }]}’

This is not a Cross Origin request – as it is in one page – so the CORS checks are not performed.   Putting the backend request into a java script did cause the CORS checking as expected.

Turning off client Certificate Authentication helped

I changed the <ssl…  clientAuthenticationSupported=”true”> to “false” to use userid and password.

With my <a href=”https://localhost:9445/ib…  > ” > direct link https ></a>  I got

{"error": [{
  "action": "Provide credentials using a client certificate, LTPA security token, or username 
   and password via HTTP basic authentication header. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   check the messages.log file for messages indicating that SAF authentication is not available. 
   Start the Liberty angel process if it is not already running. 
   You might need to restart the mqweb server for any changes to take effect.",
  "completionCode": 0,
  "explanation": "The REST API request cannot be completed because credentials were omitted from the request. 
   On z/OS, if the mqweb server has been configured for SAF authentication, 
   this can be caused by the Liberty angel process not being active.",
  "message": "MQWB0104E: The REST API request to 
  'https://localhost:9445/ibmmq/rest/v1/messaging/qmgr/QMA/queue/DEEPQ/message' is not authenticated.",
  "msgId": "MQWB0104E",
  "reasonCode": 0,
  "type": "rest"
}]}

I tried putting my userid and password in the URL as in
<a href=”https://colinpaice:mypassword@localhost:9445/ib…> “> direct link https ></a> . This gave me the same message.  In particular the message says Provide credentials using …  or username and password via HTTP basic authentication header.  So it needs credentials in  a header not inline.

As raw html cannot insert HTTP headers, it means this is more secure.   To insert headers you need a script, and then the CORS processing will capture it.

I used the logon url https://127.0.0.1:9445/ibmmq/console/login.html this prompted me for userid and password, then displayed the console.

So is using userid and password the right thing to use?

Yes and No.

  1. Yes – you know when you are accessing the back end, but then it logs you off, and you have to logon again after ltpaExpiration minutes.
  2. With long running displays, such as the monitors in the operations room – you do not want to be logging on every hour.  In this case certificates are the right answer.

See is it better to use certificate authentication or userid and password? for information.

 

Making it more secure.

I have shown that if people are authorised to process messages on a queue as part of their normal job, they could easily write a web page to browse the queue.

If the queue was access as part of a client  program accessing the queue over a channel, the end user could write or obtain a program to process the queue – but this would be harder than just writing a web page.    This is like having a combination safe with one wheel, and having multiple wheels – you need a lot more effort.

You cannot secure mqweb for message access to a subset of queues – it is all or nothing.

I would disable the end users access to message data – and just use the rest API for administration.

This means in the mqwebuser.xml the <security-role name=”MQWebUser”> … </security-role> has no users who are allowed to process sensitive queues.  You need to prevent them using curl and other tools, so

<enterpriseApplication id=”com.ibm.mq.rest”> … and <enterpriseApplication id=”com.ibm.mq.console”>…. should have no users in <security-role name=”MQWebUser”>… </security-role>

Ban the special subject.

I would also explicitly specify the groups of users – and not use the <special-subject type=”ALL_AUTHENTICATED_USERS”/>.

Best practice says not to specify user names in the security roles, but to put the user names in groups and specify the groups.   It make it much easier to manage.  For example the manager in charge of the Payroll team, can administer the PayrollGroup, and the MQ Admin team does not need to edit the mqwebuser.xml file.

mqweb- is it better to use certificate authentication or userid and password?

Like may questions the answer is it depends.

You can set up your mqweb environment

  1. Certificates are required – you cannot use userid and passwords
  2. If a valid certificate is found use that, else prompt for userid and password
  3. Do not use certificates – just use userid and passwords.

This is described in the Liberty documentation.

It says for the <ssl…> tag

  1. If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  2. If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  3. If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

No matter how you logon, the web server creates a LTPA* cookie with encrypted information about the logon.   The mqwebuser.xml attribute ltpaExpiration says how long this cookie is valid for.  If the time is exceeded you have to logon again, and it generates a new cookie.

Logging on

Certificate authorisation

If you are using certificates, the server sends down a list of valid self signed certificates or CA certificates, and the browser compares this list with the certificates in its store.   If the certificate in the browser’s store is acceptable, it is added to a list

The list is displayed, and the end user selects a certificate.  For subsequent requests with the same  http://url:port combination, in the same or a different tab within the browser window, the same certificate is used, and the browser does not prompt.

If the ltpa cookie expires then the same certificate is used, invisibly to the end user.  A new cookie is generate and sent down.

If there are no valid certificates in the list, then depending clientAuthentication, it may prompt for a userid and password.

You cannot logoff from the page as there is no option to do so.

Userid and password.

When userid and password are passed up in a header (not in the URL as some web servers accept) the ltpa token is returned.  When the ltpa token expires, the user is prompted to re-logon.   They need to enter the userid and password again.  You can also logoff from the web page.

What are the implications of certificates and userid with password

Certificate logon

  • The web browser can logon, and although the ltpa token expires, it is automatically logged on again.   This is useful for a “big screen monitor” in the operations room, where you want the display active all day.
  • Someone can create a file with some HTML in it to browse a queue  (if authorised).  So if they are authorised to the queue for their normal job, they could easily create a web page to display the queue contents.  This could be the end user, or an evil person saying “click this link”.

userid logon

  • If the user is using the mqconsole web pages, then periodically they will get logged off – and they will have to logon again.
  • If they try to access a queue from an HTML page, they will be asked for userid and password.  Depending on what the end user is doing, this may give them notification than there is some unusual activity.  If they are hacking they will enter the userid and password anyway.
  • The person may have access to the same information without using the mqweb console interface, so there is no change to the threat level.

What do I recommend?

Tricky.

  • I think long running monitors need to use certificates
  • End users should logon with userid and password, with an expiry period of about an hour or less.
  • You may want to have a Certificate  Authority just for the mqweb browsers, to stop your corporate certificates from being used to access the mqweb server.

Because it is easy to create an html page and browse a queue, I would allow administration over mqrest, and not allow processing messages over REST.

As I said in another posting

  • The admin API authorisation is managed through <security-role name=”MQWebAdmin”> and <security-role name=”MQWebAdminRO”> sections in the mqwebuser.xml file.
  • The messaging API authorisation is managed through <security-role name=”MQWebUser”> sections.

To stop messaging over the rest interface, have no entries in the <security-role name=”MQWebUser”> section.  This applies to both certificates, and userid and password.

(Personally I would have called<security-role name=”MQWebUser”> as <security-role name=”MQWebMessaging”>  to make the roles clearer. 

Update from Gwydion…

The MQWebAdmin, MQWebAdminRO and MQWebUser roles can all be used for the admin REST API. That’s why MQWebUser is not called MQWebMessaging – it’s not just for messaging. The difference between them is the user ID that’s checked by the qmgr for operations performed via the REST API.
  • Operations performed by users in the MQWebAdmin* roles take place under the context of the mqweb server user ID.
  • Operation performed by users in the MQWebUser role take place under the context of the user logged into the REST API.
To stop messaging over the REST API completely, the mqRestMessagingEnabled should be set to false in the mqweb server configuration.

What does mqRestCorsMaxAgeInSeconds in mqweb mean?

Ive blogged about CORS, and how this allows you to list sites that are permitted to use scripts to send request to the mqweb server.

I struggled with understanding what value mqRestCorsMaxAgeInSeconds has, as it did not behave as expected (this was due to a bug).

If you have a CORS transaction there is an OPTIONS request, followed by the actual DELETE/GET/POST requests.
The OPTIONS request checks that the request is coming from an authorised list of URLs, and that the parameters are valid.  The OPTIONS information can be cached.

If the check is successful then the real request can be issued.  If the requests occur in a short time, then you can have OPTIONS, DELETE, DELETE, DELETE, using the cached values.  If there is a “long” time interval between the requests you may get OPTIONS, DELETE, gap, OPTIONS, DELETE.

The OPTIONS information can be cached for up to the minimum of the mqweb mqRestCorsMaxAgeInSeconds  time, and the browser time.

For Chrome the maximum time interval is 600 seconds.  If no time is specified in the OPTIONS response, then 5 seconds is used.

There is a bug in the Liberty base code which sends down the header Access-Control-Allow-Max-Age: …, when the browser is expecting Access-Control-Max-Age.   Because of this, the default time of 5 seconds is used in Chrome.

This should not have a major impact.  For those applications using scripts to send multiple REST API request, there will be more frequent OPTIONS requests – every 5 seconds instead of up to 600 seconds.  These extra flowes are invisible to the scripts and the end user.

What value should I use?

Chrome has a maximum of 600 seconds, with a default of 5 seconds.

Firefox has a maximum of 24 hours (86400 seconds).

Setting it to 600 seconds sounds reasonable to me.

Making changes to mqRestCorsMaxAgeInSeconds

If you change mqRestCorsMaxAgeInSeconds you have to restart the mqweb server.

I do not get caching!

When researching this topic I found every GET request had an OPTIONS request, rather than the OPTIONS, GET, GET.   A quick search on the internet showed me the Chrome option ( F1 -> Settings and preferences) “Disable Cache ( while DevTOOLS is open)” was enabled. I deselected this, and I got the caching.