Are you going crazy with Chrome giving fatal: certificate_unknown, bad_certificate NET::ERR_CERT_INVALID? Me too!

I came back to using Chrome with my MQ Web browser. It was working last week, but yesterday and today it stopped working. In debugging it, I’ve learned even more ways of checking TLS handshakes to see why they fail!

Using Wireshark packet trace on Linux, and TLS trace in the web server, I could see the Client Hello, Server hello, worked; but the response was Chrome giving NET::ERR_CERT_INVALID, and the traces showing Alert Level: Fatal, Description: Certificate Unknown.

  • I had been through the checks to make sure the CA was in the client key store.
  • I double checked, and tripled checked to make sure it was the right CA.
  • I exported the z/OS CA certificate, downloaded it and imported it.
  • I displayed the z/OS version, and the Chrome keystore’s version and they matched ( ok – the not-after and not-before times were different due to different time zones).
  • I shut every thing down and restarted it the next day.
  • I felt like screaming “AHH -it worked last week it works on FireFox – it should all work on Chrome”

After a day workingb in the garden, I had time to consider a different approach.

How I traced the problem down.

From your browser you can display the “problem” certificate by clicking on the icon in front of the URL. You get into “certificate viewer”. There is a “General” display which shows you useful information about the certificate. There is also the “Details”. At the bottom of the Details is a button labelled “Export”. Click it and export the certificate to a file such as SERVER.PEM.

You can now use openssl on this. For example

openssl x509 -in ~/Downloads/SERVER.cert.bad -text -noout|less

This shows you all the details of the certificate, so you can check them again!

You can also export the CA you think is being used, from the browser keystore, eg CA.pem

You can use the openssl verify command to do its validation of the server certificate and the CA certificate.

openssl verify -CAfile ca.pem -show_chain ~/Downloads/SERVER.cert

worked, it gave

OK
Chain:
depth=0: O = ZZZZ, OU = SSS, CN = SERVER (untrusted)
depth=1: O = TEMP, OU = TEST, CN = TEMP4Certification Authority

Which shows the server certificate, and the CA certificate were OK. However the same command with -x509_strict gave

openssl verify -CAfile ca.pem -show_chain -x509_strict ~/Downloads/SERVER.cert

gave

O = ZZZZ, OU = SSS, CN = SERVER
error 24 at 0 depth lookup: invalid CA certificate
error /home/colinpaice/Downloads/SERVER.cert.bad: verification failed

I felt I was on the trail of the problem.

Depth 0 is the server certificate. Depth 1 is the signers certificate, Depth 2 is the next level up. It was strange that it said the server certificate was an invalid CA.

Looking in the openssl source openssl/crypto/x509/x509_vfy.c showed several lines giving the error code X509_V_ERR_INVALID_CALL.

It turns out that the server’s certificate had been configured to to “certsign” (I had mis-copied a line in the certificates definition from an earlier test). Certsign means the certificate is a CA because it can sign things – but the CA flag was not set in the certificate – so clearly it was a bad_certificate. The validation failed – as it failed the consistency check it reported unknown certificate, and I was getting very frustrated.

What keyusage is required?

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works.

Just do not use KEYUSAGE(CERTSIGN) .

Recreate the certificate, and add it to the key store, and use

f CSQ9WEB,refresh,keystore

to get the MQ web server to pick up the change to the keystore. You may need to restart your web browser.

I think this is a useful technique which I will use in the future when I stumble over the next TLS set up problem.

How to do an SSL/TLS trace in Firefox

I was trying to configure a firefox session to my MQ Web server, but it kept failing. I found a useful way of getting the TLS trace from firefox.

My web site was 10.1.1.2:9443

The command ssltap -s 10.1.1.2:9443 sets up a proxy. It listens on a port (default 1924). In firefox use https://localhost:1924/ibmmq/console/ 1>aa 2>bb. This then talks to the proxy which then talks to the backend. Not perfect but OK.

Hints and tips for using WordPress.

I’ve been using WordPress for a couple of years now.  On the whole it has been pretty easy, but sometime I struggled, until I found a work around.  I’ll document some of these.

How do I find my document?

I have many blog posts and found it hard to find the one I wanted.  You can display all your posts using Posts -> All Posts.

  • Use the Dates pull down, select a month, and press filter.  This shows me the documents I’ve created on in the selected month.
  • Use the Categories pull down, select one, and press filter.  This shows the matching documents.  You specify the categories when you make the post.
  • Tab across, enter a search argument and press the Search Posts button.

How do I find my drafts ?

On the dashboard page -> Quick draft has recent drafts and view all drafts.

Creating a post

You can use the “classic” or the “block” mode.  I prefer classic for small posts as I find it easier to use and uses less key strokes.

With a classic block you can enter headings, data, tables etc.  With block mode, you create a heading block, or a text block or a pre formatted block, or a native html block. A block may be one sentence.

How to edit the html in classic mode

If you go to the top of the document it will display two toolbars, use the “⋮” in the smaller toolbar.  Select “Edit as html”.   Go down to your text… make your changes, and select the “⋮” in the small box toolbar which appears as you move the cursor.  With big documents this gets impractical, so using block mode may be better.

You can be in block mode, but create small blocks of classic mode. I do this when I want to change attributes, for example in the style sheet example below, I could only add style=”line-height :90%; ” to the <pre… >, if I used classic mode. If you do not use classic mode, it removes the “failing” code. It gets a bit messy, but you get used to it.

Pasting text into pre formatted blocks.

You can create a pre formatted block.  This is basically data within a <pre>…</pre> set of html tags.

If you paste into a block, it removes multiple spaces, and so all your nicely formatted text comes out all wrong.  The solution is to go into html mode, find the <pre>…</pre> and paste your text.   This will retain multiple spaces.

I find the data displayed in a pre formatted block is too large.  I use

<pre><small>….  my data </small></pre> to make the text smaller, see below for an example.

Using tables

You can use the blocks mode “table” to define a table, then edit it as raw html to do tweaks, such as change the width of columns as in the following example.

<table>
  <thd>
    <th width=”30%“>header1</th>
    <th>header2</th>
  </thd>
  <tr>
    <td>data1</td>
    <td>data<2/td>
  </tr>
</table>

Using Style Sheets

Here is a link to information on CSS and style sheets.  You can search for attributes such as line-height.

I have not found a way of including a style sheet in a page.  If you edit the html you can use  inline styles for example

<span style=”color:red;”>…</span>,

or use CSS for example selectors like

  • <pre class=”tls”>…</pre> with CSS class definition .tls {}
  • <pre id=”special>…</pre> with CSS id definition #special{}
  • <pre>…</pre> with CSS tag definition  pre {}
  • or combinations <pre class=”tls”>…/pre> with CSS pre.tls {} meaning <pre> tags and class = tls.

then specify the style via WordPress home page -> Appearance-> Customize -> Additional CSS.  This means what you put in this section is available to all your posts and pages. 

I have

pre.tls, div.tls {
line-height: 1;
color: darkgreen;
}

This says any pre html tags with class=”tls”, or a div tag with a class=”tls” set the line height to 1, and make the color darkgreen.

My style sheet includes

/* reduce the vertical space of lists */
.compact {
line-height: 90%;
}
/* make the summary line look link a link, by underlining it and changing
the colour. Also change the cursor to a hand
*/
details > summary {
background-color: #ddd;
border: none;
text-decoration: underline;
cursor: pointer;
}
/* Make <b> bolder within <pre class="tls" */
pre.tls b {
font-weight: 900;
color: black;
}

 Note: the above pre formatted text has style=”line-height : 90%” in the <pre > tag, and the <small>…</small> tags have been used.

What are my top posts?

Dashboard -> Top pages and posts.   This gives Today, Yesterday; tab across to Summaries,  Select from  7 Days| 30 Days| Quarter| Year| All Time

How to rearrange the dashboard.

You can drag and drop with widgets

The widgets within dashboard have “˄, ˅” at the top.   This moves the widget up or down. (Up means up the right hand side to the bottom of the left, then up the left hand side to the top)

  • ⏶ close widget
  • ⏷ open widget

What are the short cuts?

I found this document which has a good list.

Adding pictures

You can add pictures via your media. Use the Add Media icon in the toolbar (Alt shift M), select from your list, and click on “Insert” .  The are strict controls as to the types of data you can upload.

Why do they ship java products on z/OS with the handbrake on? And how to take the brake off.

I noticed that it takes seconds to start MQ on my little z/OS machine, but minutes (feels like days) to start anything with Liberty Web server.  This include the MQWEB, z/OSMF,  and Z/OSConnect.  I mentioned this to an IBM colleague who asked if I was using Java Shared classes.  These get loaded into z/OS shared pages.

When I implemented it, my Liberty server came up in half the time!

I found this blog post which was very helpful, and showed me where to look for more information.  I subsequently found this document (from 2006!)

The kinder garden overview of how Java works.

  • You start with a program written in the Java language.
  • When you run this, Java converts it into byte codes
  • These byte codes get converted to native instructions  – so a byte code “push onto the stack” may become 8  390 assembler instructions.
  • This code can be optimised, for example code which is executed frequently can have the assembler instructions rewritten to go faster.  It might put code inline instead of out in a subroutine.
  • If you are using Java shared classes, this code can be written out and reused by other applications, or if you restart the server, it can reused what it created before.  Reusing the shared classes means that programs benefit because the byte codes have already been converted into native code, and optimisations have been done on the hot code.

What happens on z/OS?

By default, z/OS writes the code to virtual memory and does not save anything to disk.  If you restart your Java application within the same IPL, it can exploit the shared classes which have been converted to native code, and optimised – great- good design.   I found the second time I started the web server it took half the time.  However I IPL once a day, and start my web server once a day. I do not benefit from having it start faster a second time – as I only started it once per session. By default when you re-ipl, the shared classes code is discarded, and so next time you need the code, it has to be to convert to native instructions again, and it loses any optimisation which had been done.

What is the solution?

It is two easy steps:!

  1. Tell Java to write the information from memory to disk – to take a snaphot.
  2. After IPL tell Java to load memory from the disk image – to restore a snapshot.

It is as simple as that.

Background.

It is all to do with the java -Xshareclasses.

With your application you tell Java where to store information about the shared classed.  It defaults to Cache=/tmp/ name=javasharedresources.

In my jvm.options I overrode the defaults and specified

-Xshareclasses:nonFatal 
-Xshareclasses:groupAccess
-Xshareclasses:cacheDirPerm=0777
-Xshareclasses:cacheDir=/tmp,name=mqweb

If you give each application a name (such as mqweb)  you can isolate the cache to an application and not disrupt another JVM if you change the cache.  For example if you restore from a snapshot, only users of that “name” will be affected.

List what is in the cache

You can use the USS command,

java -Xshareclasses:cacheDir=/tmp/,listAllCaches

I used a batch job to do the same thing.

//IBMJAVA  JOB  1 
// SET V='listAllCaches' 
// SET C='/tmp/' 
//S1       EXEC PGM=BPXBATCH,REGION=0M, 
// PARM='SH java -Xshareclasses:cacheDir=&C,&V' 
//STDERR   DD   SYSOUT=* 
//STDOUT   DD   SYSOUT=*            

The output below, shows the cache name is mqweb.  Once you have created a snapshot it has an entry for it.

Listing all caches in cacheDir /tmp/                                                                          
                                                                                                              
Cache name       level         cache-type      feature         OS shmid       OS semid 
mqweb            Java8 64-bit  non-persistent  cr              8197           4101 

For MQWEB the default parameters are -Xshareclasses:cacheDir=/u/mqweb/servers/.classCache,name=liberty-%u” where /u/mqweb is the WLP parameter, where my parameter are defined, and %u is the userid the server is running under, so in my case liberty=START1.

When I had /u/mqweb/servers/.classCache, then the total command line was too long for BPXBATCH.   (Putting it into STDPARM gave me IEC020I 001-4 on the instream STDPARM because the resolved line wa greater than 80 characters.   I resolved this by adding -Xshareclasses:cacheDir=/u/mqweb,name=cache to the jvm.options file.

To take a snapshot


//IBMJAVA  JOB  1 
// SET C='/tmp/' 
// SET N='mqweb' 
// SET V='restoreFromSnapshot' 
// SET V='listAllCaches'
// SET V='snapshotCache' //S1 EXEC PGM=BPXBATCH,REGION=0M, // PARM='SH java -Xshareclasses:cacheDir=&C,name=&N,&V' //STDERR DD SYSOUT=* //STDOUT DD SYSOUT=* //

This job took a few seconds to run.

I believe you have to take the snapshot while your java application is executing – but I do not know for definite.

Restore a snapshot

To restore a snapshot just use restoreFromSnapshot in the above JCL. This took a few seconds to run. 

How to use it.

If you put the restoreFromSnaphot JCL at the start of the web server, it will preload it whenever you use your server.

If you take a snapshot every day before shutting down your server, you will get a copy with the latest optimisations.  If you do not take a new snapshot it continues to use the old one.

If you want to not use the shared cache you can get rid of it using the command destroySnapshot.

Is my cache big enough?

If you use the printStats request you get information like

Current statistics for cache "mqweb":                                                
...                                                                                     
cache size                           = 104857040                                     
softmx bytes                         = 104857040                                     
free bytes                           = 70294788 
...
Cache is 32% full                                     
                                                      
Cache is accessible to current user = true                                                 

The documentation says

When you specify -Xshareclasses without any parameters and without specifying either the -Xscmx or -XX:SharedCacheHardLimit options, a shared classes cache is created with a default size, as follows:

  • For 64-bit platforms, the default size is 300 MB, with a “soft” maximum limit for the initial size of the cache (-Xscmx) of 64MB, …

I had specified -Xscmx100m  which matches the value reported.

What is in the cache?

You can use the printAllStats command.  This displays information like

Classpath

1: 0x00000200259F279C CLASSPATH
/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs/jclSC180/vm.jar
/usr/lpp/java/J8.0_64/lib/se-service.jar
/usr/lpp/java/J8.0_64/lib/math.jar

Methods for a class
  • 0x00000200259F24A4 ROMCLASS: java/util/HashMap at 0x000002001FF7AEB8.
  • ROMMETHOD: size Signature: ()I Address: 0x000002001FF7BA88
  • ROMMETHOD: put Signature: (Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; Address: 0x000002001FF7BC50

This shows

  • there is a class HashMap. 
  • It has a method size() with no parameters returning an Int.  It is at…. in memory
  • There is another method put(Object o1, Object o2)  returning an Object.  It is at … in memory
Other stuff

There are sections with JITHINTS and other performance related data.

 

Getting SSL/TLS to work on MQ on z/OS

After I succeeded in getting TLS 1.3 to run on MQ  midrange 9.2, I thought I would try it on z/OS.  I had not used TLS on z/OS for about 10 years, so it was almost like coming to the topic with very rusty knowledge.

I searched the Knowledge centre and found no relevant hits – lots of hits which were not relevant.  I eventually found an SSL related keyword, and this got me to the topic  Working with SSL/TLS on z/OS.   I think this is well documented.  It covered all the things I had to do.

The remained of this post covers the bits not covered by the documentation.

Define SSLTASKS.

You need to define SSLTASKS to be able to use TLS on z/OS.  See the comments here. I used

%CSQ9 ALTER QMGR SSLTASKS(5)

You need to restart the CHINIT if you change the value of SSLTASKS.

Set up the keyring for the queue manager. 

See here.  This post show how to create the keyring and import a CA from z/OS, and import a CA  from a Linux system.

If you alter the keyring or certlabl you just need a refresh security type(SSL) command to pick up the changes.

Defining the channel

I tried to define the channel, as this failed for security reasons, I’ve given the RACF setup I had to do.

In this section I defined the specific commands for example DEFINE.CHANNEL.   I could have defined DEFINE.* to allow all define commands.

I used a channel called TLS, and define the resource CSQ9.CHANNEL.TLS* to allow my ID to define TLS, TLS1 etc

The command %CSQ9 DEF CHL(TLS) CHLTYPE(SVRCONN) gave me

ICH408I USER(CSQOPR ) GROUP(SYS1 ) NAME(COLIN PAICE ) 167
CSQ9.DEFINE.CHANNEL CL(MQCMDS )
WARNING: INSUFFICIENT AUTHORITY – TEMPORARY ACCESS ALLOWED
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

 /* RDELETE MQCMDS CSQ9.DEFINE.CHANNEL
RDEF MQCMDS CSQ9.DEFINE.CHANNEL UACC(NONE)
PERMIT CSQ9.DEFINE.CHANNEL CLASS(MQCMDS ) –
        ID(COLIN,IBMUSER) ACCESS(ALTER )

I also set up CSQ9.DELETE.CHANNEL and CSQ9.ALTER.CHANNEL in a similar way, so my userid could maintain the channels.

I refreshed MQ security %CSQ9 refresh security to pick up the changes.

I reissued the command %CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN) and got

ICH408I USER(COLIN ) GROUP(SYS1 ) NAME(COLIN PAICE )
CSQ9.CHANNEL.TLS CL(MQADMIN )
PROFILE NOT FOUND – REQUIRED FOR AUTHORITY CHECKING
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

  /* RDELETE MQADMIN CSQ9.CHANNEL.TLS*
RDEF MQADMIN CSQ9.CHANNEL.TLS* UACC(NONE) WARNING
PERMIT CSQ9.CHANNEL.TLS* CLASS(MQADMIN) –    
        ID(COLIN,IBMUSER) ACCESS(ALTER )
SETROPTS RACLIST(MQADMIN) REFRESH

I issued the commands

%CSQ9 refresh security
%CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN)

And successfully defined the channel.

I changed the cipher spec.

I selected a cipher spec from the list.

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ECDHE_RSA_AES_128_CBC_SHA256)

When I started the channel I got

CSQX631E … CSQXRESP Cipher specifications differ,  channel TLS local=ECDHE_RSA_AES_128_CBC_SHA256 remote=TLS_RSA_WITH_AES_256_GCM_SHA384
connection 10.1.0.2 (10.1.0.2)

This was clear;  I love clear messages.

I decided to change the z/OS end

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384 )

and the client connected successfully.

With MQ 9.2 I could (and did) change this to

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ANY_TLS12)

and the client worked successfully.  The ANY_TLS12.  provides a wide list of supported cipher specifications, includes TLS_RSA_WITH_AES_256_GCM_SHA384  and ECDHE_RSA_AES_128_CBC_SHA256. 

When I am ready to support TLS 1.3 I will use ANY_TLS12_OR_HIGHER and ANY_TLS13_OR_HIGHER.

Connect a client to it!

I had had my client connect to a midrange queue manager, so I had working client environment.  See here for the journey.

I created a .json file for the CCDT connection to z/OS.  I specified

{ "channel":
  [{
    "name": "TLS",
    "clientConnection":
    {
      "connection":
      [{
        "host": "10.1.1.2",
        "port": 1414
       }],
      "queueManager": "CSQ9"
    },
    "transmissionSecurity":
    {
      "cipherSpecification": "ANY_TLS12",
      "certificateLabel": "rsaca256_client",
      "certificatePeerName": ""
    },
    "type": "clientConnection"
  }]
}

When it connected I got messages

+CSQX511I %CSQ9 CSQXRESP Channel TLS started connection 10.1.0.2
ICH408I USER(COLINPAI) GROUP( ) NAME(??? )
LOGON/JOB INITIATION – USER AT TERMINAL NOT RACF-DEFINED
IRR012I VERIFICATION FAILED. USER PROFILE NOT FOUND.
+CSQX512I %CSQ9 CSQXRESP Channel TLS no longer active connection 10.1.0.2

COLINPAI came from the userid on the Linux machine (colinpaice) upper cased and truncated. This id will be flowed and used as the MCAUSER if you don’t set it to anything else, using CHLAUTH for example  (Thanks to Morag for this information).

Enable chlauth

To be able to map from the DN in a certificate to a z/OS userid you have to use MQ CHLAUTH.  See  Mapping a client user ID to an MCAUSER user ID.

Check it is enabled at the queue manager level and enable it it needed.

%CSQ9 DIS QMGR CHLAUTH
%CSQ9 ALTER QMGR CHLAUTH(ENABLED)

Define a mapping from certificate to userid

I used

//S1 EXEC PGM=CSQUTIL,PARM='CSQ9' 
//STEPLIB  DD DSN=COLIN.MQ921.SCSQLOAD,DISP=SHR 
//         DD DSN=COLIN.MQ921.SCSQANLE,DISP=SHR 
//SYSPRINT DD SYSOUT=* 
//SYSIN   DD * 
 COMMAND DDNAME(COMMAND) 
//COMMAND DD * 
 SET CHLAUTH('TLS') + 
     TYPE(SSLPEERMAP) SSLPEER('O="cpwebuser"') + 
     ACTION(REPLACE)   + 
     MCAUSER(ADCDD ) CHCKCLNT(ASQMGR) 
/* 

This says for channel TLS,  take the Organisation(O=..)  from the certificate, and if it is cpwebuser then set the ID to ADCDD.

Check it works

Once the channel had started I used

%CSQ( DIS CHS(TLS)
it displayed the following, where I have removed lines which are not relevant to TLS and added some comments

  • CHSTATUS(TLS)
  • CHLTYPE(SVRCONN)
  • SECPROT(TLSV12) – this is the level of the protocol
  • SSLCERTI(CN=SSCARSA1024,OU=CA,O=SSS,C=GB)- this is the DN of the issuer of the SSLPEER certificate (below)
  • SSLCERTU(START1) – the IBM documentation says “The local user ID associated with the remote certificate.”  I dont know where this comes from.. how to change it, or where it is used.
  • SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384) – The negotiated cipher spec
  • SSLRKEYS(0) -The number of successful TLS key resets.
  • SSLKEYTI() -The time on which the previous successful TLS secret key was reset.  The secret key has not been reset
  • SSLKEYDA() -The date on which the previous successful TLS secret key was reset.  The secret key has not been reset
  • SSLPEER(SERIALNUMBER=01:90,CN=rsaca256,O=cpwebuser,C=GB, UNSTRUCTUREDNAME=openssl_ca_user_cnf.keyAgreement2, UNSTRUCTUREDNAME=localhost, UNSTRUCTUREDADDRESS=127.0.0.1) . This is information from the certificate at the remote end.
  • MCAUSER(ADCDD) – This is the userid (set by the CHLAUTH above) used by this channel.
  • LOCLADDR(10.1.1.2(1414)) – This is the address the connection came in from.  This value will be different it you have different IP stacks and different listener ports.

Taking the brakes off ZFS on z/OS – move it to OMVS

From z/OS 2.2 there is a performance advantage in running the ZFS file system as part of OMVS, rather than its own address space.  The IBM documentation says When running zFS in the OMVS address space, each file system vnode operation (such as creating a directory entry, removing a directory entry, or reading from a file) will have better overall performance. Each operation will take the same amount of time while inside zFS itself. The performance benefit occurs because z/OS UNIX can call zFS for each operation in a more efficient manner.  This will be relevant when you application is doing a lot of file IO – for example using a web server.

This move is not documented – but it is really easy!  It is mentioned here. Instructions are hidden in the installation instructions here.

Before I started

The IBM doc says You can determine if zFS is in its own address space by issuing D OMVS,PFS. If the output shows an ASNAME value, zFS is running as a colony address space.


OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION                                            
 PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 ...   
  ZFS       IOEFSCM    ZFS       LOCAL     A     2021/02/17 17.35.06 

The steps I took…

  1. I added KERNELSTACKS(ABOVE) to USER.Z24A.PARMLIB(BPXPRM00).
  2. Being ultra cautious I re-ipled.
  3. The documentation talks about putting IOEZPRM DD in OMVS, then goes on to say As the preferred alternative to the IOEZPRM DDNAME specification, delete the IOEZPRM DDNAME and use the IOEPRMxx parmlib member.  So I did not change the OMVS proc.  When I reipled it worked and I got the message IOEZ00374I No IOEZPRM DD specified in OMVS proc. Parmlib search being used. 
  4. I edited USER.Z24A.PARMLIB(BPXPRM00) and removed the ASNAME in  FILESYSTYPE TYPE(ZFS) ENTRYPOINT(IOEFSCM)ASNAME(ZFS) Well I actually made a copy of the original line and put it between /* and */, then deleted the text.
  5. I reipled.

Afterwards

The  D OMVS,PFS command now gives  N/A instead of the Address Space Name

OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION                                         
 PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
...
  ZFS       IOEFSCM    N/A       LOCAL     A     2021/02/17 17.55.47  

Easy!

The hardest part was making sure I had an IPLable SARES1 in case I got it wrong!

Issuing commands…

I used to issue commands like f zfs,query,all. Now that the ZFS address space does not exist, you need to use  f omvs,pfs=zfs,query,all.

Setting up the MQ keyring on z/OS

I wanted to connect some clients to my z/OS queue manager over a TLS channel.  This post describes how I set up the z/OS keyring with the certificates.

Define the keyring.

The CSQ9 CHINIT runs with a userid of START1, so I defined a keyring belonging to that id.

I set up a dataset  called IBM.MQCSQ9.KEYRING to keep all of my JCL in for the CSQ9 queue manager.  This makes it easier to clone the definitions for another queue manager.

The definitions create the keyring, and add the z/OS CA certificate (CERTAUTH ADCD_CA) to it.

//IBMRACF JOB 1,MSGCLASS=H 
//* Use JCL for the RACF definitions
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
/*RACDCERT DELRING(MQRING) ID(START1)
RACDCERT ADDRING(MQRING) ID(START1)
RACDCERT ID(START1 ) -
   CONNECT(RING(MQRING) LABEL('ADCD-CA') CERTAUTH)
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Define the queue manager’s certificate.   This uses an Elliptic curve with key size of 256.

 /*RACDCERT ID(START1) DELETE(LABEL('CSQ9CERT')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
   SIZE(256) - 
   NISTECC - 
   SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
ALTNAME(IP(10.1.1.9)) - NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT')
RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL)) RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

It need ALTNAME(IP(10.1.1.2)) ( or similar) because some browsers check this name, with the IP address of the server.

 

Configure the queue manager

%CSQ9 ALTER QMGR SSLKEYR(START1/MQRING) CERTLABL(CSQ9CERT)

Note if you use mixed case keyring you need to put the value in quotes.

Export the CA certificate from z/OS and sent to the client machine

RACDCERT CERTAUTH EXPORT(LABEL('ADCD-CA'))- 
DSN('IBMUSER.CERT.ADCDCA.PEM') -
FORMAT(CERTB64) -
PASSWORD('password')

The data set  IBMUSER.CERT.ADCDCA.PEM  contained text, and the first line is —–BEGIN CERTIFICATE—– .   Send this file to the client machine, for example using FTP.  I sent it as zos.adcdca.pem.

Import this to the keystore

runmqakm -cert -add -file zos.adcdca.pem -type cms -stashed -db zzclient.kdb -label zosca

Upload  the certificates from Linux to z/OS.

On Linux, my CA certificiate was in a *.pem file where the first line was —–BEGIN CERTIFICATE—–.  Send this to z/OS.  I used FTP.

Import the CA into the keyring.

The command adds an existing certificate CARSA1024 for userid START1.  The CONNECT USAGE(CERTAUT) defines this as a CA certificate (without the need to have the certificate belong to CERTAUTH userid).

  /*RACDCERT DELETE  ( LABEL('CARSA1024')) ID(START1) 
 SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 
                                                                   
 RACDCERT ADD('IBMUSER.CARS1024.PEM')  - 
   ID(START1)  WITHLABEL('CARSA1024') 
 RACDCERT ID(START1) CONNECT(RING(MQRING    ) - 
                             USAGE(CERTAUTH) - 
                             ID(START1)  - 
                             LABEL('CARSA1024') 
 racdcert listring(MQRING ) id(start1) 
 SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh 

Refresh the queue manager

%CSQ9 refresh security type(SSL)

I’m thinking of using MQ MinimumRSAKeySize. What do I need to plan for?

If you enable this option in the qm.ini or mqclient.ini, it can have problems which are hard to diagnose.

This option restricts

  1. the key size of certificates with a type of RSA
  2. the key size of any CA certificates  with a type of RSA, used to sign a certificate (whether the certificate is RSA or not).

For example with MinimumRSAKeySize=2048 you cannot use a certificate or CA generated with openssl genpkey -out $name.key.pem -algorithm RSA -pkeyopt rsa_keygen_bits:1024.

On the client machine in /var/mqm/errors/*01* I got AMQ9633E: Bad SSL certificate for channel ….

On z/OS I got CSQX620E … CSQXRESP System SSL error, channel … connection … function ‘gsk_secure_socket_init’ RC=541.  Code 451 means “bad certificate” was received from the remote end.

How do I check?

There is no easy way of displaying the details of all of the certificates in a key store.

Midrange.

You can use the following command to list the labels all of the certificates

runmqakm -cert -list all -db zzclient.kdb -type cms -stashed -v

Then use the following command to display the details of each label in turn (zosca in the example)

runmqakm -cert -details -db zzclient.kdb -type cms -stashed -label zosca

This displays information like the example below for a CA certificate of type RSA and key size 1024.

Label : zosca
Key Size : 1024
Version : X509 V3
Serial : 00
Issuer : "CN=z/OSCertification Authority,OU=TEST,O=ADCD"
Subject : "CN=z/OSCertification Authority,OU=TEST,O=ADCD"
Not Before : 7 July 2020 00:00:00 GMT+01:00
Not After : 7 July 2021 23:59:59 GMT+01:00
Public Key
    ...
Public Key Type : RSA (1.2.840.113549.1.1.1)
  

or for an Elliptic certificate with key size 256.

Label : ca256
Key Size : 256
Version : X509 V3
Serial : ...
Issuer : CN=SSCA256,OU=CA,O=SSS,C=GB
Subject : CN=SSCA256,OU=CA,O=SSS,C=GB
Not Before : 7 February 2021 11:24:56 GMT
Not After : 7 February 2024 11:24:56 GMT
Public Key
    ... 
Public Key Type :  EC_ecPublicKey 

You can only check the certificates that are in your key store, not certificates that are sent as part of the handshake. 

z/OS

The listring command displays the contents of the ring (owner and label).

RACDCERT LISTRING(MQRING) ID(START1)

The list command displays the details of a certificate.

RACDCERT certauth LIST(label(‘ADCD-CA’))

displays information like, for the RSA certificate with a small key size,

 Label: ADCD-CA                                                       
 Certificate ID: 2QiJmZmDhZmjgcHEw8Rgw8FA                             
 Status: TRUST                                                        
 Start Date: 2020/07/06 23:00:00                                      
 End Date:   2021/07/07 22:59:59                                      
 Serial Number: ...                                                
 Issuer's Name:                                                      
      >CN=z/OSCertification Authority.OU=TEST.O=ADCD<                 
 Subject's Name:                                                      
      >CN=z/OSCertification Authority.OU=TEST.O=ADCD<                 
 Signing Algorithm: sha1RSA                                           
 Key Usage: CERTSIGN                                                  
 Key Type: RSA                                                        
 Key Size: 1024                                                       

Certificates signed by this CA would not work if MinimumRSAKeySize=2048 was specified.

Are there any requirements on the certificates used by MQ?

Having set up web servers using digital certificates I know some components are sensitive to the content of a certificate.  For example

  1. Some web browsers want the IP address in the  Alternate Name to match the IP address of the server.
  2. Some clients check that the server’s certificate has been signed with “server auth” – saying it can be used as a server’s certificate.

The short answer is no, there are no requirements. The longer answer is maybe.

For the chinit

I used the following to define certificate for z/OS.

RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('CSQ9CERT')  - 
             O('ADCD') - 
             OU('TEST')) - 
  SIZE(2048) - 
  RSA     - 
  SIGNWITH (CERTAUTH LABEL('ADCD-CA')) - 
NOTAFTER( DATE(2021-12-29))- WITHLABEL('CSQ9CERT') RACDCERT id(START1) ALTER(LABEL('CSQ9CERT'))TRUST RACDCERT ID(START1) CONNECT(RING(MQRING) - LABEL('CSQ9CERT') USAGE(PERSONAL))
RACDCERT LISTRING(MQRING) ID(START1) RACDCERT LIST(LABEL('CSQ9CERT' )) ID(START1) SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Things to be careful about

  1. Check the NOTAFTER date
  2. It needs to be TRUST
  3. The CONNECT of the certificate to the keyring needs USAGE(PERSONAL).  It it does not have it you get CSQX645E +cpf CSQXRESP Certificate … missing for channel …
  4. If the certificate is RSA, or the CA certificate is RSA, then check the keysize of the certificate and the CA.  On MQ 9.2 the size needs to be larger than the MinimumRSAKeySize any QM.INI or mqclient.ini file SSL stanza, in any queue manager or client.

For the MQ Web server

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works. Just do not use KEYUSAGE(CERTSIGN) .

With CERTSIGN Chrome gives NET::ERR_CERT_INVALID, and the TLS trace gives BAD_CERTIFICATE or UNKNOWN_CERTIFICATE. See here.

Colins updates to MQ messages for TLS

As I was trying to get TLS to work on midrange, I had many MQ error messages. Sometimes the messages were a bit vague “you’ve had a problem. Resolve it and restart the channel”.

Below is the list of messages I’ve added comments to. I’ve done it as a blog post as well-known search engines are not finding the pages.

Mid range

z/OS