zFS on z/OS concepts, from a performance perspective

I was looking into a Java performance problem, and thought the problem may be connected to the performance of the unix files in the ZFS file system. I found it hard to find out useful information on how to investigate the problems. ZFS can produce a lot of information, but I found it hard to know which reports to look at, and what the key fields were.

This blog post gives the overall concept of a cached file system, it is based on my experience of other cached “file” systems. I have no special knowledge about zFS. I hopes it explains the concepts, it may not reflect reality.

Aside:

It reminds be of a lectures at University, where they explained matter was atoms with electrons whizzing around a small, solid nucleus. This was a good picture but entirely inaccurate. We then learned that the nucleus was composed to protons and neutrons. This was also a good picture, and entirely inaccurate. We then learned that protons and neutrons are composed of quarks particles, a good picture, but inaccurate. We then got into string theory and got knotted. Which ever picture you used, it help with the understanding but was not accurate.

zFS on z/OS concepts, from a performance perspective
How to collect zFS statistics
Example of zFS statistics
zFS performance reports I would like to use on z/OS (but can’t)

General background of cache systems.

A cached file system is common in IT. DB2 has buffer pools, MQ has buffer pools, and ZFS has a cache. The concepts are very similar. Over the years the technology has improved and the technology is efficient. For example all of the above system, use data in 4KB pages, and the IO to external media has been optimised.

I like to think of the technology in different layers.

The application layer, where the application does an fread(), MQGET or SQL query.
The interface layer, where it knows which records to get, which MQ message to gets, or which tables, rows and columns to get. This layer has a logical view of the data, and will request the next level down. “Please get me the data for this 4KB page on this data set at this position.
The buffer manager layer. The aim of the buffer manager is to keep the optimum amout of data in cache, and minimise I/O.
- If the requested 4KB page is in the cache then return it. This counts as a cache hit.
- If it is not in the cache then call the data layer, and say please read this page from disk at this location, into this buffer. This counts as a cache miss.
- The buffer manager may have logic which can detect if a file is being read sequentially and perform read ahead. Logic like
  - Read page 19 of the data set, wait for the I/O to complete, return
  - Read page 20 of the data set, wait for the I/O to complete, return
  - Read page 21 of the data set, wait for the I/O to complete, return
  - Read page 22 … Hmm – this looks like a sequential read. Get pages 22 to 30 from the data set, wait for the IO to complete, return page 22
  - Read page 23 – get it from the buffer and return, no I/O
  - Read page 24 – get it from the buffer and return, no I/O
- When a page has been updated, usually it is not written directly to the disk. It is more efficient to write multiple pages in one I/O. This means the application does not have to wait for the I/O. This is often called “write behind” or “lazy write”. When the application has to be sure the write to disk has worked, for example the fsync() request, or a transactional commit; the requester has to wait until the I/O has completed. The write to the disk is a collection of pages possibly from different applications. It is totally separate to the applications writing the records.
- If the cache fills up, the buffer manager is responsible for making space. This might be to reuse the space for pages which have not been used for a long time, or if there are a lot of updated pages, writing these out – or doing both.
- If the same file is often used, then the pages should be in the buffer. If a file is used for the first time, it will need to be read in – some pages synchronously by the application, then pages read in by the read ahead processing.
The data layer. This does the IO to the disk or other external media.

What statistics make sense?

The application can time the request and provide a true duration of the request.

At the interface level, one file read requests may have resulted in many calls to the buffer manager. The first few “get page” requests may be slow because it had to do I/O to the disk. After read ahead became active the reads from the buffer were very fast. “The average get page time” may have little value.

It may be possible to record the number of synchronous disk writes an application did (the fsync() request), but if the write was a lazy write it will not be recorded by against the file. If one I/O wrote 10 pages, four pages were for this application,six pages for that application. Recording the duration of the lazy write for each application has no value.

You can tell how many read and write requests there were to a data set (file system), and how long these requests took. You can also record how many bytes were read or written to a data set.

Overall there may be many statistics that tell you what each level is doing, and how it is performing, but they may not be very helpful when looking from an application viewpoint.

Simple file access example

fopen file name
fread
fwrite
fclose

fopen – Under the covers

Conceptually, the fopen may have logic like

zfs_open. This looks up to see if the file has been used before. It looks for the path name in the meta data cache. The meta data cache has information about the file, for example the file owner, the permissions for the owner, last time the file was read and pointer to the file system it is on, and its location on the file system.

If the path name is not in the meta cache then go to the file system and get the information. To get the information for file /u/colin/doc/myhelp.txt it may have to get a list of the files under /u/colin, then find where the ‘doc’ directory is. Then get information of the files under /u/colin/doc, this has record for myhelp.txt which has information on where this file is on disk. Set “next page” to where the file is on disk. Each of these steps may need or more pages to be read from disk.

fread – under the covers

The fread may have logic like

zfs_reads. Within this it may have logic like

Get the next page value. Does this page exist for in the cache? If so, return the contents, else read it from the file system, store it in the cache, update the next page pointer, and then return the contents.
Loop until enough data bas been read. As the pages are in 4KB units – to read a 10KB message will need 3 pages.

There are smarts; the code has read ahead support. If the system detects there have been a sequence of get next page requests, instead of “Loop until enough data has been read” it can do

Loop until enough data has been read, and start reading the next N pages, ready for the next request from the application.

By the time the application issues the next fread request, the data it needs may already have been read from disk. To the application it looks like there was no file I/O.

There may also be calls to zfs_getattrs, zfs_lookups.

fwrite – under the covers

The fwrite does not write directly to the disk. It writes some log data, and writes the data to the cache. This is known as “dirty data” because it has been changed. There is an internal process that writes the data out to the file system. Writing many records to the file system in one I/O is more efficient than writing one page each time in many IO.

Applications can use the fsync() request to force the writes to the disk.

Characteristics of the cached file system

It changes over time

The behaviour of the ZFS cache will change over time.

At start up, as files are used, the files will be read from disk into the cache.

Once the system has “warmed up”, the frequently used files will be in the cache, and should not need to be read from disk.

You could IPL the system at 0600, and for the first hour it warms up, and the cache settles down to a steady state for the rest of the day. In the evening, you may start other applications for the overnight processing, and these new applications will have a warm up period, and the cache will reach another steady state.

Data in the cache

Data in the file cache can be

Read only, for example a java program uses .jar files to execute. Some .jar files may be used by many applications and be access frequently.
Read only application data. For example a list of names.
Write application data – for example an output list of names, a trace or log file. For some writes this may be an update and so the previous contents need to be read in.

Read only jar files

The cache needs to be big enough to hold the files. Once the files have been read in, there may be no reads from the file system. Any files that had not been used before will need I/O to the file system. If the cache is not big enough then some of the data can be thrown away, and reloaded next time it is needed.

Read only application data

This data may only be used once a day. Typically it will be read in as needed, and once it has been used it is the cache storage could be stolen and reused for other applications.

Write application data

If the write updates existing pages for example writing to the end of a file, or updating a record within the file then the pages will be needed to be in cache. This may require disk I/O, or the page may be in the cache from a previous operation.

If the data is written to an empty page, then the page need not be in the cache before it is written to. Once the page has been updated, it will be written asynchronously, as it is more efficient to write multiple pages in one I/O than multiple I/Os with just one page.

File system activity

A program product file system

This will typically be used read only (even though it may be mounted read/write), so you can expect pages read from the data set, but no write I/O.

User’s data

A user will typically read and write files read I/O and write I/O.

Using subystems like Liberty Web Server ( and so products like z/OSConnect, z/OSMF, ZOWE) these will have read and write activity, as configuration information is used, and data is written to logs.

What happens as the cache is used?

Writing to a file

When data is written to a file, the cache gets updated. Modified pages get queued to be written to disk when

A segment of 64KB of data for a file is filled up.
The application does a fsync() request to say write the file out to disk.
The file is closed
The zfsadm config -sync_interval n has expired.
The cache is very full of updated (dirty) pages the so called Page Reclaim Writes

Reading a file

When a page has been used, it gets put on a queue, with most recent used pages at the front. A hot page (with frequent use) will always be near the front of the queue. If all the pages have data, and the buffer pool needs a buffer page, then the oldest page on this queue is stolen, (or reclaimed) for the new request.

Ideally the buffer pool needs to be big enough so there is always unused space.

If you have a cache of size 100 pages and read a 50 page file, it will occupy 50 pages in the cache. The first time the file is used data will have to be read from disk. The second time the file is used, all the pages are in memory and there is no disk I/O.

If the cache is only 40 pages, then the first 40 pages of data will fill the cache. When page 41 is read it will use replace the buffer with page 1 in it (the oldest page). When page 42 is read, it will replace the buffer with page 2 in it.

If you now read the file a second time – page 1 is no longer in the cache, so it will need to be read from disk, and will replace a buffer. All 40 pages will be read from disk.

Will making the cache bigger help? If you make the cache 45 pages it will have the same problem. If you make it 50 pages the file will just fit – and may still have a problem. If the cache is bigger than 50 pages the file should fit in – but other applications may be using other files, so you need to make the cache big enough for the 50 page file, and any other files being used. There is nothing to tell you how big to make it. The solution seems to be make the cache bigger until the I/O stops (or reduces), and you have 5-10% free pages. If you make the cache very large it might cause paging, so you have a balancing act. It is more important to have no paging, as paging makes it difficult for the buffer manager to manage. (For example it wants to write out a dirty page. It may need to page in the data, then write it out!)

Reusing pages

A page cannot be stolen(reused) if it needs to be written to disk. Once the contents have been written to disk the page can be stolen.

In reality it looks like blocks of 64 KB segments are used, not pages.

There is a VM statistic called Steal Invocations. This is a count of the number of 64KB blocks which were reused.

Overall performance objective

The cache needs to be big enough to keep frequently used files in the cache. If the cache is not big enough then it has to do more work, for example discard files, to make space, and reading files in from disk.

The system provides statistics on

How big the cache is
How many free pages there are
How many segments have been stolen (should be zero)
How many read requests were met from data in the cache (Cache hit), and so by calculation the number of requests that were not in the cache (cache miss), and required disk I/O.

Typically you will not achieve a cache hit of 100% because the application data may not be hot.

A little whoops

I had a little whoops. I wrote to a file, and filled the file system. When I deleted the file, and tried again, it reported there was no space on the device. When I waited for a few seconds, and repeated the command, it worked! This shows there are background tasks running asynchronously which clean up after a file has been used.

Just to make it more complex

ZFS uses 8KB as its “page” which is 2 * 4K pages on disk.
Small files live in the meta data, and not in the user cache!
There is also a Directory Backing Cache also known as Metadata Backing Cache. This seems to be a cache for the meta data, which doesn’t have the same locking. It is described in a Share presentation, zFS Diagnosis I: Performance Monitoring and Tuning Guidelines from 2012. It looks from the more recent documentation as if this has been rolled into the meta cache.

Sysplex support

The sysplex support makes it just a little more complex.

The ZFS support behaves like a client server.

One LPAR has the file system mounted read write – acting as the server. Other system act as the client.

If SYSA has the file system mounted Read Write, and SYSB wants to access a file, it sends a request through XCF. The access is managed by use of Tokens, and a Token Cache.

If you display KNPFS (Kernel Nodes Physical File System?) you get operations such as zfs_open

On Owner. On my single LPAR sysplex, I get values here
On Client. These are all zeros for me.

SMF 92-51 provides statistics on the zfs verbs such as zfs_open

Count of calls made to file systems owned locally or R/O file systems
Count of calls that required a transmit to another sysplex member to complete for locally-owned file systems.
Count of calls made to file systems owned remotely from this member.
Count of calls that required a transmit to another sysplex member to complete for remotely-owned file systems.
Average number of microseconds per call for locally-owned file systems.
Average number of microseconds per call for remotely-owned file systems

Configuration

The ZFS configuration is driven from the SYS1.PARMLIB(BPXPRM00), member with

FILESYSTYPE TYPE(ZFS) ENTRYPOINT(IOEFSCM)

This can have PRM=(aa, bb, …, zz) for SYS1.PARMLIB(IOEPRMaa)… It defaults to parmlib member IOEPRM00. See here for the contents.

“Why were the options I passed to Java ignored” – or “how to tell what options were passed to my Java?”

I was struggling to understand a problem with shared classes in Java and I found the options being used by my program were not as I expected.

I thought it would be a very simple task to display at start up options used. It may be, but I could not find how to do it. If anyone knows the simple answer please tell me.

I found one way – take a dump! This seems a little extreme, but it was all I could find. With Liberty you can take a javacore dump (F IZUSVR1,JAVACORE) and display it, or you can take a dump at start up.

In the jvm.options I specified

-Xdump:java:events=vmstart

This gave me in //STDERR

JVMDUMP039I Processing dump event “vmstart”, detail “” at 2021/05/20 13:19:06 – please wait.
JVMDUMP032I JVM requested Java dump using ‘/S0W1/var/…/javacore.20210520.131906.65569.0001.txt’
JVMDUMP010I Java dump written to /S0W1/var…/javacore.20210520.131906.65569.0001.txt
JVMDUMP013I Processed dump event “vmstart”, detail “”.

In this file was list of all the options passed to the JVM

1CIUSERARGS UserArgs:
2CIUSERARG -Xoptionsfile=/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs/options.default
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/Ma
2CIUSERARG -Xjcl:jclse29
2CIUSERARG -Djava.home=/usr/lpp/java/J8.0_64
2CIUSERARG -Djava.ext.dirs=/usr/lpp/java/J8.0_64/lib/ext
2CIUSERARG -Xthr:tw=HEAVY
2CIUSERARG -Xshareclasses:name=liberty-%u,nonfatal,cacheDirPerm=1000,cacheDir=…
2CIUSERARG -XX:ShareClassesEnableBCI
2CIUSERARG -Xscmx60m
2CIUSERARG -Xscmaxaot4m
2CIUSERARG -Xdump:java:events=vmstart
2CIUSERARG -Xscminjitdata5m
2CIUSERARG -Xshareclasses:nonFatal
2CIUSERARG -Xshareclasses:groupAccess
2CIUSERARG -Xshareclasses:cacheDirPerm=0777
2CIUSERARG -Xshareclasses:cacheDir=/tmp,name=zosmf2
2CIUSERARG -Xshareclasses:verbose
2CIUSERARG -Xscmx100m

…

the storage limits

1CIUSERLIMITS User Limits (in bytes except for NOFILE and NPROC)
NULL ------------------------------------------------------------------------
NULL         type            soft limit  hard limit
2CIUSERLIMIT RLIMIT_AS       1831837696   unlimited
2CIUSERLIMIT RLIMIT_CORE        4194304     4194304
2CIUSERLIMIT RLIMIT_CPU       unlimited   unlimited
2CIUSERLIMIT RLIMIT_DATA      unlimited   unlimited
2CIUSERLIMIT RLIMIT_FSIZE     unlimited   unlimited
2CIUSERLIMIT RLIMIT_NOFILE        10000       10000
2CIUSERLIMIT RLIMIT_STACK     unlimited   unlimited
2CIUSERLIMIT RLIMIT_MEMLIMIT 4294967296  4294967296

and environment variables used.

1CIENVVARS Environment Variables
2CIENVVAR LOGNAME=IZUSVR
2CIENVVAR _CEE_ENVFILE_S=DD:STDENV
2CIENVVAR _EDC_ADD_ERRNO2=1
2CIENVVAR _EDC_PTHREAD_YIELD=-2
2CIENVVAR WLP_SKIP_MAXPERMSIZE=true
2CIENVVAR _BPX_SHAREAS=NO
2CIENVVAR JAVA_HOME=/usr/lpp/java/J8.0_64

…

All interesting stuff including the -X.. parameters. I could see that the parameters I had specified were not being picked up because they were specified higher up! Another face palm moment.

There was a lot more interesting stuff in the file, but this was not relevant to my little problems.

Once z/OSMF was active I took a dump using the f izusvr1,javacore command and looked at the information on the shared classes cache

1SCLTEXTCMST Cache Memory Status
1SCLTEXTCNTD Cache Name      Cache path
2SCLTEXTCMDT sharedcc_IZUSVR /tmp/javasharedresources/..._IZUSVR_G37
2SCLTEXTCPF Cache is 85% full
2SCLTEXTCSZ Cache size = 104857040
2SCLTEXTSMB Softmx bytes = 104857040
2SCLTEXTFRB Free bytes = 14936416

This is where I found the shared cache was not what I was expecting! I originally spotted that the cache was too small – and made it bigger.

Afterwards…

Remember to delete the javacore files.

I removed the -Xdump:java:events=vmstart statement, because I found it more convenient to use the f izusvr1,javacore command to take a dump when needed.

Compiling a C program on z/OS

You can compile programs in USS, or with JCL. I tend to prefer JCL, but do use USS (but it takes time to get it the command right). It took me several attempts to compile and bind a program that uses USS services.

I thought people might be interested in the JCL I use, and the C Compiler options I specified.

I’ll give the JCL (so you can see how much you understand of it) then I’ll annotate it

//ADCDC4 JOB 1,MSGCLASS=H,COND=(4,LE)
//S1 JCLLIB ORDER=CBC.SCCNPRC
// SET LOADLIB=IBMUSER.LOAD
// SET LIBPRFX=CEE
//COMPILE EXEC PROC=EDCCB,
// LIBPRFX=&LIBPRFX,
// CPARM='OPTFILE(DD:SYSOPTF),NOLSEARCH,LSEARCH(/usr/include/)',
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31'
//COMPILE.SYSOPTF DD *
LIST,SOURCE
aggregate(offsethex) 
xref
SEARCH(//'ADCD.C.H',//'SYS1.SIEAHDR.H')
TEST
RENT ILP32
OE
INFO(PAR,USE)
NOMARGINS EXPMAC SHOWINC XREF
LANGLVL(EXTENDED) sscom dll
DEFINE(_ALL_SOURCE)
DEBUG/*
//COMPILE.SYSIN DD *
#pragma linkage(IRRSDL00 ,OS)
#line 26
#pragma runopts(POSIX(ON))
/*Include standard libraries */
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
#include <stdarg.h> 

int main( int argc, char *argv??(??))
{
printf("I'm here program %s. ",argv[0]);
for (int i = 1;i < argc; i++)
printf("Arg %d %s",i,argv[i]);
printf("\n");
}//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB.
//BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.OBJLIB
//BIND.GSK DD DISP=SHR,DSN=SYS1.SIEALNKE
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB
//BIND.SYSIN DD *
INCLUDE CSS(IRRSDL00)
NAME MAINPROG(R)
//ISTEST EXEC PGM=MAINPROG,REGION=0M,PARM='PARM1, Parm2'
//STEPLIB DD DISP=SHR,DSN=&LOADLIB
//SYSPRINT DD SYSOUT=,DCB=(LRECL=200) //SYSOUT DD SYSOUT=
//SYSERR DD SYSOUT=*

Annotate JCL

The compile options are defined here

//ADCDC4 JOB 1,MSGCLASS=H,COND=(4,LE) If the return code from each step is less equal to 4, then it does the next step. If the compile fails with return code 8, the job stops.
//S1 JCLLIB ORDER=CBC.SCCNPRC Use the procedures from this library
// SET LOADLIB=IBMUSER.LOAD A symbol used to say where to store the load module
// SET LIBPRFX=CEE This is used by the Compile procedure
//COMPILE EXEC PROC=EDCCB, Execute the C (component EDC) Compile and Bind
// LIBPRFX=&LIBPRFX, using this library prefix (defined above)
// CPARM=’OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/)’, Read C options from //SYSOPTF, and use /usr/Include to find header files
// BPARM=’SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31′ Binder parameters
//COMPILE.SYSOPTF DD * These additional C compiler options
LIST, Give the assembler output
SOURCE display the source of the program (useful for showing compile errors)
aggregate(offsethex) Show c structures with hex offsets
xref show where thing are used
SEARCH(//’ADCD.C.H’,//’SYS1.SIEAHDR.H’) Look for C header files in these libraries
TEST produce information for some debug tools.
RENT Produce re-entrant code
ILP32 produce 32 bit code ( compare to option LP64)
OE Use Posix standards when looking for header files
INFO(PAR,USE) Print out information messages. Par=Emits warning messages on unused. parameters. USE=Emits information about usage of variables ( eg defined but not used).
NOMARGINS Which columns to use. Default is columns 1-72
EXPMAC expand all macros in the source
SHOWINC display the source of any include in the output
LANGLVL(EXTENDED) Use language extensions, for example use of long long.
sscom Use of Slash Slash COMents “// comments…”
dll Produce code which can be used in a DLL. Useful for some USS type programs
DEFINE(_ALL_SOURCE) Create a #define ALL_SOURCE variable
DEBUG produce debugging information, such as line numbers in stack traces. It turns optimisation off.
/*
//COMPILE.SYSIN DD * Main program follows
#pragma linkage(IRRSDL00 ,OS) A C program called a z/OS function called IRRSDL00. This says it uses a standard z/OS parameter list
#line 26 This tells the C compiler to reset its line number – so the error messages come with the correct line number.
#pragma runopts(POSIX(ON)) This is so posix (Uss) posix functions can be used
/*Include standard libraries */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
int main( int argc, char *argv??(??))
{
printf(“I’m here program %s. “,argv[0]);
for (int i = 1;i < argc; i++)
printf(“Arg %d %s”,i,argv[i]);
printf(“\n”);
}
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB. Where to store the output from the bind
//BIND.SYSLIB DD DISP=SHR,DSN=&LIBPRFX..SCEELKED Use these libraries to resolve external routines
//BIND.OBJLIB DD DISP=SHR,DSN=COLIN.OBJLIB Load the compiled output from here
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB My program needs a z/OS specific function. Get it from here
//BIND.SYSIN DD *
- INCLUDE CSS(IRRSDL00) My program needed the z/OS callable service IRRSDL00 which is in the library SYS1.CSSLIBs
NAME MAINPROG(R) This is the name of the program. It is stored in //BIND.SYSLMOD above
//ISTEST EXEC PGM=MAINPROG,REGION=0M,PARM=’PARM1, Parm2′
//STEPLIB DD DISP=SHR,DSN=&LOADLIB
//SYSPRINT DD SYSOUT=,DCB=(LRECL=200) it prints long lines of output up to 200 chars long
//SYSOUT DD SYSOUT=
//SYSERR DD SYSOUT=*

Certificate logon to MQWEB on z/OS, the hard way.

I described here different ways of logging on to the MQ Web Server on z/OS. This post describes how to use a digital certificate to logon. There is a lot of description, but the RACF statements needed are listed at the bottom.

I had set up my keystore and could logon to MQWEB on z/OS using certificates. I just wanted to not be prompted for a password.

Once it is set up it works well. I thought I would deliberately try to get as many things wrong, so I could document the symptoms and the cure. Despite this, I often had my head in the hands, asking “Why! – it worked yesterday”.

Can I use CHLAUTH ? No – because that is for the CHINIT, and you do not need to have the CHINIT running to run the web server.

Within one MQ Web Server, you can use both “certificate only” logon as well as using “certificate, userid and password” logon.

When using the SAF interface you specify parameters in the mqwebuser.xml file, such as keyrings, and what level of certificate checking you want.

Enable SAF messages.

If you use <safCredentials suppressAuthFailureMessage=”false” …> in the mqwebuser.xml then if a SAF request fails, there will be a message on the z/OS console. You would normally have this value set to “true” because when the browser (or REST client) reauthenticates (it could be every 10 seconds) you will get a message saying a userid does not have access to an APPL, or EJBROLE profile. If you change this (or make any change the mqwebuser.xnml file), issue the command

f CSQ9WEB,refresh,config

To pick up the changes.

Configure the server name

In the mqwebuser.xml file is <safCredentials profilePrefix=”MQWEB“…> there MQWEB identifies the server, and is used in the security profiles (see below).

SSL parameters

In the mqwebuser.xml file you specify

<ssl …
clientAuthenticationSupported=”true”|”false. The doc says The server requests that a client sends a certificate. The client’s certificate is optional
clientAuthentication=”true”|”false” if true, then client must send a certificate.
ssslProtocol=”TLSV1.2″
keyStoreRef=”…”
trustStoreRef=”…”
id=”…”
<sslDefault … sslRef=”…” this points to a particular <ssl id=…> definition. It allows you to have more than one <ssl definition, and pick one.

I think it would have been clearer if the parameters were clientAuthentication=”yes”|”no”|”optional”. See my interpretation of what these mean here.

Client authentication

The client certificate maps to a userid on z/OS, and this userid is used for access control.

The TLS handshake: You have a certificate on your client machine. There is a handshake with the server, where the certificate from the server is sent to the client, and the client verifies it. With TLS client authentication the client sends a certificate to the server. The server validates it.

If any of the following are false, it drops through to Connecting with a client certificate, and authenticate with userid and password below.

Find the z/OS userid for the certificate

The certificate is looked up in a RACDCERT MAP to get a userid for the certificate (see below for example statements). It could be a one to one mapping, or depending on say OU=TEST or C=GB, it can check on part of the DN. If this fails you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0194)
SUBJECT(CN=ADCDC.O=cpwebuser.C=GB) ISSUER(CN=SSCARSA1024.OU=CA.O=SSS.
C=GB).

Check the userid against the APPL class.

The userid is checked against the MQWEB profiles in the APPL class. (Where MQWEB is the name you configured in the web server configuration files). If this fails you get

ICH408I USER(ADCDE ) GROUP(TEST ) NAME(ADCDE ) MQWEB CL(APPL )
WARNING: INSUFFICIENT AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Pick the EJBROLE for the userid

There are several profiles in the EJBROLES class. If the userid has read access to the class, it userid gets the attribute. For example for the profile MQWEB.com.ibm.mq.console.MQWebAdmin, if the userid has at least READ access to the profile, it gets MQWEBADMIN privileges.
If these fail you get messages in the MQWEB message logs(s).

To suppress the RACF messages use option suppressAuthFailureMessage=”false” described above.

The userid needs access to at least one profile to be able to use the MQ Web server.

Use the right URL

The URL is like https://10.1.1.2:9443/ibmmq/console/

No password is needed to logon. If you get this far, displaying the userid information (click on the ⓘ icon) gives you Principal:ADCDE – Read-Only Administrator (Client Certificate Authentication) where ADCDE is the userid from the RACDDEF MAP mapping.

Connecting with a client certificate, and authenticate with userid and password.

The handshake as described above is done as above. If clientAuthentication=”true” is specified, and the handshake fails, then the client gets This site can’t be reached or similar message.

If the site can be reached, and a URL like https://10.1.1.2:9443/ibmmq/console/login.html is used, this displays a userid and password panel.

The password is verified, and if successful the specified userid is looked up in the APPL and EJBROLES profiles as described above.

If you get this far, and have logged on, displaying the userid information (click on the ⓘ icon) gives you Principal:colin – Read-Only Administrator (Client Certificate Authentication) where colin is the userid I entered.

The short solution to implement certificate authentication

If you already have TLS certificates for connecting to the MQ Web Server, you may be able to use a URL like https://10.1.1.2:9443/ibmmq/console/ to do the logon. If you use an invalid URL, it will substitute it with https://10.1.1.2:9443/ibmmq/console/login.html .

My set up.

I set up a certificate on Linux with a DN of C=GB,O=cpwebuser,CN=ADCDC and signed by C=GB,O=SSS,OU=CA,CN=SSCARSA1024. The Linux CA had been added to the trust store on z/OS.

Associate a certificate with a z/OS userid

I set up a RACF MAP of certificate to userid. It is sensible to run these using JCL, and to save the JCL for each definition.

 /*RACDCERT DELMAP( LABEL('ADCDZXX'  )) ID(ADCDE  ) 
 /*RACDCERT DELMAP( LABEL('CA'  )) ID(ADCDZ  )   
RACDCERT MAP ID(ADCDE  )  - 
    SDNFILTER('CN=ADCDC.O=cpwebuser.C=GB') - 
    WITHLABEL('ADCDZXX') 
                                                 
 RACDCERT MAP ID(ADCDZ  )  - 
    IDNFILTER('CN=SSCARSA1024.OU=CA.O=SSS.C=GB') 
    WITHLABEL('CA       ') 
                                                 
 RACDCERT LISTMAP ID(ADCDE) 
 RACDCERT LISTMAP ID(ADCDZ) 
 SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

This mapped the certificate CN=ADCDC.OU=cpwebuser.C=GB to userid ADCDE. Note the “.” between the parts, and the order has changed from least significant to most significant. For other certificates coming in with the Issuer CA of CN=SSCARSA1024.OU=CA.O=SSS.C=GB they will get a userid of ADCDZ.

You do not need to refresh anything as this change becomes visible when the SETROPTS RACLIST REFESH is issued.

First logon attempt

I stopped and restarted my Chrome browser, and used the URL https://10.1.1.2:9443/ibmmq/console. I was prompted for a list of valid certificates. I chose “Subject:ADCD: Issuer:SSCARSA1024 Serial:0194”.

Sometimes it gave me a blank screen, other times it gave me the logon screen with username and Password fields. It had a URL of https://10.1.1.2:9443/ibmmq/console/login.html.

On the z/OS console I got

ICH408I USER(ADCDE ) GROUP(TEST ) NAME(ADCDE ) MQWEB CL(APPL )
WARNING: INSUFFICIENT AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

I could see the the userid(ADCDE) from the RACDCERT MAP was being used (as expected). To give the userid access to the MQWEB resource, I issued the commands

 /* RDEFINE APPL MQWEB UACC(NONE)
PERMIT MQWEB CLASS(APPL) ACCESS(READ) ID(ADCDE)
SETROPTS RACLIST(APPL) REFRESH

And tried again. The web screen remained blank (even with the correct URL). There were no messages on the MQWEB job log. Within the MQWEB stdout (and /u/mqweb/servers/mqweb/logs/messages.log) were messages like

[AUDIT ] CWWKS9104A: Authorization failed for user ADCDE while invoking com.ibm.mq.console on
/ui/userregistry/userinfo. The user is not granted access to any of the required roles: [MQWebAdmin, MQWebAdminRO, MQWebUser].

Give the userid access to the EJBroles

In my mqwebuser.xml I have <safCredentials profilePrefix=”MQWEB”. The MQWEB is the prefix of the EJBROLE resource name. I had set up a group MQPA Web Readonly Admin (MQPAWRA) to make the administration easier. Give the group permission, and connect the userid to the group.

 /* RDEFINE EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO  UACC(NONE) 
PERMIT MQWEB.com.ibm.mq.console.MQWebAdminRO CLASS(EJBROLE) - 
  ACCESS(READ) ID(MQPAWRA) 
CONNECT ADCDE group(MQPAWRA)
SETROPTS RACLIST(EJBROLE) REFRESH

Once the security change has been made, it is visible immediately to the MQWEB server. I clicked the browser’s refresh button and successfully got the IBM MQ welcome page (without having to enter a userid or password). When I clicked on the ⓘ icon it said

Principal:ADCDE – Read-Only Administrator (Client Certificate Authentication)

Logoff doesn’t

If you click the logoff icon, you get logged off – but immediately get logged on again – that’s what certificate authorisation does for you. You need to go to a different web site. If you come back to the ibmmq/console web site, it will use the same certificate as you used before.

Ways of logging on to MQWEB on z/OS.

There are different ways of connecting to the MQ Web Server on z/OS (this is based on the z/OS Liberty Web server). Some ways use the SAF interface. This is an interface to the z/OS security manager. IBM provides RACF, there are other security managers such as TOP SECRET, and ACF2. Userid information is stored in the security manager database.

The ways of connecting to the MQ Web server on z/OS.

No security. Use no_security.xml to set up the MQ Web Server.
Hard coded userids and passwords in a file. Using the basic_registry.xml. This defines userid information like <user name=”mqadmin” password=”mqadmin”> . This is suitable only for a sandbox. The password can be obscured or left in plain text.
Logon by z/OS userid and password. Use zos_saf_registry.xml. Logon is by userid and password and checked by a SAF call to the z/OS security manager. The userid is checked for access to a resource like MQWEB.com.ibm.mq.console.MQWebAdmin in class(EJBROLE) and MQWEB in class(APPL).
Connect with a client certificate, and authenticate using userid and password. This uses zos_saf_registry.xml plus additional configuration. The userid, password and access to the EJBROLE and APPL resources is checked by the SAF interface. The certificate id is not used to check access, it is just used to do the TLS handshake.
Certificate authentication, a password is not required. Connecting use a client certificate. This uses zos_saf_registry.xml plus additional configuration. Using the SAF interface, the certificate maps to a z/OS userid; this ID is used for checking access to the EJBROLE and APPL resource.

The configuration for using TLS is not clear.

I found the documentation for the TLS configuration to be unclear. Two parameters are <ssl clientAuthentication clientAuthenticationSupported…/> The documentation says

If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

I experimented with the different options and the results are below.

I used a web browser with several possible certificates that could be used for authentication. I was given a pop up which listed them. Chrome remembers the choice. With Firefox, you can click an option “set as default“. If this is unticked you get prompted every time.
I used a browser with no certificates for authentication.

When a session was not allowed, I got (from Firefox) Secure Connection Failed. An error occurred during a connection to 10.1.1.2:9443. PR_END_OF_FILE_ERROR

Client Authentication	Client Authentication Supported	Browser with certificates	Browser without certificates
true	ignored	Pick certificate, userid and password NOT required	PR_END_OF_FILE_ERROR
false	true	Pick certificate, userid and password NOT required	A variety of results. One of PR_END_OF_FILE_ERROR, Blank screen Userid and password required
false	false	Userid and password required	Userid and password required

When using certificates, you can chose to specify userid and password instead of client authentication, by using the appropriate URL with https://10.1.1.2:9443/ibmmq/console/login.html, instead of https://10.1.1.2:9443/ibmmq/console .

Note well.

The server caches credential information. If you change the configuration and refresh the server, the change may not be picked up immediately.

Once you have logged on successfully, a cookie is stored in your browser. This may be used to authenticate, until the token has expired. To be sure of clearing this token I restarted my browser.

Why do they ship java products on z/OS with the handbrake on? And how to take the brake off.

I noticed that it takes seconds to start MQ on my little z/OS machine, but minutes (feels like days) to start anything with Liberty Web server. This include the MQWEB, z/OSMF, and Z/OSConnect. I mentioned this to an IBM colleague who asked if I was using Java Shared classes. These get loaded into z/OS shared pages.

When I implemented it, my Liberty server came up in half the time!

I found this blog post which was very helpful, and showed me where to look for more information. I subsequently found this document (from 2006!)

The kinder garden overview of how Java works.

You start with a program written in the Java language.
When you run this, Java converts it into byte codes
These byte codes get converted to native instructions – so a byte code “push onto the stack” may become 8 390 assembler instructions.
This code can be optimised, for example code which is executed frequently can have the assembler instructions rewritten to go faster. It might put code inline instead of out in a subroutine.
If you are using Java shared classes, this code can be written out and reused by other applications, or if you restart the server, it can reused what it created before. Reusing the shared classes means that programs benefit because the byte codes have already been converted into native code, and optimisations have been done on the hot code.

What happens on z/OS?

By default, z/OS writes the code to virtual memory and does not save anything to disk. If you restart your Java application within the same IPL, it can exploit the shared classes which have been converted to native code, and optimised – great- good design. I found the second time I started the web server it took half the time. However I IPL once a day, and start my web server once a day. I do not benefit from having it start faster a second time – as I only started it once per session. By default when you re-ipl, the shared classes code is discarded, and so next time you need the code, it has to be to convert to native instructions again, and it loses any optimisation which had been done.

What is the solution?

It is two easy steps:!

Tell Java to write the information from memory to disk – to take a snaphot.
After IPL tell Java to load memory from the disk image – to restore a snapshot.

It is as simple as that.

Background.

It is all to do with the java -Xshareclasses.

With your application you tell Java where to store information about the shared classed. It defaults to Cache=/tmp/ name=javasharedresources.

In my jvm.options I overrode the defaults and specified

-Xshareclasses:nonFatal 
-Xshareclasses:groupAccess 
-Xshareclasses:cacheDirPerm=0777 
-Xshareclasses:cacheDir=/tmp,name=mqweb

If you give each application a name (such as mqweb) you can isolate the cache to an application and not disrupt another JVM if you change the cache. For example if you restore from a snapshot, only users of that “name” will be affected.

List what is in the cache

You can use the USS command,

java -Xshareclasses:cacheDir=/tmp/,listAllCaches

I used a batch job to do the same thing.

//IBMJAVA  JOB  1 
// SET V='listAllCaches' 
// SET C='/tmp/' 
//S1       EXEC PGM=BPXBATCH,REGION=0M, 
// PARM='SH java -Xshareclasses:cacheDir=&C,&V' 
//STDERR   DD   SYSOUT=* 
//STDOUT   DD   SYSOUT=*

The output below, shows the cache name is mqweb. Once you have created a snapshot it has an entry for it.

Listing all caches in cacheDir /tmp/                                                                          
                                                                                                              
Cache name       level         cache-type      feature         OS shmid       OS semid 
mqweb            Java8 64-bit  non-persistent  cr              8197           4101

For a different product I got

Incompatible shared caches                                     
rseapi                  Java8 32-bit  non-persistent  default

The Incompatible shared caches looks like it means you are using 64 bit Java – but there is a cache using 32 bit Java.

For MQWEB the default parameters are -Xshareclasses:cacheDir=/u/mqweb/servers/.classCache,name=liberty-%u” where /u/mqweb is the WLP parameter, where my parameter are defined, and %u is the userid the server is running under, so in my case liberty=START1.

When I had /u/mqweb/servers/.classCache, then the total command line was too long for BPXBATCH. (Putting it into STDPARM gave me IEC020I 001-4 on the instream STDPARM because the resolved line wa greater than 80 characters. I resolved this by adding -Xshareclasses:cacheDir=/u/mqweb,name=cache to the jvm.options file.

To take a snapshot


//IBMJAVA  JOB  1 
// SET C='/tmp/' 
// SET N='mqweb' 
// SET V='restoreFromSnapshot' 
// SET V='listAllCaches'
// SET V='snapshotCache'
//S1       EXEC PGM=BPXBATCH,REGION=0M, 
// PARM='SH java -Xshareclasses:cacheDir=&C,name=&N,&V' 
//STDERR   DD   SYSOUT=* 
//STDOUT   DD   SYSOUT=* 
//

This job took a few seconds to run.

I believe you have to take the snapshot while your Java application is executing – but I do not know for definite.

Restore a snapshot

To restore a snapshot just use restoreFromSnapshot in the above JCL. This took a few seconds to run.

How to use it.

If you put the restoreFromSnaphot JCL at the start of the web server, it will preload it whenever you use your server.

If you take a snapshot every day before shutting down your server, you will get a copy with the latest optimisations. If you do not take a new snapshot it continues to use the old one.

If you want to not use the shared cache you can get rid of it using the command destroySnapshot.

Is my cache big enough?

If you use the printStats request you get information like

Current statistics for cache "mqweb":                                                
...                                                                                     
cache size                           = 104857040                                     
softmx bytes                         = 104857040                                     
free bytes                           = 70294788 
...
Cache is 32% full                                     
                                                      
Cache is accessible to current user = true

The documentation says

When you specify -Xshareclasses without any parameters and without specifying either the -Xscmx or -XX:SharedCacheHardLimit options, a shared classes cache is created with a default size, as follows:

For 64-bit platforms, the default size is 300 MB, with a “soft” maximum limit for the initial size of the cache (-Xscmx) of 64MB, …

I had specified -Xscmx100m which matches the value reported.

What is in the cache?

You can use the printAllStats command. This displays information like

Classpath

1: 0x00000200259F279C CLASSPATH
/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs/jclSC180/vm.jar
/usr/lpp/java/J8.0_64/lib/se-service.jar
/usr/lpp/java/J8.0_64/lib/math.jar

Methods for a class

0x00000200259F24A4 ROMCLASS: java/util/HashMap at 0x000002001FF7AEB8.
ROMMETHOD: size Signature: ()I Address: 0x000002001FF7BA88
ROMMETHOD: put Signature: (Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; Address: 0x000002001FF7BC50

This shows

there is a class HashMap.
It has a method size() with no parameters returning an Int. It is at…. in memory
There is another method put(Object o1, Object o2) returning an Object. It is at … in memory

Other stuff

There are sections with JITHINTS and other performance related data.

Getting SSL/TLS to work on MQ on z/OS

After I succeeded in getting TLS 1.3 to run on MQ midrange 9.2, I thought I would try it on z/OS. I had not used TLS on z/OS for about 10 years, so it was almost like coming to the topic with very rusty knowledge.

I searched the Knowledge centre and found no relevant hits – lots of hits which were not relevant. I eventually found an SSL related keyword, and this got me to the topic Working with SSL/TLS on z/OS. I think this is well documented. It covered all the things I had to do.

The remained of this post covers the bits not covered by the documentation.

Define SSLTASKS.

You need to define SSLTASKS to be able to use TLS on z/OS. See the comments here. I used

%CSQ9 ALTER QMGR SSLTASKS(5)

You need to restart the CHINIT if you change the value of SSLTASKS.

Set up the keyring for the queue manager.

See here. This post show how to create the keyring and import a CA from z/OS, and import a CA from a Linux system.

If you alter the keyring or certlabl you just need a refresh security type(SSL) command to pick up the changes.

Defining the channel

I tried to define the channel, as this failed for security reasons, I’ve given the RACF setup I had to do.

In this section I defined the specific commands for example DEFINE.CHANNEL. I could have defined DEFINE.* to allow all define commands.

I used a channel called TLS, and define the resource CSQ9.CHANNEL.TLS* to allow my ID to define TLS, TLS1 etc

The command %CSQ9 DEF CHL(TLS) CHLTYPE(SVRCONN) gave me

ICH408I USER(CSQOPR ) GROUP(SYS1 ) NAME(COLIN PAICE ) 167
CSQ9.DEFINE.CHANNEL CL(MQCMDS )
WARNING: INSUFFICIENT AUTHORITY – TEMPORARY ACCESS ALLOWED
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

/* RDELETE MQCMDS CSQ9.DEFINE.CHANNEL
RDEF MQCMDS CSQ9.DEFINE.CHANNEL UACC(NONE)
PERMIT CSQ9.DEFINE.CHANNEL CLASS(MQCMDS ) –
ID(COLIN,IBMUSER) ACCESS(ALTER )

I also set up CSQ9.DELETE.CHANNEL and CSQ9.ALTER.CHANNEL in a similar way, so my userid could maintain the channels.

I refreshed MQ security %CSQ9 refresh security to pick up the changes.

I reissued the command %CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN) and got

ICH408I USER(COLIN ) GROUP(SYS1 ) NAME(COLIN PAICE )
CSQ9.CHANNEL.TLS CL(MQADMIN )
PROFILE NOT FOUND – REQUIRED FOR AUTHORITY CHECKING
ACCESS INTENT(ALTER ) ACCESS ALLOWED(NONE )

I used the RACF commands in a batch job.

/* RDELETE MQADMIN CSQ9.CHANNEL.TLS*
RDEF MQADMIN CSQ9.CHANNEL.TLS* UACC(NONE) WARNING
PERMIT CSQ9.CHANNEL.TLS* CLASS(MQADMIN) –
ID(COLIN,IBMUSER) ACCESS(ALTER )
SETROPTS RACLIST(MQADMIN) REFRESH

I issued the commands

%CSQ9 refresh security
%CSQ9 DEF CHL(TLS ) CHLTYPE(SVRCONN)

And successfully defined the channel.

I changed the cipher spec.

I selected a cipher spec from the list.

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ECDHE_RSA_AES_128_CBC_SHA256)

When I started the channel I got

CSQX631E … CSQXRESP Cipher specifications differ, channel TLS local=ECDHE_RSA_AES_128_CBC_SHA256 remote=TLS_RSA_WITH_AES_256_GCM_SHA384
connection 10.1.0.2 (10.1.0.2)

This was clear; I love clear messages.

I decided to change the z/OS end

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384 )

and the client connected successfully.

With MQ 9.2 I could (and did) change this to

%CSQ9 alter chl(TLS) chltype(SVRCONN) SSLCIPH(ANY_TLS12)

and the client worked successfully. The ANY_TLS12. provides a wide list of supported cipher specifications, includes TLS_RSA_WITH_AES_256_GCM_SHA384 and ECDHE_RSA_AES_128_CBC_SHA256.

When I am ready to support TLS 1.3 I will use ANY_TLS12_OR_HIGHER and ANY_TLS13_OR_HIGHER.

Connect a client to it!

I had had my client connect to a midrange queue manager, so I had working client environment. See here for the journey.

I created a .json file for the CCDT connection to z/OS. I specified

{ "channel":
  [{
    "name": "TLS",
    "clientConnection":
    {
      "connection":
      [{
        "host": "10.1.1.2",
        "port": 1414
       }],
      "queueManager": "CSQ9"
    },
    "transmissionSecurity":
    {
      "cipherSpecification": "ANY_TLS12",
      "certificateLabel": "rsaca256_client",
      "certificatePeerName": ""
    },
    "type": "clientConnection"
  }]
}

When it connected I got messages

+CSQX511I %CSQ9 CSQXRESP Channel TLS started connection 10.1.0.2
ICH408I USER(COLINPAI) GROUP( ) NAME(??? )
LOGON/JOB INITIATION – USER AT TERMINAL NOT RACF-DEFINED
IRR012I VERIFICATION FAILED. USER PROFILE NOT FOUND.
+CSQX512I %CSQ9 CSQXRESP Channel TLS no longer active connection 10.1.0.2

COLINPAI came from the userid on the Linux machine (colinpaice) upper cased and truncated. This id will be flowed and used as the MCAUSER if you don’t set it to anything else, using CHLAUTH for example (Thanks to Morag for this information).

Enable chlauth

To be able to map from the DN in a certificate to a z/OS userid you have to use MQ CHLAUTH. See Mapping a client user ID to an MCAUSER user ID.

Check it is enabled at the queue manager level and enable it it needed.

%CSQ9 DIS QMGR CHLAUTH
%CSQ9 ALTER QMGR CHLAUTH(ENABLED)

Define a mapping from certificate to userid

I used

//S1 EXEC PGM=CSQUTIL,PARM='CSQ9' 
//STEPLIB  DD DSN=COLIN.MQ921.SCSQLOAD,DISP=SHR 
//         DD DSN=COLIN.MQ921.SCSQANLE,DISP=SHR 
//SYSPRINT DD SYSOUT=* 
//SYSIN   DD * 
 COMMAND DDNAME(COMMAND) 
//COMMAND DD * 
 SET CHLAUTH('TLS') + 
     TYPE(SSLPEERMAP) SSLPEER('O="cpwebuser"') + 
     ACTION(REPLACE)   + 
     MCAUSER(ADCDD ) CHCKCLNT(ASQMGR) 
/*

This says for channel TLS, take the Organisation(O=..) from the certificate, and if it is cpwebuser then set the ID to ADCDD.

Check it works

Once the channel had started I used

%CSQ( DIS CHS(TLS)
it displayed the following, where I have removed lines which are not relevant to TLS and added some comments

CHSTATUS(TLS)
CHLTYPE(SVRCONN)
…
SECPROT(TLSV12) – this is the level of the protocol
SSLCERTI(CN=SSCARSA1024,OU=CA,O=SSS,C=GB)- this is the DN of the issuer of the SSLPEER certificate (below)
SSLCERTU(START1) – the IBM documentation says “The local user ID associated with the remote certificate.” I dont know where this comes from.. how to change it, or where it is used.
SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384) – The negotiated cipher spec
SSLRKEYS(0) -The number of successful TLS key resets.
SSLKEYTI() -The time on which the previous successful TLS secret key was reset. The secret key has not been reset
SSLKEYDA() -The date on which the previous successful TLS secret key was reset. The secret key has not been reset
SSLPEER(SERIALNUMBER=01:90,CN=rsaca256,O=cpwebuser,C=GB, UNSTRUCTUREDNAME=openssl_ca_user_cnf.keyAgreement2, UNSTRUCTUREDNAME=localhost, UNSTRUCTUREDADDRESS=127.0.0.1) . This is information from the certificate at the remote end.
…
MCAUSER(ADCDD) – This is the userid (set by the CHLAUTH above) used by this channel.
LOCLADDR(10.1.1.2(1414)) – This is the address the connection came in from. This value will be different it you have different IP stacks and different listener ports.

Taking the brakes off ZFS on z/OS – move it to OMVS

From z/OS 2.2 there is a performance advantage in running the ZFS file system as part of OMVS, rather than its own address space. The IBM documentation says When running zFS in the OMVS address space, each file system vnode operation (such as creating a directory entry, removing a directory entry, or reading from a file) will have better overall performance. Each operation will take the same amount of time while inside zFS itself. The performance benefit occurs because z/OS UNIX can call zFS for each operation in a more efficient manner. This will be relevant when you application is doing a lot of file IO – for example using a web server.

This move is not documented – but it is really easy! It is mentioned here. Instructions are hidden in the installation instructions here.

Before I started

The IBM doc says You can determine if zFS is in its own address space by issuing D OMVS,PFS. If the output shows an ASNAME value, zFS is running as a colony address space.

OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION                                            
 PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
 ...   
  ZFS       IOEFSCM    ZFS       LOCAL     A     2021/02/17 17.35.06

The steps I took…

I added KERNELSTACKS(ABOVE) to USER.Z24A.PARMLIB(BPXPRM00).
Being ultra cautious I re-ipled.
The documentation talks about putting IOEZPRM DD in OMVS, then goes on to say As the preferred alternative to the IOEZPRM DDNAME specification, delete the IOEZPRM DDNAME and use the IOEPRMxx parmlib member. So I did not change the OMVS proc. When I reipled it worked and I got the message IOEZ00374I No IOEZPRM DD specified in OMVS proc. Parmlib search being used.
I edited USER.Z24A.PARMLIB(BPXPRM00) and removed the ASNAME in FILESYSTYPE TYPE(ZFS) ENTRYPOINT(IOEFSCM)~~ASNAME(ZFS)~~ Well I actually made a copy of the original line and put it between /* and */, then deleted the text.
I reipled.

Afterwards

The D OMVS,PFS command now gives N/A instead of the Address Space Name

OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB)                
PFS CONFIGURATION INFORMATION                                         
 PFS TYPE   ENTRY      ASNAME    DESC      ST    START/EXIT TIME         
...
  ZFS       IOEFSCM    N/A       LOCAL     A     2021/02/17 17.55.47

Easy!

The hardest part was making sure I had an IPLable SARES1 in case I got it wrong!

Issuing commands…

I used to issue commands like f zfs,query,all. Now that the ZFS address space does not exist, you need to use f omvs,pfs=zfs,query,all.

Colins updates to MQ messages

As I was trying to get TLS to work on midrange, I had many MQ error messages. Sometimes the messages were a bit vague “you’ve had a problem. Resolve it and restart the channel”.

Below is the list of messages I’ve added comments to. I’ve done it as a blog post as well-known search engines are not finding the pages.

Mid range

z/OS

Client

JMSCMQ0001: IBM MQ call failed with compcode ‘2’ (‘MQCC_FAILED’) reason ‘2278’ (‘MQRC_CLIENT_CONN_ERROR’)

Control unit cache section (SMF 42.2) statistics appear strange and unloved. Should I use them? No!

Why should I not use them?

The SMF 42 subtype 2 control unit cache records seem to have strange data in them – and seem to add little value. The RMF 74 subtype 5 – Cache Subsystem Device Activity include most of the data in the 74.2, and contains additional data. The 74.5 record layout has good documentation. The documentation for 42.2 does not.

Why do I think the record is unloved?

Some of the record formats are non standard.

Many SMF records use a time since midnight in hundred’s of a second. SMF 42.2 does it in seconds (so I had to change my formatting routine – sigh).
SMF dates tend to be packed decimal. SMF 42.2 a date is documented as “Year, in the form 0cyy, where c is 0 for 19xx and 1 for 20xx, and yy”. It is really just the year – 1900. The value x79 is decimal 121 so 1900 + 121 is 2021 – this year.
It has two data sections “Statistics gathered from last update period” and Statistics gathered from current update period. The current is 90 seconds after the “last” ( should that be previous interval?). I dont know what data these sections contain. Is it for the whole duration between the SMF records, or just the last 90 seconds of each interval.
It has a field “Fast write bypasses per minute (an integer).” for each of the intervals. I don’t know what it means especially when you consider the interval between the two sections is 90 seconds.
The SMF records were produced every 30 minutes. I dont know how to get the data for the other 27 minutes.
There are sections for each SMS managed volume. The data basically says “Is Cache and Fast write” enabled for this volume. Yes/Yes. So what?

As this has a subtype of 2 I expect this record was produced 30+ years ago when people did not really know what they wanted in the SMF records. I expect Development quietly ignored this subtype, and created other subtypes with useful data. This is why I think you should use other records instead of this 42.2 records.