What RACF audit records are produced with pass tickets?

A pass ticket is a one time password for a userid, valid with the specified application. I’ve blogged Creating and using pass tickets on z/OS.

I’ve also blogged Of course – JCL subroutines is the answer about using ICETOOL to process RACF audit records in SMF.

Create a pass ticket

After I had created the pass ticket, I used the following JCL below to format the RACF SMF PTCREATE record.

//IBMPTICK JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(63,8,CH) HEADER('USER ID') -
ON(184,8,CH) HEADER('FROMJOB') -
ON(14,8,CH) HEADER('RESULT') -
ON(23,8,CH) HEADER('TIME') -
ON(286,8,CH) HEADER('APPL ') -
ON(295,8,CH) HEADER('FORUSER ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'PTCREATE')
OPTION VLSHRT
//

to produce

USER ID    FROMJOB    RESULT     TIME       APPL       FORUSER     USER NAME   
-------- -------- -------- -------- -------- --------- ------------
ZWESVUSR ZWE1AZ SUCCESS 15:00:55 MQWEB COLIN ZOWE SERVER

Which shows from Job ZWE1AZ running with userid ZWESVUSR; it successfully created a pass ticket for userid COLIN with application MQWEB.

Show where the pass ticket is used

Once the pass ticket had been used, I used the following JCL to display the JOBINIT audit record.

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(63,8,CH) HEADER('USER ID ') -
ON(14,8,CH) HEADER('RESULT ') -
ON(23,8,CH) HEADER('TIME ') -
ON(184,8,CH) HEADER('JOBNAME ') -
ON(286,8,CH) HEADER('APPL ') -
ON(631,8,CH) HEADER('SESSTYPE')-
ON(4604,4,CH) HEADER('PTOEVAL ') -
ON(4609,4,CH) HEADER('PSUCC ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
OPTION VLSHRT
//

it produced the output

USER ID    RESULT     TIME       JOBNAME    APPL       SESSTYPE   PTOEVAL    PSUCC 
-------- -------- -------- -------- -------- -------- -------- --------
COLIN SUCCESSP 15:01:02 CSQ9WEB MQWEB OMVSSRV YES YES
COLIN RACINITD 15:01:02 CSQ9WEB MQWEB OMVSSRV NO NO

The first record shows,

  • in job CSQ9WEB,
  • running with APPLication id of MQWEB.
  • Sesstype OMVSSVR is a z/OS UNIX server application. See RACROUTE TYPE=VERIFY under SESSION=type.
  • userid COLIN SUCCCESSfully logged on with Passticket (SUCCESSP)
  • PTOEVAL – YES the supplied password was evaluated as a PassTicket,
  • PSUCC – YES the supplied password was evaluated successfully as a PassTicket.

The second record shows RACINITD (Successful RACINIT deletion) for the userid COLIN in the job CSQ9WEB, and the password was not used.

Zowe: Getting data from Zowe

As part of an effort to trace the https traffic from Zowe, I found there are trace points you can enable.

You can get a list of these from a request like “https://10.1.1.2:7558/application/loggers”. In the browser it returns one long string like (my formatting)

{"levels":["OFF","ERROR","WARN","INFO","DEBUG","TRACE"],
"loggers":{"ROOT":{"configuredLevel":"INFO","effectiveLevel":"INFO"},
"_org":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework.web":{"configuredLevel":null,"effectiveLevel":"INFO"},
...

Once you know the trace point, you can change it. See here.

Using https module

certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
url="https://10.1.1.2:7558/application/loggers"
https GET ${url} $certs $verify

This displayed the data, nicely formatted. But if you pipe it, the next stage receives one long character string.

Using Python

#!/usr/bin/env python3

import ssl
import json
import sys
from http.client import HTTPConnection 
import requests
import urllib3
# trace the traffic flow
HTTPConnection.debuglevel = 1

my_header = {  'Accept' : 'application/json' }

urllib3.disable_warnings()
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

certificate="colinpaice.pem"
key="colinpaice.key.pem"
cpcert=(certificate,key)
jar = requests.cookies.RequestsCookieJar()

s = requests.Session()
geturl="https://10.1.1.2:7558/application/loggers"

res = s.get(geturl,headers=my_header,cookies=jar,cert=cpcert,verify=False)

if res.status_code != 200:
    print("error code",res.status_code)
    sys.exit(8)

headers = res.headers

for h in headers:
    print(h,headers[h])

cookies = res.cookies.get_dict()
for c in cookies:
    print("cookie",c,cookies[c])

js = json.loads(res.text)
print("type",js.keys())
print(js['levels'])
print(js['groups'])
loggers = js['loggers']
for ll in loggers:
    print(ll,loggers[ll])

This prints out one line per item.

The command

python3  zloggers.py |grep HTTP

gives

...
org.apache.http {'configuredLevel': 'DEBUG', 'effectiveLevel': 'DEBUG'}
org.apache.http.conn {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
org.apache.http.conn.ssl {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
...

My WLM definitions were not behaving as I expected.

I had configured WLM so the MQ started tasks (CSQ*) were defined as a low priority STC.

  Subsystem-Type  Xref  Notes  Options  Help                              
--------------------------------------------------------------------------
Modify Rules for the Subsystem Type Row 22 to 25 of 25
Command ===> ___________________________________________ Scroll ===> CSR

Subsystem Type . : STC Fold qualifier names? Y (Y or N)
Description . . . All Started Tasks

Action codes: A=After C=Copy M=Move I=Insert rule
B=Before D=Delete row R=Repeat IS=Insert Sub-rule
More ===>
-------Qualifier-------- -------Class--------
Action Type Name Start Service Report
DEFAULTS: STCLOM ________
____ 1 TN CSQ9WEB ___ STCLOM MQ
____ 1 TN CSQ9CHIN ___ STCLOM MQ
____ 1 TN CSQ9ANG ___ STCLOM MQ

But I could see from SDSF, that the CSQ9CHIN’s SrvClass was STCHIM, and CSQ9WEB’s was STCHIM. It took me a couple of hours digging to find out why.

Higher up the list, the WLM definitions had

         -------Qualifier--------                 -------Class--------    
Action Type Name Start Service Report
DEFAULTS: STCLOM ________
____ 1 TN %MASTER% ___ SYSTEM MASTER
____ 1 SPM SYSTEM ___ SYSTEM ________
____ 1 SPM SYSSTC ___ SYSSTC ________
____ 1 TNG STCHI ___ SYSSTC ________
____ 1 TNG STCMD ___ STCMDM ________
____ 1 TNG MONITORS ___ ________ MONITORS
____ 1 TNG SERVERS ___ STCMDM ________
____ 1 TNG ONLPRD ___ STCHIM ________

There is a definition for ONLPRD (online production), a group of transaction names (Transaction Name Group).

From option 5 Classification Groups, of the main WLM panel it displays

                         Classification Group Menu                        
Select one of the following options.
__ 1. Accounting Information Groups 14. Plan Name Groups
2. Client Accounting Info Groups 15. Procedure Name Groups
3. Client IP Address Groups 16. Process Name Groups
4. Client Transaction Name Groups 17. Scheduling Environment Groups
5. Client Userid Groups 18. Subsystem Collection Groups
6. Client Workstation Name Groups 19. Subsystem Instance Groups
7. Collection Name Groups 20. Subsystem Parameter Groups
8. Connection Type Groups 21. Sysplex Name Groups
9. Correlation Information Groups 22. System Name Groups
10. LU Name Groups 23. Transaction Class Groups
11. Net ID Groups 24. Transaction Name Groups
12. Package Name Groups 25. Userid Groups
13. Perform Groups 26. Container Qualifier Groups

Most of these had no definition, but option 24. Transaction Name Groups gave me

                           Group Selection List                Row 1 to 5 of 5
Command ===> ____________________________________________________________

Qualifier type . . . . . . . : Transaction Name

Action Codes: 1=Create, 2=Copy, 3=Modify, 4=Browse, 5=Print, 6=Delete,
/=Menu Bar
-- Last Change ---
Action Name Description User Date
__ MONITORS Online System Activity monitors TODD 1999/11/16
__ ONLPRD Online Production Subsystems IBMUSER 2023/01/10
__ SERVERS Server Address Spaces TODD 1999/11/16
__ STCHI High STC's TODD 1999/11/16
__ STCMD Medium STC's TODD 1999/11/16

and these names match what is in the classification rules section above.

Option 3 to modify ONLPRD, gave

                              Modify a Group                   Row 1 to 8 of 8
Command ===> ____________________________________________________________

Enter or change the following information:

Qualifier type . . . . . . . : Transaction Name
Group name . . . . . . . . . : ONLPRD
Description . . . . . . . . . Online Production Subsystems
Fold qualifier names? . . . . Y (Y or N)

Qualifier Name Start Description
%%%%DBM1 ___ DB2 Subsystems
%%%%MSTR ___ DB2 Subsystems
%%%%DIST ___ DB2 Subsystems
%%%%SPAS ___ DB2 Subsystems
CICS* ___ CICS Online Systems
IMS* ___ IMS Online Systems
CSQ* ___ MQ Series

and we can see that MQ started tasks starting with CSQ are in this group.

As this definition is higher in the classification rules list – it will take precedence over any definitions I had defined lower down.

Because there was a definition (within the Started classification)

____  1 TNG       ONLPRD   ___                    STCHIM      ________   

Started tasks in the group ONLPRD are classified as STCHIM, and so this explains why the classification of the MQ address spaces were “wrong”.

I had several options

  • Change the groups and put MQ in its own group with STCLOM
  • Move my CSQ9* specific definitions above the group.

What’s the best way of connecting to an HTTPS server. Pass ticket or JWT?

This blog post was written as background to some blog posts on Zowe API-ML. It provides back ground knowledge for HTTPS servers running on z/OS, and I think it is useful on its own. Ive written about an MQWEB server – because I have configured this on my system.

The problem

I want to manage my z/OS queue manager from my Linux machine.I have several ways of doing it.

Which architecture?

  • Use an MQ client. Establish a client connect to the CHINIT, and use MQPUT and MQGET administration messages to the queue manager.
    • You can issue a command string, and get back a response string which you then have to parse
    • You can issue an MQINQ API request to programmatically query attributes, and get the values back in fields. No parsing, but you have to write a program to do the work.
  • Use the REST API. This is an HTTP request in a standard format into the MQWEB server.
    • You can issue a command string, and get back a response string which you then have to parse to extract the values.
    • You can issue a JSON object where the request is encoded in a URL, and get the response back in JSON format. It is trivial to extract individual fields from the returned data.

Connecting to the MQWEB server

If you use REST (over HTTPS) there are several ways of doing this

  • You can connect using userid and password. It may be OK to enter your password when you are at the keyboard, but not if you are using scripts and you may be away from your keyboard. If hackers get hold of the password, they have weeks to use it, before the password expires. You want to give your password once per session, not for every request.
  • You can connect using certificates, without specifying userid and password.
    • It needs a bit of set up at the server to map your certificate to a userid.
    • It takes some work to set up how to revoke your access, if you leave the company, or the certificate is compromised.
    • Your private key could be copied and used by hackers. There is discussion about reducing the validity period from over a year to 47 days. For some people this is still too long! You can have your private certificate on a dongle which you have to present when connecting to a back end. This reduces the risk of hackers using your private key.
  • You can connect with a both certificate and userid and password. The certificate is used to establish the TLS session, and the userid and password are used to logon to the application.
  • You can use a pass ticket. You issue a z/OS service which, if authorised, generates a one time password valid for 10 minutes or less. If hackers get hold of the pass ticket, they do not have long to be able to exploit it. The application generating the pass ticket, does not need the password of the userid, because the application has been set up as trusted.
  • You can use a JSON Web Token (JWT). This has some similarities with certificates. In the payload is a userid value and issuer value . I think of issuer as the domain the JWT has come from – it could be TEST or a company name. From the issuer value, and IP address range, you configure the server to specify a realm value. From the userid and realm you can map this to a userid on the server. This JWT can be valid from minutes to many hours (but under a day). The userid and realm mapping to a userid is different to certificate mapping to a userid.

Setting up a pass ticket

The passticket is used within the sysplex. It cannot be used outside of a sysplex. The pass ticket is a password – so needs to be validated against the RACF database.

The application that generates the pass ticket must be authorised to a profile for the application. For example, define the profile for the application TSO on system S0W1, the profile is TSOS0W1.

 RDEFINE PTKTDATA TSOS0W1 

and a profile to allow a userid to create a pass ticket for the application

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE) 

PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(COLIN) ACCESS(UPDATE)
PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(IBMUSER)ACCESS(UPDATE)

Userids COLIN and IBMUSER can issue the callable service IRRSPK00 to generate a pass ticket for a user for the application TSOS0W1.

The output is a one-use password which has a validity of up to 10 minutes.

As an example, you could configure your MQWEB server to use profile name MQWEB, or CSQ9WEB.

How is it used

A typical scenario is for an application running on a work station to issue a request to an “application” on z/OS, like z/OSMF, to generate a pass ticket for a userid and application name.

The client on the work station then issues a request to the back end server, with the userid and pass ticket. If the back end server matches the application name then the pass ticket will be accepted as a password. The logon will fail if a different application is used, so a pass ticket for TSO cannot be used for MQWEB.
This is more secure than sending a userid and password up with every back end request, but there is additional work in creating the pass ticket, and two network flows.

This solution scales because very little work needs to be done on the work station, and there is some one-off work for the setup to generate the pass tickets.

JSON Web Tokens

See What are JSON Web Tokens and how do they work?

The JWT sent from the client has an expiry time. This can be from seconds to hours. I think it should be less than a day – perhaps a couple of hours at most. If a hacker has a copy of the JWT, they can use it until it expires.

The back end server needs to authenticate the token. It could do this by having a copy of the public certificate in the server’s keyring, or send a request down to the originator to validate it.

If validation is being done with public certificates, because the client’s private key is used to generate the JWT, the server needs a copy of the public certificate in the server’s keyring. This can make it hard to manage if there are many clients.

The Liberty web server has definitions like

<openidConnectClient id="RSCOOKIE" 
clientId="COLINCOO2"
realmName="zOSMF"
inboundPropagation="supported"
issuerIdentifier="zOSMF"
mapIdentityToRegistryUser="false"
signatureAlgorithm="RS384"
trustAliasName="CONN1.IZUDFLT"
trustStoreRef="defaultKeyStore"
userIdentifier="sub"
>
<authFilter id="afint">
<remoteAddress id="myAddress" ip="10.1.0.2" matchType="equals" />
</authFilter >

</openidConnectClient>

For this entry to be used various parameters need to match

  • The issuerIdentifier. This string identifies the client. It could be MEGABANK, TEST, or another string of your choice. It has to match what is in the JWT.
  • signatureAlgorithm. This matches the incoming JWT.
  • trustAliasName and trustStoreRef. These identify the certificate used to validate the certificate
  • remoteAddress. This is the address, or address range of the client’s IP addresses.

If you have 1000 client machines, you may need 1000 <openidConnectClient…/> definitions, because of the different certificate and IP addresses.

You may need 1000 entries in the RACMAP mapping of userid + realm to userid to be used on the server.

How is it used

You generate the JWT. There are different ways of doing this.

  • Use a service like z/OSMF
  • Use a service on your work station. I have used Python to do this. The program is 30 lines long and uses the Python jwt package

You get back a long string. You can see what is in the string by pasting the JWT in to jwt.io.
You pass this to the backend as a cookie. The cookie name depends on what the server is expecting. For example

'Authorization': "Bearer " + token

The JWT has limited access

For the server to use the JWT, it needs definitions to recognise it. If you have two back end servers

  • Both servers could be configured to accept the JWT
    • If the server specified a different REALM, then the mapped userid from the JWT could be different for each server because the userid/realm to userid mapping can be different.
  • One server is configured to accept the JWT
    • If only one server has the definitions for the JWT, then trying to use the JWT to logon to another server will fail.

Tracing input and output of the Liberty web server.

The Liberty web server is used by many IBM products on z/OS, for example z/OSMF, MQSeries and z/OSConnect (but not Zowe).

When using Zowe, I struggled finding out what data was input to the server. As usual, when you have found the answer it is easy.

Once it worked, I had

<httpAccessLogging id="accessLogging" 
logFormat="%a %s ET=%D %r i=%i c=%C "
enabled="true"
/>
<httpEndpoint id="defaultHttpEndpoint"
accessLoggingRef="accessLogging"
httpPort="9080"
httpsPort="10443"
/>

Where the httpEndpoint defines the port 10443 , and references httpAccessLogging.

It one point I had two ports defined for https. I separated the output for each port using

filepath="${server.output.dir}/logs/http_10443_access.log" 

within the httpAccessLogging definition, to output the data to a specific file to match the port.

What data is output?

You can control what data is output. I used logFormat to output what I was interested in.

logFormat="%a %s ET=%D %r i=%i c=%C " 

Where

  • %a is the remote IP address 10.1.0.2
  • %s is the status – if it worked the value is 200.
  • ET=%D. This is the duration of the request in microseconds. It appears as ET=667601
  • %r the first line of the request POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.
  • i=%i the header name from the request. My request did not have one so this comes out as i=-
  • c=%C gives the cookies. You can request a specific cookie. My output had c=jwtToken:eyJraWQiOiJhYVBkRzd5N…. which is the JSON Web Token. To see the contents, I took this token, and pasted it into http:jwt.io.

You can ask for the datetime, but this comes out as a long string with year,month, day hh:mm:ss.uuuuuu. I found the year month and day were not needed, but I could not find how to display just the time.

The output from the above format was

10.1.0.2 200 ET=667601 POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.1 i=- c=jwtToken:eyJraWQiOiJhYVBkRzd5NmZETk5UT....

RACF: Processing audit records

RACF can write to SMF data information about which userid logged on, what resources it accessed etc.. This can be used to check there are no unexpected accesses, and any violations are actioned.

The data tends to be “this userid had access to that resource”. It does not contain numeric values, such as response time.

Overview of SMF data

SMF data is a standard across z/OS. Each product has an SMF record type, and record subtypes are used to provide granularity within a product’s records. It is common for an SMF record to have sections within it. There may be 0 or more sections, and the sections can be of varying length. A SMF formatting program needs to build and report useful information from these sections.

There are many tools or products to process SMF records. Individual products may produce tools for formatting records, and there are external tools available to process the records.

Layout of RACF SMF records

The layout of the RACF SMF records are described in the publications. Record type 80: RACF processing record. It describes the field names, at which offsets, and how to interpret the data (what each bit means), this information is sufficient for someone to write a formatting program.


RACF also provides a formatter. The formatter runs as a SORT exit, and expands the data. For example in the SMF data is a bit saying a userid has the SPECIAL attribute. The formatter expands this and creates a column “SPECIAL” with the value YES or NO. This makes it easy to filter and display records, because you do not need to map bits to their meaning – the exit has done it for you. The layout of the expanded records is described here.

What tools format the records?

A common(free) tool for processing the records that RACF produces is an extension to DFSORT called ICETOOL. (The IBM sort modules all begin with ICE… so calling it ICETOOL was natural).

With ICETOOL you can say include rows where…., display and format these fields, count the occurrences of this field, and add page titles. You can quickly generate tabular reports.

The output file of the RACF exit has different format records mixed up. You need to filter by record type and display the subset of records you need.

JCL to extract the RACF SMF record and convert to the expanded format

//* DUMP THE SMF DATASETS 
// SET SMFPDS=SYS1.S0W1.MAN1
// SET SMFSDS=SYS1.S0W1.MAN2
//*
//SMFDUMP EXEC PGM=IFASMFDP,REGION=0M
//DUMPINA DD DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB DD DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT DD DISP=(NEW,PASS),DSN=&RMF,SPACE=(CYL,(1,1))
//OUTDD DD DISP=(NEW,PASS),DSN=&OUTDD,
// SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//ADUPRINT DD SYSOUT=*
//*XMLFORM DD DSN=COLIN.XMLFORM,DISP=(MOD,CATLG),
//* SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//SYSPRINT DD SYSOUT=*

//SYSIN DD *
INDD(DUMPINA,OPTIONS(DUMP))
INDD(DUMPINB,OPTIONS(DUMP))
OUTDD(DUMPOUT, TYPE(30,80,81,83))
START(0000)
END(2359)
DATE(2025230,2025360)
ABEND(NORETRY)
USER2(IRRADU00)
USER3(IRRADU86)
/*

The RACF exits produce several files.

  • //OUTDD the expanded records are written to this dataset
  • //ADUPRINT contains information on how many of each record type the exit processed
  • //XMLFORM you can have it write data in XML format – for post processing

JCL to process the expanded records

The JCL below invokes the ICETOOL processing.

//S1      EXEC  PGM=ICETOOL,REGION=0M 
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//TEMP DD DSN=&&TEMP3,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PRINT DD SYSOUT=*

Where

  • //IN refers to the //OUTDD statement in the earlier step
  • //TEMP is an intermediate dataset. The sort program writes filtered records to this data set.
  • //PRINT is where the formatted output goes

ICETOOL Processing

The whole job is

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
// INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(5,8,CH) HEADER('EVENT') -
ON(63,8,CH) HEADER('USER ID') -
ON(14,8,CH) HEADER('RESULT') -
ON(23,8,CH) HEADER('TIME') -
ON(175,8,CH) HEADER('TERMINAL') -
ON(184,8,CH) HEADER('JOBNAME') -
ON(286,8,CH) HEADER('APPL ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
OPTION VLSHRT
//

You have to be careful about the offsets. The record has a 4 byte length field on the front of each record. So the field in the layout of the expanded records described here is column 1 for length 8, in the JCL you specify column 5 of length 8. In the documentation the userid is columns 59 of length 8, in the JCL it is ON(63,8,CH).

The processing is ….

  • Copy the data from the dataset in //IN and copy it to the dataset in //TEMP. Using the sort instructions in TEMPCNTL. You take name name in USING(TEMP) and put CNTL on the end to locate the DDname.
  • The sort instructions say include only those records where columns 5 of length 8 of the record are the string ‘JOBINIT ‘ ( so columns 1 for length 8 in the mapping description).
  • The DISPLAY step copies record from the //TEMP dataset to the //PRINT DDNAME.
  • The ON() selects the data from the record, giving start column, length and formatting. For each field, it uses the specified column heading.

The output

In the //PRINT is

EVENT      USER ID    RESULT     TIME       TERMINAL   JOBNAME    APPL    
-------- -------- -------- -------- -------- -------- --------
JOBINIT START1 SUCCESS 09:54:11 SMFCLEAR
JOBINIT START1 TERM 09:54:20 SMFCLEAR
JOBINIT START1 TERM 09:55:13 CSQ9WEB
JOBINIT IBMUSER SUCCESS 10:12:14 LCL702 IBMUSER
JOBINIT IBMUSER SUCCESS 10:14:18 IBMJOBI
JOBINIT IBMUSER TERM 10:14:18 IBMJOBI
JOBINIT IBMUSER SUCCESS 10:21:39 IBMACCES
JOBINIT IBMUSER TERM 10:21:40 IBMACCES
JOBINIT IBMUSER SUCCESS 10:22:10 IBMACCES
JOBINIT IBMUSER TERM 10:22:11 IBMACCES
JOBINIT IBMUSER SUCCESS 10:23:01 IBMPASST
JOBINIT IBMUSER TERM 10:23:05 IBMPASST

Extending this

Knowing the format of the RACF extend record, you can add more fields to the reports.

You can filter which records you want. For example all records for userid START1. You can link filters with AND and OR statements.

Of course – JCL subroutines is the answer

I was processing RACF SMF records to report how clients were logging into MQ. This as a multi step job, and with each report I added, the JCL got more and more messy.
The requirements were simple

  • JCL to copy the dump the SMF data sets to a temporary data set
  • Run a tool against this data set to product the reports
    • There were reports for logon and logoff, and pass tickets, and access to profiles and…
  • I wanted it to be easy to use – and the JCL to fit on one screen!

All of this was easy except my JCL file got bigger with every report I wanted, and I spent a lot of time scrolling up and down, and changing the wrong file!

The solution was to use JCL subroutines – or INCLUDE JCL.

Examples

JCL to process the SMF data sets

You do not need to know what the JCL does – but you need to know it was in COLIN.JCL(RACFSMF)

//* DUMP THE SMF DATASETS 
// SET SMFPDS=SYS1.S0W1.MAN1
// SET SMFSDS=SYS1.S0W1.MAN3
//*
//SMFDUMP EXEC PGM=IFASMFDP,REGION=0M
//DUMPINA DD DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB DD DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT DD DISP=(NEW,PASS),DSN=&RMF,SPACE=(CYL,(1,1))
//OUTDD DD DISP=(NEW,PASS),DSN=&OUTDD,
// SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//ADUPRINT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
//SYSIN DD *
INDD(DUMPINA,OPTIONS(DUMP))
INDD(DUMPINB,OPTIONS(DUMP))
OUTDD(DUMPOUT, TYPE(30,80,81,83))
START(1040)
END(2359)
DATE(2025229,2025360)
ABEND(NORETRY)
USER2(IRRADU00)
USER3(IRRADU86)

/*

Use it

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.JCL
// INCLUDE MEMBER=RACFSMF

//S1 EXEC PGM=ICETOOL,REGION=1024K
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//JOBI DD DSN=&&TEMPJOBI,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PJOBI DD SYSOUT=*
//TOOLIN DD *
COPY FROM(IN) TO(JOBI) USING(JOBI)
DISPLAY FROM(JOBI) LIST(PJOBI) -
...
//JOBICNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
//

The clever bits are the JCLLIB which gives the JCL library, and the INCLUDE MEMBER=RACFSMF which copies in the JCL.

To use the JOBI content, I needed to specify JOBI, PJOBI and JOBICTL, and similarly for each data component. 10 components meant 30 data sets – all with similar content and names, this lead to a mess of JCL.

Going further, I could use a template with the same data set names, (TEMP, PRINT etc) and just change the content.

I coverted the above JCL to create a member ICETOOL

//S1      EXEC  PGM=ICETOOL,REGION=1024K 
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//TEMP DD DSN=&&TEMP3,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PRINT DD SYSOUT=*

and use it with

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.JCL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL

//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -

...
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
//

Where I just had to change the data in italics – and not the boiler plate.

For each RACF record type, I had a different JCL member, based on the above file.
To select SMF records with a date and time range, I just edited member RACFSMF, and submitted the jobs, and they all used it.

This was easy to do and it let me focus on the problem – rather than on the JCL.

I can’t automatically allocate a data set, and my SMS set up is not helping.

I’m running my little zD&T z/OS system on my laptop. I am the only person on this system, so I have to do every thing myself.

I started my MQ system last week, and now it is complaining that it cannot allocate archive logs. From my experience with MQ, I know this is serious. I know I have lots of space on my disks, so why can’t MQ use it.
I’ll go through the diagnostic path I took, which shows the SMS commands I used, and give the solution.

The blog post One minute SMS covers many of the concepts (and commands used).

The error messages

CSQJ072E %CSQ9 ARCHIVE LOG DATA SET 'CSQARC2.CSQ9.B0000002' HAS BEEN ALLOCATED TO NON-TAPE DEVICE AND CATALOGUED, OVERRIDING CATALOG PARAMETER                                    
IGD17272I VOLUME SELECTION HAS FAILED FOR INSUFFICIENT SPACE FOR DATA SET CSQARC2.CSQ9.A0000002 JOBNAME (CSQ9MSTR) STEPNAME (CSQ9MSTR) PROGNAME (CSQYASCP)
REQUESTED SPACE QUANTITY = 120960 KB
STORCLAS (SCMQS) MGMTCLAS ( ) DATACLAS ( )
STORGRPS (SGMQS SGBASE SGEXTEAV )
IKJ56893I DATA SET CSQARC2.CSQ9.A0000002 NOT ALLOCATED+
IGD17273I ALLOCATION HAS FAILED FOR ALL VOLUMES SELECTED FOR DATA SET
CSQARC2.CSQ9.A0000002
IGD17277I THERE ARE (247) CANDIDATE VOLUMES OF WHICH (7) ARE ENABLED OR
QUIESCED
IGD17290I THERE WERE 3 CANDIDATE STORAGE GROUPS OF WHICH THE FIRST 3 814
WERE ELIGIBLE FOR VOLUME SELECTION.
THE CANDIDATE STORAGE GROUPS WERE:SGMQS SGBASE SGEXTEAV
IGD17279I 240 VOLUMES WERE REJECTED BECAUSE THEY WERE NOT ONLINE
IGD17279I 240 VOLUMES WERE REJECTED BECAUSE THE UCB WAS NOT AVAILABLE
IGD17279I 7 VOLUMES WERE REJECTED BECAUSE THEY DID NOT HAVE SUFFICIENT
SPACE (041A041D)

Why is it using the storage class SCMQS?

From the ISMF panels,

  • option 7 Automatic Class Selection
  • option 5 Display – Display ACS Object Information

Gives a panel

   Panel  Utilities  Help                                                       
──────────────────────────────────────────────────────────────────────────────
ACS OBJECT DISPLAY
Command ===>

CDS Name : ACTIVE

ACS Rtn Source Data Set ACS Member Last Trans Last Date Last Time
Type Routine Translated from Name Userid Translated Translated
-------- ----------------------- -------- ---------- ---------- ----------
DATACLAS SYS1.S0W1.DFSMS.CNTL DATACLAS IBMUSER 2019/12/17 15:21
MGMTCLAS ----------------------- -------- -------- ---------- -----
STORCLAS SYS1.S0W1.DFSMS.CNTL STORCLAS IBMUSER 2020/12/02 11:23
STORGRP SYS1.S0W1.DFSMS.CNTL STORGRP IBMUSER 2019/12/17 15:23

So the ACS routine is in SYS1.S0W1.DFSMS.CNTL(STORCLAS)

This file has

PROC STORCLAS 
FILTLIST MQS_HLQ INCLUDE(CSQ*.**,
CSQ.**,
MQS.**,
MQS*.**)
...
SELECT
...
WHEN (&DSN = &MQS_HLQ)
DO
SET &STORCLAS = 'SCMQS'
EXIT CODE(0)
END
...
END
END

This says for any data set name (&DSN) that match the list (&MQS_HLQ) whic has CSQ* or MQS*, then set the Storage class to ‘SCMQS’

What storage groups are connected with the MQ data set?

Member SYS1.S0W1.DFSMS.CNTL(STORGRP) has

...
WHEN (&STORCLAS= 'SCMQS')
DO
SET &STORGRP = 'SGMQS','SGBASE','SGEXTEAV'
EXIT CODE(0)
END
...

so these are the storage groups that MQ data sets will use.

What DASD volumes are in the storage group?

D SMS,SG(SGbase)                             
IGD002I 13:34:38 DISPLAY SMS 699

STORGRP TYPE SYSTEM= 1
SGBASE POOL +
SPACE INFORMATION:
TOTAL SPACE = 29775MB USAGE% = 98 ALERT% = 0
TRACK-MANAGED SPACE = 29775MB USAGE% = 98 ALERT% = 0

Hows there is 29775 M allocated -and it is 98% full.

D SMS,SG(SGMQS)                                                        
IGD002I 13:31:33 DISPLAY SMS 678

STORGRP TYPE SYSTEM= 1
SGMQS POOL +
SPACE INFORMATION:
NOT AVAILABLE TO BE DISPLAYED
***************************** LEGEND *****************************
. THE STORAGE GROUP OR VOLUME IS NOT DEFINED TO THE SYSTEM
+ THE STORAGE GROUP OR VOLUME IS ENABLED
- THE STORAGE GROUP OR VOLUME IS DISABLED
* THE STORAGE GROUP OR VOLUME IS QUIESCED
D THE STORAGE GROUP OR VOLUME IS DISABLED FOR NEW ALLOCATIONS ONLY
Q THE STORAGE GROUP OR VOLUME IS QUIESCED FOR NEW ALLOCATIONS ONLY
> THE VOLSER IN UCB IS DIFFERENT FROM THE VOLSER IN CONFIGURATION
SYSTEM 1 = S0W1

There are no volumes allocated to this storage group.

What volumes are in the storage group?

D SMS,SG(SGBASE),LISTVOL                                             
IGD002I 13:39:07 DISPLAY SMS 705

STORGRP TYPE SYSTEM= 1
SGBASE POOL +
SPACE INFORMATION:
TOTAL SPACE = 29775MB USAGE% = 98 ALERT% = 0
TRACK-MANAGED SPACE = 29775MB USAGE% = 98 ALERT% = 0

VOLUME UNIT MVS SYSTEM= 1 STORGRP NAME
B3USR1 0ADA ONRW + SGBASE
USER0A + SGBASE
USER0B + SGBASE
USER0C + SGBASE
USER0D + SGBASE
USER0E + SGBASE
USER0F + SGBASE
USER00 0A9C ONRW + SGBASE
USER01 + SGBASE
USER02 0AB0 ONRW + SGBASE
USER03 0ACE ONRW + SGBASE
USER04 0AB2 ONRW + SGBASE
USER05 0AB5 ONRW + SGBASE
USER06 0A83 ONRW + SGBASE
...
+ THE STORAGE GROUP OR VOLUME IS ENABLED

How do I see how much space is available in my disks?

ISMF,

  • option 2 – Volume
  • option 1 – DASD

This gives a panel

                          VOLUME SELECTION ENTRY PANEL              Page 1 of 3
Command ===>

Select Source to Generate Volume List . . 2 (1 - Saved list, 2 - New list)
1 Generate from a Saved List Query Name To
List Name . . COLIN Save or Retrieve
2 Generate a New List from Criteria Below
Specify Source of the New List . . 1 (1 - Physical, 2 - SMS)
Optionally Specify One or More:
Enter "/" to select option Generate Exclusive list
Type of Volume List . . . 1 (1-Online,2-Not Online,3-Either)
Volume Serial Number . . USER* (fully or partially specified)
Device Type . . . . . . . (fully or partially specified)
Device Number . . . . . . (fully specified)
To Device Number . . . (for range of devices)
Acquire Physical Data . . Y (Y or N)
Acquire Space Data . . . Y (Y or N)
Storage Group Name . . . (fully or partially specified)
CDS Name . . . . . . .
(fully specified or 'Active')
Use ENTER to Perform Selection; Use DOWN Command to View next Selection Panel;
Use HELP Command for Help; Use END Command to Exit.

or

        Enter "/" to select option      Generate Exclusive list                 
Type of Volume List . . . 1 (1-Online,2-Not Online,3-Either)
Volume Serial Number . . * (fully or partially specified)
Device Type . . . . . . . (fully or partially specified)
Device Number . . . . . . (fully specified)
To Device Number . . . (for range of devices)
Acquire Physical Data . . Y (Y or N)
Acquire Space Data . . . Y (Y or N)
Storage Group Name . . . SGBASE (fully or partially specified)
CDS Name . . . . . . . 'ACTIVE'
(fully specified or 'Active')

You can specify a Volume Serial prefix, a Storage Group Name, or a combination of both.

You need to select Acquire Physical Data, and Acquire Space Data.

You get output like

 LINE       VOLUME FREE       %     ALLOC      FRAG   LARGEST    FREE     
OPERATOR SERIAL SPACE FREE SPACE INDEX EXTENT EXTENTS ... ...
---(1)---- -(2)-- ---(3)--- (4)- ---(5)--- -(6)- ---(7)--- --(8)--
B3USR1 149186K 2 8165315K 375 34032K 36
USER00 67067K 1 8247434K 718 2490K 133
USER02 30601K 1 2740899K 412 11621K 31
USER03 3209K 0 2768291K 333 2213K 6
USER04 146198K 5 2625302K 280 42332K 19
USER05 64466K 2 2707034K 9 63802K 3
USER06 273304K 10 2498196K 177 105581K 14

Which shows I do not have much free space.

Add more space

As it looks like my storage group pools are low on disk space, I need to allocate more volumes.

See Adding more disk space to z/OS, creating volumes and adding them to SMS.

Once I added the volume to the SGBASE storage group, it usage went from

TOTAL SPACE = 29775MB USAGE% = 98 ALERT% = 0                      
TRACK-MANAGED SPACE = 29775MB USAGE% = 98 ALERT% = 0

to

TOTAL SPACE = 32482MB USAGE% = 89 ALERT% = 0                      
TRACK-MANAGED SPACE = 32482MB USAGE% = 89 ALERT% = 0

Using keyrings and certificates on z/OS with Liberty

This is a work in progress… but someone asked me for this information so I’ll publish as it is.

I had a lot of playing around with MQWEB on Linux, and I thought that z/OS would be just a small step more than Linux.  After many false steps, it is very similar to Linux – but because z/OS is more secure than Linux you have to do more work to get the right permissions.

Once you have set up the keyrings and got your web browser to talk to the Liberty at the back end, you can then

  • logon with userid and password
  • logon with digital certificate, so there is now set up to map your Linux  digital certificate to a z/OS userid

Once you have got the z/OS userids set up,  you need to set up the RACF groups which determine if you are allowed to use the server, and if you are a read only user, or a full function.

z/OS certificates background.

  • You use the RACF RACDCERT command to manage certificates
  • Certificates either stored in an (encrypted) RACF data base.
  • You provide similar information as with midrange machines to define certificates, such as Organisation, Common Name, Country etc.
  • On z/OS you cannot specify Extended Key Usage (EKU) attributes.
  • A certificate has two parts,
    • a private part, this allows you to encrypt data for that certificate.  It must be kept securely.
    • a public part  – anyone can have a copy of this – allows them to decrypt a message created using the private key.
  • Certificate ownership.  The RACF documentation says
    • User certificate A certificate that is associated with a RACF user ID and is used to authenticate the user’s identity. The RACF user ID can represent a traditional user or be assigned to a server or started procedure.  These are identified by ID(…).
    • Certificate-authority certificate. A certificate that is associated with a certificate authority and is used to verify signatures in other certificates.   These are identified by CERTAUTH.
    • Site certificate. A certificate that is associated with an off-platform server or other network entity, such as a peer VPNserver. This category of certificate can also be used to share a single certificate and its private key among multiple RACF user IDs. When used for sharing, a certificate might be referred to as a placeholder certificate.  These are identified by SITE.
  • A certificate has to have the TRUST attribute to be visible in a keyring.
  • Applications access certificate via a user’s keyring.
    • A userid can have many keyrings
    • A keyring can have many certificates.  They can belong to User, Certificate Authority, or Site.
    • You can control access at an individual keyring level.
  • Your web browser will likely have two keyrings (similar to midrange)
    • A Server key ring containing the private certificate to identify the server, and has the private key.   This keyring might only be used by a group of MQWEB servers and no other applications. A web server in CICS would have a its own keyring.
    • A Trust store.  This has the public certificates and allows people to communicate with the web server.  It typically has Certificate Authority certificates.   A CA certificate can validate all users who’s certificate was  signed by the CA. So one CA in the keyring could validate 1000 users.   If the end users are using self signed certificates, then the z/OS keyring has to have a copy of every self signed certificate.  If you add a new certificate then you may have to restart the server.
    • You might share this trust store with all web servers, either native or within CICS.

Steps for setting up the certificates and keyrings to connect the browser to the z/OS web server

  • Server keyring
    • Create a z/OS certificate authority
    • Create a server keyring
    • Create a certificate for the server
    • Add the certificate to the keying
    • Add the CA to the keyring
    • Give the mqweb userid access to the keyring.
    • Export the CA certificate, download it, distribute it to all users.  Import into their web browsers.
  • Trust keyring
    • Create a (shared) keyring
    • Get the CA from your users, upload to z/OS import them into the keyring
    • Give the server userids access to the keyring

Once you have been sucessfull with setting up the keyrings you will either be logged on on, or get a logon panel.

You now have to configure the environment to allow people to logon with the correct authorities.

Detailed steps.

I found it easiest to create JCL using batch TSO  with the various commands rather than use panels, or typing commands.  It means you can repeat the steps until they work.

//IBMRABB JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
  Put your commands here
  /* this (with a space in front) is ignored 
/* 

Configuration

  • The MQWEB server runs with userid START1
  • It will have a server key keyring called KEY
  • It will have a trust store keyring called TRUST
  • The IP address of the z/OS image is 10.1.1.2.   This is needed  when defining the server’s certificate because the clients compare this with web server’s location.  It can be IP(‘nn.nn.nn.nn’) or DOMAIN(‘ABC.COM’).

Create the CA

 /* Delete it in case it existed
RACDCERT CERTAUTH   DELETE(LABEL('TEMP-CA'))
 /* the CA needs size >= 2048 otherwise Chrome browser may ignore it
 /* The certificate shows up in browsers under org-TEST 
RACDCERT GENCERT  - 
  CERTAUTH - 
  SUBJECTSDN(CN('TEMPCertification Authority')- 
             O('TEMP') - 
             OU('TEST')) - 
  NOTAFTER(   DATE(2021-07-01  ) -   
  KEYUSAGE(HANDSHAKE,CERTSIGN,DOCSIGN,DATAENCRYPT          ) - 
  RSA SIZE(2048) - 
  WITHLABEL('TEMP-CA') 

RACDCERT CERTAUTH EXPORT(LABEL('TEMP-CA'))- 
       DSN('IBMUSER.CERT.TEMPCA.PEM') - 
       FORMAT(CERTB64) - 
       PASSWORD('password') 

You can now download the dataset IBMUSER.CERT.TEMPCA.PEM and import it into the browser’s keystore.

Create the server keyring

RACDCERT DELRING(KEY) ID(START1)
RACDCERT ADDRING(KEY) ID(START1)

Create the certificate to identify the server

This needs the ALTNAME so the browsers  accept the certificate

RACDCERT ID(START1) DELETE(LABEL('TEMP')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('TEMP') - 
             O('TEMP') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIGNWITH (CERTAUTH LABEL('TEMP-CA')) - 
   RSA SIZE(2048) 
   NOTAFTER(   DATE(2021-12-29))- 
   WITHLABEL('TEMP')
 /* KEYUSAGE is not needed 

RACDCERT ID(START1) ALTER(LABEL(‘TEMP’))TRUST

      
RACDCERT ID(START1) CONNECT(RING(KEY) LABEL('TEMP')) 
RACDCERT ID(START1) CONNECT(RING(KEY) - 
       LABEL('TEMP-CA') CERTAUTH) 
RACDCERT LIST(LABEL('TEMP-CA'             ))  CERTAUTH 
RACDCERT LIST(LABEL('TEMP'                ))  ID(START1) 
                                                                  
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                                                                

Give the servers userid access to the server’s keyring

See here.

RDEFINE RDATALIB START1.KEY.LST UACC(NONE)
 /* READ access enables retrieving one's own private key
 /* UPDATE access enables retrieving other's.
PERMIT START1.KEY.LST CLASS(RDATALIB) ID(START1 ) ACCESS(UPDATE)
PERMIT START1.KEY.LST CLASS(RDATALIB) ID(IBMUSER) ACCESS(UPDATE)

SETROPTS RACLIST(RDATALIB) REFRESH
RLIST FACILITY START1.KEY.LST

 

Upload the CA from the midrange machine and store in the key store.

The certificate ( .pem file) looks like

—–BEGIN CERTIFICATE—–

—–END CERTIFICATE—–

So should be uploaded as a source file.

RACDCERT CERTAUTH DELETE(LABEL('Linux-CA2')) 
RACDCERT SITE ADD('IBMUSER.CACERT.PEM') - 
    WITHLABEL('Linux-CA2') TRUST 
RACDCERT DELRING(TRUST) ID(START1) 
RACDCERT ADDRING(TRUST) ID(START1) 


RACDCERT ID(START1) CONNECT(RING(TRUST ) - 
       LABEL('Linux-CA2') SITE 

RACDCERT LISTRING(TRUST ) ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Of course you run this just once, and use the connect statement to add as many CAs as you have.