Why can’t Liberty see my bootstrap.properties? The answer is in the title!

I put the osgi address and port information into the bootstrap.properties file, but it wasn’t being picked up. A couple of hours later I found out why – the answer is that Liberty could see my bootstrap.properties file.

The command

ls -lT bootstrap.properties

gave

- untagged    T=off -rwx------   1 OMVSKERN SYS1 ... bootstrap.properties     

which means the started task running Liberty did not have access to it, only the owner had rwx access. Java did not process it, because it could not see it.

I used the command

chmod 744 bootstrap.properties

and next time I started the Liberty instance, it could find the file.

My file had

osgi.console = 10.1.1.2:5400

The TSO command

tso netstat allconn (port 5400

saying – show all the connections which are, or were recently active, filtered by port 5400

gave

User Id  Conn     State
------- ---- -----
CSQ9WEB 00000139 Listen
Local Socket: ::ffff:10.1.1.2..5400
Foreign Socket: ::ffff:0.0.0.0..0

Showing it is active.

Understanding LTPA tokens for accessing a web site.

What is an LTPA token? – the short answer

When a client connects to a web server and logs on (for example with a userid and password), the server can send back a cookie containing encrypted logon information.

When the client sends this cookie in the next request, the server decrypts it, and so shortens the logon process. Eventually the token will expire and the client will need to logon again.

What is an LTPA token? – a longer answer.

There is a long (perhaps too long) description of LTPA here.

The Lightweight Third-Party Authentication says When accessing web servers that use the LTPA technology it is possible for a web user to re-use their login across physical servers.

The server has an encryption key which it uses to encrypt the “reuse” content. This key could change every time the server start, in fact it is good practice to change the key “often”.
For Liberty the information is configured in the <ltpa…/> tag. See Configuring LTPA in Liberty.

If a client has an LTPA2 token, and the server restarts, if they encryption keys are the same as before, the LTPA2 will be accepted (providing it hasn’t expired). If they encryption key has changed, the client will need to logon to get a new LTPA.

How to configure LTPA?

The ltpa tag in server.xml looks like

<ltpa keysFileName="yourLTPAKeysFileName.keys" 
      keysPassword="keysPassword" 
      expiration="120" />

Where

  • keysFileName defaults to ${server.output.dir}/resources/security/ltpa.keys Note: This is a file, not a keyring.
  • expiration defaults to 120 minutes.

Often password or key data is just base64 encoded, so it trivial to decode. You can encrypt these using the Liberty securityUtility command. The createLTPAKeys option creates a set of LTPA keys for use by the server, or that can be shared with multiple servers. If no server or file is specified, an ltpa.keys file is created in the current working directory.

Single server mode

If the web server is isolated and has its own encryption key then the LTPA can be passed to the server. It can use its encryption key to decrypt the userid information and use it.

If you try to use the LTPA on another web server, which has a different encryption key, the decryption will fail, and the LTPA cannot be used.

Sharing an encryption key

If multiple web servers share the same key, then the LTPA can be used on those servers. For example, you have multiple back-end servers for availability, and a work request can be routed to any server. Once the client has logged on and got the LTPA, future requests can be routed to any of the servers, without the need to logon. The LTPA can be decrypted because of the shared key.

Does this give a different access?

If MQWEB and z/OSMF share the same encryption key, a client logs on to MQWEB and gets a LTPA. The client then uses the LTPA to logon to z/OSMF. All this does is replace the userid and password. MQWEB and z/OSMF still have to determine what permissions the userid has. LTPA does not affect the access.

What happens when the LTPA expires?

Using CURL to send a request to MQWEB with an expired LTPA, I got

 
< HTTP/2 401
...
* Replaced cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
* Added cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
* Added cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
...
{"error": [{
"msgId": "MQWB0112E",
"action": "Login to the REST API to obtain a valid authentication cookie.",
"completionCode": 0,
"reasonCode": 0,
"type": "rest",
"message": "MQWB0112E: The 'LtpaToken2_8443' authentication token cookie failed verification.",
"explanation": "The REST API request cannot be completed because the authentication token failed verification."

Where MQ uses the token name LtpaToken2_${httpsPort}, and so has the https port in it.

Because the Ltpa token had expired, a null token was stored in the cookie.

MQ returned a message giving an explanation. Each product will be different.

Tracing LTPA2

Using a LTPA2 token.

I had problems with LTPA2 – I thought an LTPA2 token would expire – but it continued to work.
A Liberty trace with

com.ibm.ws.security.token.ltpa.internal.LTPAToken2=finest

gave me a lot more data which was not useful – but it did give me

LTPAToken2 3 Current time = Wed Aug 27 17:21:31 GMT 2025, expiration time = Wed Aug 27 19:16:23 GMT 2025

This shows the LTPA was valid from 17:21 to 19:16 or 115 minutes remaining.

Logging on and creating a LTPA token

With

com.ibm.ws.security.token.ltpa.internal.LTPAToken2=finer

I got lots of data including

00000055 LTPAToken2    >  <init> Entry 
user:ADCDPL/COLIN
120

Which shows the userid, and the expiry interval in minutes

When the LTPA had expired

Current time = Thu Aug 28 06:51:35 GMT 2025, expiration time = Wed Aug 27 19:42:20 GMT 2025  
The token has expired: current time = Thu Aug 28 06:51:35 GMT 2025, expire time = Wed Aug 27 19:42:20 GMT 2025.

What RACF audit records are produced with pass tickets?

A pass ticket is a one time password for a userid, valid with the specified application. I’ve blogged Creating and using pass tickets on z/OS.

I’ve also blogged Of course – JCL subroutines is the answer about using ICETOOL to process RACF audit records in SMF.

Create a pass ticket

After I had created the pass ticket, I used the following JCL below to format the RACF SMF PTCREATE record.

//IBMPTICK JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(63,8,CH) HEADER('USER ID') -
ON(184,8,CH) HEADER('FROMJOB') -
ON(14,8,CH) HEADER('RESULT') -
ON(23,8,CH) HEADER('TIME') -
ON(286,8,CH) HEADER('APPL ') -
ON(295,8,CH) HEADER('FORUSER ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'PTCREATE')
OPTION VLSHRT
//

to produce

USER ID    FROMJOB    RESULT     TIME       APPL       FORUSER     USER NAME   
-------- -------- -------- -------- -------- --------- ------------
ZWESVUSR ZWE1AZ SUCCESS 15:00:55 MQWEB COLIN ZOWE SERVER

Which shows from Job ZWE1AZ running with userid ZWESVUSR; it successfully created a pass ticket for userid COLIN with application MQWEB.

Show where the pass ticket is used

Once the pass ticket had been used, I used the following JCL to display the JOBINIT audit record.

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(63,8,CH) HEADER('USER ID ') -
ON(14,8,CH) HEADER('RESULT ') -
ON(23,8,CH) HEADER('TIME ') -
ON(184,8,CH) HEADER('JOBNAME ') -
ON(286,8,CH) HEADER('APPL ') -
ON(631,8,CH) HEADER('SESSTYPE')-
ON(4604,4,CH) HEADER('PTOEVAL ') -
ON(4609,4,CH) HEADER('PSUCC ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
OPTION VLSHRT
//

it produced the output

USER ID    RESULT     TIME       JOBNAME    APPL       SESSTYPE   PTOEVAL    PSUCC 
-------- -------- -------- -------- -------- -------- -------- --------
COLIN SUCCESSP 15:01:02 CSQ9WEB MQWEB OMVSSRV YES YES
COLIN RACINITD 15:01:02 CSQ9WEB MQWEB OMVSSRV NO NO

The first record shows,

  • in job CSQ9WEB,
  • running with APPLication id of MQWEB.
  • Sesstype OMVSSVR is a z/OS UNIX server application. See RACROUTE TYPE=VERIFY under SESSION=type.
  • userid COLIN SUCCCESSfully logged on with Passticket (SUCCESSP)
  • PTOEVAL – YES the supplied password was evaluated as a PassTicket,
  • PSUCC – YES the supplied password was evaluated successfully as a PassTicket.

The second record shows RACINITD (Successful RACINIT deletion) for the userid COLIN in the job CSQ9WEB, and the password was not used.

Zowe: Getting data from Zowe

As part of an effort to trace the https traffic from Zowe, I found there are trace points you can enable.

You can get a list of these from a request like “https://10.1.1.2:7558/application/loggers&#8221;. In the browser it returns one long string like (my formatting)

{"levels":["OFF","ERROR","WARN","INFO","DEBUG","TRACE"],
"loggers":{"ROOT":{"configuredLevel":"INFO","effectiveLevel":"INFO"},
"_org":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework.web":{"configuredLevel":null,"effectiveLevel":"INFO"},
...

Once you know the trace point, you can change it. See here.

Using https module

certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
url="https://10.1.1.2:7558/application/loggers"
https GET ${url} $certs $verify

This displayed the data, nicely formatted. But if you pipe it, the next stage receives one long character string.

Using Python

#!/usr/bin/env python3

import ssl
import json
import sys
from http.client import HTTPConnection 
import requests
import urllib3
# trace the traffic flow
HTTPConnection.debuglevel = 1

my_header = {  'Accept' : 'application/json' }

urllib3.disable_warnings()
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

certificate="colinpaice.pem"
key="colinpaice.key.pem"
cpcert=(certificate,key)
jar = requests.cookies.RequestsCookieJar()

s = requests.Session()
geturl="https://10.1.1.2:7558/application/loggers"

res = s.get(geturl,headers=my_header,cookies=jar,cert=cpcert,verify=False)

if res.status_code != 200:
    print("error code",res.status_code)
    sys.exit(8)

headers = res.headers

for h in headers:
    print(h,headers[h])

cookies = res.cookies.get_dict()
for c in cookies:
    print("cookie",c,cookies[c])

js = json.loads(res.text)
print("type",js.keys())
print(js['levels'])
print(js['groups'])
loggers = js['loggers']
for ll in loggers:
    print(ll,loggers[ll])

This prints out one line per item.

The command

python3  zloggers.py |grep HTTP

gives

...
org.apache.http {'configuredLevel': 'DEBUG', 'effectiveLevel': 'DEBUG'}
org.apache.http.conn {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
org.apache.http.conn.ssl {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
...

My WLM definitions were not behaving as I expected.

I had configured WLM so the MQ started tasks (CSQ*) were defined as a low priority STC.

  Subsystem-Type  Xref  Notes  Options  Help                              
--------------------------------------------------------------------------
Modify Rules for the Subsystem Type Row 22 to 25 of 25
Command ===> ___________________________________________ Scroll ===> CSR

Subsystem Type . : STC Fold qualifier names? Y (Y or N)
Description . . . All Started Tasks

Action codes: A=After C=Copy M=Move I=Insert rule
B=Before D=Delete row R=Repeat IS=Insert Sub-rule
More ===>
-------Qualifier-------- -------Class--------
Action Type Name Start Service Report
DEFAULTS: STCLOM ________
____ 1 TN CSQ9WEB ___ STCLOM MQ
____ 1 TN CSQ9CHIN ___ STCLOM MQ
____ 1 TN CSQ9ANG ___ STCLOM MQ

But I could see from SDSF, that the CSQ9CHIN’s SrvClass was STCHIM, and CSQ9WEB’s was STCHIM. It took me a couple of hours digging to find out why.

Higher up the list, the WLM definitions had

         -------Qualifier--------                 -------Class--------    
Action Type Name Start Service Report
DEFAULTS: STCLOM ________
____ 1 TN %MASTER% ___ SYSTEM MASTER
____ 1 SPM SYSTEM ___ SYSTEM ________
____ 1 SPM SYSSTC ___ SYSSTC ________
____ 1 TNG STCHI ___ SYSSTC ________
____ 1 TNG STCMD ___ STCMDM ________
____ 1 TNG MONITORS ___ ________ MONITORS
____ 1 TNG SERVERS ___ STCMDM ________
____ 1 TNG ONLPRD ___ STCHIM ________

There is a definition for ONLPRD (online production), a group of transaction names (Transaction Name Group).

From option 5 Classification Groups, of the main WLM panel it displays

                         Classification Group Menu                        
Select one of the following options.
__ 1. Accounting Information Groups 14. Plan Name Groups
2. Client Accounting Info Groups 15. Procedure Name Groups
3. Client IP Address Groups 16. Process Name Groups
4. Client Transaction Name Groups 17. Scheduling Environment Groups
5. Client Userid Groups 18. Subsystem Collection Groups
6. Client Workstation Name Groups 19. Subsystem Instance Groups
7. Collection Name Groups 20. Subsystem Parameter Groups
8. Connection Type Groups 21. Sysplex Name Groups
9. Correlation Information Groups 22. System Name Groups
10. LU Name Groups 23. Transaction Class Groups
11. Net ID Groups 24. Transaction Name Groups
12. Package Name Groups 25. Userid Groups
13. Perform Groups 26. Container Qualifier Groups

Most of these had no definition, but option 24. Transaction Name Groups gave me

                           Group Selection List                Row 1 to 5 of 5
Command ===> ____________________________________________________________

Qualifier type . . . . . . . : Transaction Name

Action Codes: 1=Create, 2=Copy, 3=Modify, 4=Browse, 5=Print, 6=Delete,
/=Menu Bar
-- Last Change ---
Action Name Description User Date
__ MONITORS Online System Activity monitors TODD 1999/11/16
__ ONLPRD Online Production Subsystems IBMUSER 2023/01/10
__ SERVERS Server Address Spaces TODD 1999/11/16
__ STCHI High STC's TODD 1999/11/16
__ STCMD Medium STC's TODD 1999/11/16

and these names match what is in the classification rules section above.

Option 3 to modify ONLPRD, gave

                              Modify a Group                   Row 1 to 8 of 8
Command ===> ____________________________________________________________

Enter or change the following information:

Qualifier type . . . . . . . : Transaction Name
Group name . . . . . . . . . : ONLPRD
Description . . . . . . . . . Online Production Subsystems
Fold qualifier names? . . . . Y (Y or N)

Qualifier Name Start Description
%%%%DBM1 ___ DB2 Subsystems
%%%%MSTR ___ DB2 Subsystems
%%%%DIST ___ DB2 Subsystems
%%%%SPAS ___ DB2 Subsystems
CICS* ___ CICS Online Systems
IMS* ___ IMS Online Systems
CSQ* ___ MQ Series

and we can see that MQ started tasks starting with CSQ are in this group.

As this definition is higher in the classification rules list – it will take precedence over any definitions I had defined lower down.

Because there was a definition (within the Started classification)

____  1 TNG       ONLPRD   ___                    STCHIM      ________   

Started tasks in the group ONLPRD are classified as STCHIM, and so this explains why the classification of the MQ address spaces were “wrong”.

I had several options

  • Change the groups and put MQ in its own group with STCLOM
  • Move my CSQ9* specific definitions above the group.

What’s the best way of connecting to an HTTPS server. Pass ticket or JWT?

This blog post was written as background to some blog posts on Zowe API-ML. It provides back ground knowledge for HTTPS servers running on z/OS, and I think it is useful on its own. Ive written about an MQWEB server – because I have configured this on my system.

The problem

I want to manage my z/OS queue manager from my Linux machine.I have several ways of doing it.

Which architecture?

  • Use an MQ client. Establish a client connect to the CHINIT, and use MQPUT and MQGET administration messages to the queue manager.
    • You can issue a command string, and get back a response string which you then have to parse
    • You can issue an MQINQ API request to programmatically query attributes, and get the values back in fields. No parsing, but you have to write a program to do the work.
  • Use the REST API. This is an HTTP request in a standard format into the MQWEB server.
    • You can issue a command string, and get back a response string which you then have to parse to extract the values.
    • You can issue a JSON object where the request is encoded in a URL, and get the response back in JSON format. It is trivial to extract individual fields from the returned data.

Connecting to the MQWEB server

If you use REST (over HTTPS) there are several ways of doing this

  • You can connect using userid and password. It may be OK to enter your password when you are at the keyboard, but not if you are using scripts and you may be away from your keyboard. If hackers get hold of the password, they have weeks to use it, before the password expires. You want to give your password once per session, not for every request.
  • You can connect using certificates, without specifying userid and password.
    • It needs a bit of set up at the server to map your certificate to a userid.
    • It takes some work to set up how to revoke your access, if you leave the company, or the certificate is compromised.
    • Your private key could be copied and used by hackers. There is discussion about reducing the validity period from over a year to 47 days. For some people this is still too long! You can have your private certificate on a dongle which you have to present when connecting to a back end. This reduces the risk of hackers using your private key.
  • You can connect with a both certificate and userid and password. The certificate is used to establish the TLS session, and the userid and password are used to logon to the application.
  • You can use a pass ticket. You issue a z/OS service which, if authorised, generates a one time password valid for 10 minutes or less. If hackers get hold of the pass ticket, they do not have long to be able to exploit it. The application generating the pass ticket, does not need the password of the userid, because the application has been set up as trusted.
  • You can use a JSON Web Token (JWT). This has some similarities with certificates. In the payload is a userid value and issuer value . I think of issuer as the domain the JWT has come from – it could be TEST or a company name. From the issuer value, and IP address range, you configure the server to specify a realm value. From the userid and realm you can map this to a userid on the server. This JWT can be valid from minutes to many hours (but under a day). The userid and realm mapping to a userid is different to certificate mapping to a userid.

Setting up a pass ticket

The passticket is used within the sysplex. It cannot be used outside of a sysplex. The pass ticket is a password – so needs to be validated against the RACF database.

The application that generates the pass ticket must be authorised to a profile for the application. For example, define the profile for the application TSO on system S0W1, the profile is TSOS0W1.

 RDEFINE PTKTDATA TSOS0W1 

and a profile to allow a userid to create a pass ticket for the application

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE) 

PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(COLIN) ACCESS(UPDATE)
PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(IBMUSER)ACCESS(UPDATE)

Userids COLIN and IBMUSER can issue the callable service IRRSPK00 to generate a pass ticket for a user for the application TSOS0W1.

The output is a one-use password which has a validity of up to 10 minutes.

As an example, you could configure your MQWEB server to use profile name MQWEB, or CSQ9WEB.

How is it used

A typical scenario is for an application running on a work station to issue a request to an “application” on z/OS, like z/OSMF, to generate a pass ticket for a userid and application name.

The client on the work station then issues a request to the back end server, with the userid and pass ticket. If the back end server matches the application name then the pass ticket will be accepted as a password. The logon will fail if a different application is used, so a pass ticket for TSO cannot be used for MQWEB.
This is more secure than sending a userid and password up with every back end request, but there is additional work in creating the pass ticket, and two network flows.

This solution scales because very little work needs to be done on the work station, and there is some one-off work for the setup to generate the pass tickets.

JSON Web Tokens

See What are JSON Web Tokens and how do they work?

The JWT sent from the client has an expiry time. This can be from seconds to hours. I think it should be less than a day – perhaps a couple of hours at most. If a hacker has a copy of the JWT, they can use it until it expires.

The back end server needs to authenticate the token. It could do this by having a copy of the public certificate in the server’s keyring, or send a request down to the originator to validate it.

If validation is being done with public certificates, because the client’s private key is used to generate the JWT, the server needs a copy of the public certificate in the server’s keyring. This can make it hard to manage if there are many clients.

The Liberty web server has definitions like

<openidConnectClient id="RSCOOKIE" 
clientId="COLINCOO2"
realmName="zOSMF"
inboundPropagation="supported"
issuerIdentifier="zOSMF"
mapIdentityToRegistryUser="false"
signatureAlgorithm="RS384"
trustAliasName="CONN1.IZUDFLT"
trustStoreRef="defaultKeyStore"
userIdentifier="sub"
>
<authFilter id="afint">
<remoteAddress id="myAddress" ip="10.1.0.2" matchType="equals" />
</authFilter >

</openidConnectClient>

For this entry to be used various parameters need to match

  • The issuerIdentifier. This string identifies the client. It could be MEGABANK, TEST, or another string of your choice. It has to match what is in the JWT.
  • signatureAlgorithm. This matches the incoming JWT.
  • trustAliasName and trustStoreRef. These identify the certificate used to validate the certificate
  • remoteAddress. This is the address, or address range of the client’s IP addresses.

If you have 1000 client machines, you may need 1000 <openidConnectClient…/> definitions, because of the different certificate and IP addresses.

You may need 1000 entries in the RACMAP mapping of userid + realm to userid to be used on the server.

How is it used

You generate the JWT. There are different ways of doing this.

  • Use a service like z/OSMF
  • Use a service on your work station. I have used Python to do this. The program is 30 lines long and uses the Python jwt package

You get back a long string. You can see what is in the string by pasting the JWT in to jwt.io.
You pass this to the backend as a cookie. The cookie name depends on what the server is expecting. For example

'Authorization': "Bearer " + token

The JWT has limited access

For the server to use the JWT, it needs definitions to recognise it. If you have two back end servers

  • Both servers could be configured to accept the JWT
    • If the server specified a different REALM, then the mapped userid from the JWT could be different for each server because the userid/realm to userid mapping can be different.
  • One server is configured to accept the JWT
    • If only one server has the definitions for the JWT, then trying to use the JWT to logon to another server will fail.

Tracing input and output of the Liberty web server.

The Liberty web server is used by many IBM products on z/OS, for example z/OSMF, MQSeries and z/OSConnect (but not Zowe).

When using Zowe, I struggled finding out what data was input to the server. As usual, when you have found the answer it is easy.

Once it worked, I had

<httpAccessLogging id="accessLogging" 
logFormat="%a %s ET=%D %r i=%i c=%C "
enabled="true"
/>
<httpEndpoint id="defaultHttpEndpoint"
accessLoggingRef="accessLogging"
httpPort="9080"
httpsPort="10443"
/>

Where the httpEndpoint defines the port 10443 , and references httpAccessLogging.

It one point I had two ports defined for https. I separated the output for each port using

filepath="${server.output.dir}/logs/http_10443_access.log" 

within the httpAccessLogging definition, to output the data to a specific file to match the port.

What data is output?

You can control what data is output. I used logFormat to output what I was interested in.

logFormat="%a %s ET=%D %r i=%i c=%C " 

Where

  • %a is the remote IP address 10.1.0.2
  • %s is the status – if it worked the value is 200.
  • ET=%D. This is the duration of the request in microseconds. It appears as ET=667601
  • %r the first line of the request POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.
  • i=%i the header name from the request. My request did not have one so this comes out as i=-
  • c=%C gives the cookies. You can request a specific cookie. My output had c=jwtToken:eyJraWQiOiJhYVBkRzd5N…. which is the JSON Web Token. To see the contents, I took this token, and pasted it into http:jwt.io.

You can ask for the datetime, but this comes out as a long string with year,month, day hh:mm:ss.uuuuuu. I found the year month and day were not needed, but I could not find how to display just the time.

The output from the above format was

10.1.0.2 200 ET=667601 POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.1 i=- c=jwtToken:eyJraWQiOiJhYVBkRzd5NmZETk5UT....

RACF: Processing audit records

RACF can write to SMF data information about which userid logged on, what resources it accessed etc.. This can be used to check there are no unexpected accesses, and any violations are actioned.

The data tends to be “this userid had access to that resource”. It does not contain numeric values, such as response time.

Overview of SMF data

SMF data is a standard across z/OS. Each product has an SMF record type, and record subtypes are used to provide granularity within a product’s records. It is common for an SMF record to have sections within it. There may be 0 or more sections, and the sections can be of varying length. A SMF formatting program needs to build and report useful information from these sections.

There are many tools or products to process SMF records. Individual products may produce tools for formatting records, and there are external tools available to process the records.

Layout of RACF SMF records

The layout of the RACF SMF records are described in the publications. Record type 80: RACF processing record. It describes the field names, at which offsets, and how to interpret the data (what each bit means), this information is sufficient for someone to write a formatting program.


RACF also provides a formatter. The formatter runs as a SORT exit, and expands the data. For example in the SMF data is a bit saying a userid has the SPECIAL attribute. The formatter expands this and creates a column “SPECIAL” with the value YES or NO. This makes it easy to filter and display records, because you do not need to map bits to their meaning – the exit has done it for you. The layout of the expanded records is described here.

What tools format the records?

A common(free) tool for processing the records that RACF produces is an extension to DFSORT called ICETOOL. (The IBM sort modules all begin with ICE… so calling it ICETOOL was natural).

With ICETOOL you can say include rows where…., display and format these fields, count the occurrences of this field, and add page titles. You can quickly generate tabular reports.

The output file of the RACF exit has different format records mixed up. You need to filter by record type and display the subset of records you need.

JCL to extract the RACF SMF record and convert to the expanded format

//* DUMP THE SMF DATASETS 
// SET SMFPDS=SYS1.S0W1.MAN1
// SET SMFSDS=SYS1.S0W1.MAN2
//*
//SMFDUMP EXEC PGM=IFASMFDP,REGION=0M
//DUMPINA DD DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB DD DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT DD DISP=(NEW,PASS),DSN=&RMF,SPACE=(CYL,(1,1))
//OUTDD DD DISP=(NEW,PASS),DSN=&OUTDD,
// SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//ADUPRINT DD SYSOUT=*
//*XMLFORM DD DSN=COLIN.XMLFORM,DISP=(MOD,CATLG),
//* SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//SYSPRINT DD SYSOUT=*

//SYSIN DD *
INDD(DUMPINA,OPTIONS(DUMP))
INDD(DUMPINB,OPTIONS(DUMP))
OUTDD(DUMPOUT, TYPE(30,80,81,83))
START(0000)
END(2359)
DATE(2025230,2025360)
ABEND(NORETRY)
USER2(IRRADU00)
USER3(IRRADU86)
/*

The RACF exits produce several files.

  • //OUTDD the expanded records are written to this dataset
  • //ADUPRINT contains information on how many of each record type the exit processed
  • //XMLFORM you can have it write data in XML format – for post processing

JCL to process the expanded records

The JCL below invokes the ICETOOL processing.

//S1      EXEC  PGM=ICETOOL,REGION=0M 
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//TEMP DD DSN=&&TEMP3,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PRINT DD SYSOUT=*

Where

  • //IN refers to the //OUTDD statement in the earlier step
  • //TEMP is an intermediate dataset. The sort program writes filtered records to this data set.
  • //PRINT is where the formatted output goes

ICETOOL Processing

The whole job is

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.RACF.ICETOOL
// INCLUDE MEMBER=RACFSMF
// INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL
//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -
BLANK -
ON(5,8,CH) HEADER('EVENT') -
ON(63,8,CH) HEADER('USER ID') -
ON(14,8,CH) HEADER('RESULT') -
ON(23,8,CH) HEADER('TIME') -
ON(175,8,CH) HEADER('TERMINAL') -
ON(184,8,CH) HEADER('JOBNAME') -
ON(286,8,CH) HEADER('APPL ')
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
OPTION VLSHRT
//

You have to be careful about the offsets. The record has a 4 byte length field on the front of each record. So the field in the layout of the expanded records described here is column 1 for length 8, in the JCL you specify column 5 of length 8. In the documentation the userid is columns 59 of length 8, in the JCL it is ON(63,8,CH).

The processing is ….

  • Copy the data from the dataset in //IN and copy it to the dataset in //TEMP. Using the sort instructions in TEMPCNTL. You take name name in USING(TEMP) and put CNTL on the end to locate the DDname.
  • The sort instructions say include only those records where columns 5 of length 8 of the record are the string ‘JOBINIT ‘ ( so columns 1 for length 8 in the mapping description).
  • The DISPLAY step copies record from the //TEMP dataset to the //PRINT DDNAME.
  • The ON() selects the data from the record, giving start column, length and formatting. For each field, it uses the specified column heading.

The output

In the //PRINT is

EVENT      USER ID    RESULT     TIME       TERMINAL   JOBNAME    APPL    
-------- -------- -------- -------- -------- -------- --------
JOBINIT START1 SUCCESS 09:54:11 SMFCLEAR
JOBINIT START1 TERM 09:54:20 SMFCLEAR
JOBINIT START1 TERM 09:55:13 CSQ9WEB
JOBINIT IBMUSER SUCCESS 10:12:14 LCL702 IBMUSER
JOBINIT IBMUSER SUCCESS 10:14:18 IBMJOBI
JOBINIT IBMUSER TERM 10:14:18 IBMJOBI
JOBINIT IBMUSER SUCCESS 10:21:39 IBMACCES
JOBINIT IBMUSER TERM 10:21:40 IBMACCES
JOBINIT IBMUSER SUCCESS 10:22:10 IBMACCES
JOBINIT IBMUSER TERM 10:22:11 IBMACCES
JOBINIT IBMUSER SUCCESS 10:23:01 IBMPASST
JOBINIT IBMUSER TERM 10:23:05 IBMPASST

Extending this

Knowing the format of the RACF extend record, you can add more fields to the reports.

You can filter which records you want. For example all records for userid START1. You can link filters with AND and OR statements.

Of course – JCL subroutines is the answer

I was processing RACF SMF records to report how clients were logging into MQ. This as a multi step job, and with each report I added, the JCL got more and more messy.
The requirements were simple

  • JCL to copy the dump the SMF data sets to a temporary data set
  • Run a tool against this data set to product the reports
    • There were reports for logon and logoff, and pass tickets, and access to profiles and…
  • I wanted it to be easy to use – and the JCL to fit on one screen!

All of this was easy except my JCL file got bigger with every report I wanted, and I spent a lot of time scrolling up and down, and changing the wrong file!

The solution was to use JCL subroutines – or INCLUDE JCL.

Examples

JCL to process the SMF data sets

You do not need to know what the JCL does – but you need to know it was in COLIN.JCL(RACFSMF)

//* DUMP THE SMF DATASETS 
// SET SMFPDS=SYS1.S0W1.MAN1
// SET SMFSDS=SYS1.S0W1.MAN3
//*
//SMFDUMP EXEC PGM=IFASMFDP,REGION=0M
//DUMPINA DD DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB DD DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT DD DISP=(NEW,PASS),DSN=&RMF,SPACE=(CYL,(1,1))
//OUTDD DD DISP=(NEW,PASS),DSN=&OUTDD,
// SPACE=(CYL,(1,1)),DCB=(RECFM=VB,LRECL=12288)
//ADUPRINT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
//SYSIN DD *
INDD(DUMPINA,OPTIONS(DUMP))
INDD(DUMPINB,OPTIONS(DUMP))
OUTDD(DUMPOUT, TYPE(30,80,81,83))
START(1040)
END(2359)
DATE(2025229,2025360)
ABEND(NORETRY)
USER2(IRRADU00)
USER3(IRRADU86)

/*

Use it

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.JCL
// INCLUDE MEMBER=RACFSMF

//S1 EXEC PGM=ICETOOL,REGION=1024K
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//JOBI DD DSN=&&TEMPJOBI,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PJOBI DD SYSOUT=*
//TOOLIN DD *
COPY FROM(IN) TO(JOBI) USING(JOBI)
DISPLAY FROM(JOBI) LIST(PJOBI) -
...
//JOBICNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
//

The clever bits are the JCLLIB which gives the JCL library, and the INCLUDE MEMBER=RACFSMF which copies in the JCL.

To use the JOBI content, I needed to specify JOBI, PJOBI and JOBICTL, and similarly for each data component. 10 components meant 30 data sets – all with similar content and names, this lead to a mess of JCL.

Going further, I could use a template with the same data set names, (TEMP, PRINT etc) and just change the content.

I coverted the above JCL to create a member ICETOOL

//S1      EXEC  PGM=ICETOOL,REGION=1024K 
//DFSMSG DD SYSOUT=*
//TOOLMSG DD SYSOUT=*
//IN DD DISP=(SHR,PASS,DELETE),DSN=*.SMFDUMP.OUTDD
//TEMP DD DSN=&&TEMP3,DISP=(NEW,PASS),SPACE=(CYL,(1,1))
//PRINT DD SYSOUT=*

and use it with

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
// JCLLIB ORDER=COLIN.JCL
// INCLUDE MEMBER=RACFSMF
//* INCLUDE MEMBER=PRINT
// INCLUDE MEMBER=ICETOOL

//TOOLIN DD *
COPY FROM(IN) TO(TEMP) USING(TEMP)
DISPLAY FROM(TEMP) LIST(PRINT) -

...
//TEMPCNTL DD *
INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ')
//

Where I just had to change the data in italics – and not the boiler plate.

For each RACF record type, I had a different JCL member, based on the above file.
To select SMF records with a date and time range, I just edited member RACFSMF, and submitted the jobs, and they all used it.

This was easy to do and it let me focus on the problem – rather than on the JCL.