Using the z/OS DNS on ADCD

This came out of a question. It is another of the little questions that get much bigger.

Background to Domain Name System(DNS)

DNS allows you to get an IP address from a string such as “WWW.MY.COM”.

You can have some files on your local system which provide this mapping, or you can exploit DNS Servers in the big internet.

Some people configure their system so it tries the internet first, and if that fails, uses local files.

You can do reverse DNS lookup, mapping an IP address to a string. For example you want to allow access from sites in WWW.MYFRIEND.COM. When a connection is started, you get the IP address, and can then do a reverse DNS lookup to get a name, which you can check in your “allow” list.

DNS commands for the end user

You can use the “old” tso command NSLOOKUP http://www.ibm.com, or the “new” command dig http://www.ibm.com. Neither of which seemed to give me any output!

The NSLOOKUP and DIG commands send their output to SYSOUT. In my TSO system, SYSOUT has been configured to JES. If I use SDSF, and display the output of my TSO userid, there is a SYSOUT, with the output in it!

NSLOOKUP

The NSLOOKUP command

NSLOOKUP http://www.my.com

NSLOOKUP http://www.my.com this.dns.site

NSLOOKUP 10.1.1.2

Tracing a DNS request

This does not provide much useful information! It does not tell you what happened, or what failed. It is described here.

Starting and stopping the DNS

This is not obvious. At IPL the ADCD.Z24C.PARMLIB(BPXPRM00) member has

RESOLVER_PROC(RESOLVER)

the resolver procedure must be in a data set that is specified by the IEFPDSI DD card specification of the MSTJCLxx PARMLIB member.

If you use D A,L it does not show up.

D A,RESOLVER gives you the normal output.

When I issued 

P RESOLVER
S RESOLVER

It used the RESOLVER procedure from USER.Z24C.PROCLIB, the normal concatenation.

Displaying and changing the configuration.

You can display some of the current resolver configuration using

f resolver,display

The output is like

EZZ9298I RESOLVERSETUP - USER.Z24C.TCPPARMS(GBLRESOL)                   
EZZ9298I DEFAULTTCPIPDATA - USER.Z24C.TCPPARMS(GBLTDATA)                
EZZ9298I GLOBALTCPIPDATA - /etc/resolv.conf                             
EZZ9298I DEFAULTIPNODES - ADCD.Z24C.TCPPARMS(ZPDTIPN1)                  
EZZ9298I GLOBALIPNODES - /etc/hosts                                     
EZZ9304I COMMONSEARCH                                                   
EZZ9304I CACHE                                                          
EZZ9298I CACHESIZE - 200M                                               
EZZ9298I MAXTTL - 2147483647                                            
EZZ9298I MAXNEGTTL - 2147483647                                         
EZZ9304I NOCACHEREORDER                                                 
EZZ9298I UNRESPONSIVETHRESHOLD - 25

The only way I could display all of the resolver configuration was to get a resolver trace!

//IBMRESO JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTCPT DD SYSOUT=* 
//SYSPRINT DD SYSOUT=* 
//SYSTSIN DD * 
NSLOOKUP 99.99.99.99 
/*

This gave me in //SYSTCPT

Resolver Trace Initialization Complete -> 2023/02/26 18:01:56.725504                      
res_init Parse error on line     1: /etc/resolv.conf                                      
                                                                                          
res_init Resolver values:                                                                 
 Setup file warning messages = No                                                         
 CTRACE TRACERES option = No                                                              
 Global Tcp/Ip Dataset  = /etc/resolv.conf                                                
 Default Tcp/Ip Dataset = USER.Z24C.TCPPARMS(GBLTDATA)                                    
 Local Tcp/Ip Dataset   = None                                                            
 Translation Table      = TCPIP.STANDARD.TCPXLBIN                                         
 UserId/JobName         = IBMUSER                                                         
 Caller API             = TCP/IP C Sockets                                                
 Caller Mode            = EBCDIC                                                          
 System Name            = S0W1 (from VMCF)                                                
 UnresponsiveThreshold  = 25                                                              
 (D) DataSetPrefix = TCPIP                                                                
 (D) HostName      = S0W1                                                                 
 (D) TcpIpJobName  = TCPIP                                                                
 (*) DomainOrigin  = None                                                                   
 (*) NameServer(s) = None                                                                   
 (*) NsPortAddr    = 53            (*) ResolverTimeout    = 5                               
 (*) ResolveVia    = UDP           (*) ResolverUdpRetries = 1                               
 (*) Options NDots = 1                                                                      
 (D) Trace Resolver                (*) SockNoTestStor                                       
 (D) AlwaysWto     = NO            (D) MessageCase        = MIXED                           
 (*) LookUp        = DNS LOCAL                                                              
 (*) Cache                                                                                  
 (*) NoCacheReorder                                                                         
res_init Succeeded                                                                          
res_init Started: 2023/02/26 18:01:56.794280                                                
res_init Ended: 2023/02/26 18:01:56.794305

This is documented here.

The source of the value is

  • (*) Default value
  • (A) Modified by application
  • (D) Default file (not used if the local file is found)
  • (E) Environment variable
  • (G) Global file
  • (L) Local file

This means the “LookUp = DNS LOCAL ” value came from the default value.

The resolver JCL in USER.Z24C.PROCLIB had

//SETUP DISP=SHR,DSN=USER.Z24C.TCPPARMS(GBLRESOL)

When I changed this member to have LOOKUP LOCAL DSN, and used the F RESOLVER,REFRESH command, this changed the value.

Sample hosts file

The sample host file in TCPIP.SEZAINST(HOSTS) has 

; The format of this file is documented in RFC 952, "DoD Internet 
; Host Table Specification". 
; 
; The format for entries is: 
; 
; NET : ADDR : NETNAME : 
; GATEWAY : ADDR, ALT-ADDR : HOSTNM : CPUTYPE : OPSYS : PROTOCOLS : 
; HOST : ADDR, ALT-ADDR : HOSTNM, NICKNM : CPUTYPE : OPSYS : PROTOCOLS : 
; 
; Where: 
;   ADDR, ALT-ADDR = IP address in decimal, e.g., 26.0.0.73 
;   HOSTNM, NICKNM = the fully qualified host name and any nicknames 
;   CPUTYPE = machine type (PDP-11/70, VAX-11/780, IBM-3090, C/30, etc.) 
;   OPSYS = operating system (UNIX, TOPS20, TENEX, VM/SP, etc.) 
;   PROTOCOLS = transport/service (TCP/TELNET,TCP/FTP, etc.) 
;   : (colon) = field delimiter 
;   :: (2 colons) = null field 
; *** CPUTYPE, OPSYS, and PROTOCOLS are optional fields. 
; 
;   MAKESITE does not allow continuation lines, as described in 
;   note 2 of the section "GRAMMATICAL HOST TABLE SPECIFICATION" 
;   in RFC 952.  Entries should be specified on a single line of 
;   up to a maximum of 512 characters per line. 
HOST : 129.34.128.245, 129.34.128.246 : YORKTOWN, WATSON :::: 
; 
NET  : 9.67.43.0 : RALEIGH.IBM.COM : 
; 
GATEWAY : 129.34.0.0 : YORKTOWN-GATEWAY ::::

Unix application trace

Enable the trace by issuing the Unix command

export RESOLVER_TRACE=~/trace

Run the command

pip install mfpandas

gave

-[33mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x500B209580>: 
Failed to establish a new connection: 
[Errno 1] EDC9501I The name does not resolve for the supplied parameters.')': /simple/mfpandas/-[0m-[33m

Look at the trace file

oedit trace

gave

GetAddrInfo Started: 2025/11/25 15:58:36.890589 
GetAddrinfo Invoked with following inputs: 
   Host Name:  pypi.org 
   Service Name:  443 
   Hints parameter supplied with settings: 
       ai_family = 0, ai_flags = 0x00000000 
       ai_protocol = 0, ai_socktype = 1 
No NameServers specified, no DNS activity 
GetAddrInfo Opening Socket for IOCTLs 
 BPX1SOC:  RetVal = 0, RC = 0, Reason = 0x00000000, Type=IPv4 
 BPX1IOC:  RetVal = 0, RC = 0, Reason = 0x00000000 
GetAddrInfo Opened Socket 0x00000005 
GetAddrInfo Only IPv4 Interfaces Exist 
GetAddrInfo Searching Local Tables for IPv6 Address 
Global IpNodes Dataset  = ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
Default IpNodes Dataset = ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
Search order            = CommonSearch 
 BPX1ENV Get _BPXK_AUTOCVT:  RetVal = 0, RC = 0, Reason = 0x00000000 
_BPXK_AUTOCVT current value is ON 
 BPX1ENV Set _BPXK_AUTOCVT:  RetVal = 0, RC = 0, Reason = 0x00000000 
_BPXK_AUTOCVT set to OFF 
  Parse error on line   22: ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
 SITETABLE from globalipnodes ADCD.Z31B.TCPPARMS(ZPDTIPN1) 
 - Lookup for pypi.org 
GetAddrInfo Searching Local Tables for IPv4 Address 
 - Lookup for pypi.org 
GetAddrInfo Searching Local Tables for IPv6 Address 
 - Lookup for pypi.org.DAL-EBIS.IHOST.COM 
GetAddrInfo Searching Local Tables for IPv4 Address 
 - Lookup for pypi.org.DAL-EBIS.IHOST.COM 
GetAddrInfo Closing IOCTL Socket 0x00000005 
 BPX1CLO:  RetVal = 0, RC = 0, Reason = 0x00000000 
GetAddrInfo Failed:  RetVal = -1, RC = 1, Reason = 0x78AE1004 
GetAddrInfo Ended: 2025/11/25 15:58:36.904995 

EDIT       /etc/hosts
Command ===>                                                      
****** ******************************************************* Top
==MSG> -Warning- The UNDO command is not available until you chang
==MSG>           your edit profile using the command RECOVERY ON. 
000001 # BEGIN ANSIBLE MANAGED BLOCK 
000002 #72.26.1.2  s0w1.dal-ebis.ihost.com S0W1 
000003 127.0.0.1 localhost 
000004 # END ANSIBLE MANAGED BLOCK 
000005 #IPAddress             Hostname   alias 
000006 151.101.128.223        pypi.org    pip

Why can’t I connect to a z/OS port?

I’ve found couple of those little problems which took me a day to resolve – but which are obvious when you understand the problem.

The problems

I was trying to connect the Health Center in Eclipse to the Health agent in Liberty on z/OS.

The first problem was the health center agent on z/OS could not connect to the port. This was due to bad TCPIP configuration

The second problem was I could not connect to it from Eclipse. I had configured the port to be on the local rather than external interface.

My setup

In my jvm.options I had

-Xhealthcenter:level=off,readonly=off,jmx=on,port=1972

Problem 1: The health center agent on z/OS could not connect to the port

In the Liberty startup output I received (after about a timeout of about a minute)

SEVERE: Health Center agent failed to start. java.io.IOException: Cannot bind to URL [rmi://S0W1:1972/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host:

Where my system is called S0W1.

It is trying to connect to system S0W1 port 1972, and failing.

TSO PING S0W1 gave

 CS 3.1: Pinging host S0W1.DAL-EBIS.IHOST.COM (172.26.1.2)
 Ping #1 timed out

This was a surprise to me – I was expecting it to be my local z/OS machine…. I do not have an interface with address 172.26.1.2. This explains why it timed out.

In my ADCD.Z31B.TCPPARMS(GBLTDATA) I had 

S0W1:   HOSTNAME   S0W1 
; 
; 
; NOTE - Use either DOMAINORIGIN/DOMAIN or SEARCH to specify your domain 
;         origin value 
; 
; DOMAINORIGIN or DOMAIN statement 
; ================================ 
; DOMAINORIGIN or DOMAIN specifies the domain origin that will be 
; appended to host names passed to the resolver.  If a host name 
; ends with a dot, then the domain origin will not be appended to the 
; host name. 
; 
DOMAINORIGIN  DAL-EBIS.IHOST.COM

Because S0W1 did not end with a dot – TCPIP put the DOMAINORIGIN on the end.

ADCD.Z31B.TCPPARMS(ZPDTIPN1)

had

172.26.1.2 S0W1.DAL-EBIS.IHOST.COM S0W1      
127.0.0.1       LOCALHOST

Which says for S0W1…. use IP address 172.26.1.2.

I changed this to

S0W1:   HOSTNAME   S0W1.
   127.0.0.1       S0W1        
   127.0.0.1       LOCALHOST

With these changes, I restarted TCPIP, and told the resolver to use the updated configuration.

F RESOLVER,REFRESH,SETUP=ADCD.Z31b.TCPPARMS(GBLRESOL)

I then got 

INFO: Health Center agent started on port 1972.

So my first success. However…

Problem 2 : I could not connect Eclipse to the port

… once I had managed to get get the server to connect to the port. When the server issues a TCPIP binds to a port, you need to specify the IP address and port. I had configured the hostname S0W1 as the local interface (127.0.0.1). When I tried to connect from Eclipse, I was trying to connect to port 1972 on interface 10.1.1.2 – which had not been configured!

The Liberty output had

WARNING: RMI TCP Accept-1972: accept loop for erverSocket[addr=0.0.0.0/0.0.0.0, localport=1972] throws java.io.IOException: EDC5122I Input/output error. (errno2=0x12B804B9)

I changed ADCD.Z31B.TCPPARMS(ZPDTIPN1) to have

10.1.1.2 S0W1
127.0.0.1 LOCALHOST

so the name S0W1 is associated with interface 10.1.1.2. I started restart TCPIP and the resolver and it manage to connect. It only took a day to resolve these problems.

Why can’t Liberty see my bootstrap.properties? The answer is in the title!

I put the osgi address and port information into the bootstrap.properties file, but it wasn’t being picked up. A couple of hours later I found out why – the answer is that Liberty could see my bootstrap.properties file.

The command

ls -lT bootstrap.properties

gave

- untagged    T=off -rwx------   1 OMVSKERN SYS1 ... bootstrap.properties

which means the started task running Liberty did not have access to it, only the owner had rwx access. Java did not process it, because it could not see it.

I used the command

chmod 744 bootstrap.properties

and next time I started the Liberty instance, it could find the file.

My file had

osgi.console = 10.1.1.2:5400

The TSO command

tso netstat allconn (port 5400

saying – show all the connections which are, or were recently active, filtered by port 5400

gave

User Id  Conn     State
-------  ----     -----
CSQ9WEB  00000139 Listen
  Local Socket:   ::ffff:10.1.1.2..5400
  Foreign Socket: ::ffff:0.0.0.0..0

Showing it is active.

Understanding LTPA tokens for accessing a web site.

What is an LTPA token? – the short answer

When a client connects to a web server and logs on (for example with a userid and password), the server can send back a cookie containing encrypted logon information.

When the client sends this cookie in the next request, the server decrypts it, and so shortens the logon process. Eventually the token will expire and the client will need to logon again.

What is an LTPA token? – a longer answer.

There is a long (perhaps too long) description of LTPA here.

The Lightweight Third-Party Authentication says When accessing web servers that use the LTPA technology it is possible for a web user to re-use their login across physical servers.

The server has an encryption key which it uses to encrypt the “reuse” content. This key could change every time the server start, in fact it is good practice to change the key “often”.
For Liberty the information is configured in the <ltpa…/> tag. See Configuring LTPA in Liberty.

If a client has an LTPA2 token, and the server restarts, if they encryption keys are the same as before, the LTPA2 will be accepted (providing it hasn’t expired). If they encryption key has changed, the client will need to logon to get a new LTPA.

How to configure LTPA?

The ltpa tag in server.xml looks like

<ltpa keysFileName="yourLTPAKeysFileName.keys" 
      keysPassword="keysPassword" 
      expiration="120" />

Where

  • keysFileName defaults to ${server.output.dir}/resources/security/ltpa.keys Note: This is a file, not a keyring.
  • expiration defaults to 120 minutes.

Often password or key data is just base64 encoded, so it trivial to decode. You can encrypt these using the Liberty securityUtility command. The createLTPAKeys option creates a set of LTPA keys for use by the server, or that can be shared with multiple servers. If no server or file is specified, an ltpa.keys file is created in the current working directory.

Single server mode

If the web server is isolated and has its own encryption key then the LTPA can be passed to the server. It can use its encryption key to decrypt the userid information and use it.

If you try to use the LTPA on another web server, which has a different encryption key, the decryption will fail, and the LTPA cannot be used.

Sharing an encryption key

If multiple web servers share the same key, then the LTPA can be used on those servers. For example, you have multiple back-end servers for availability, and a work request can be routed to any server. Once the client has logged on and got the LTPA, future requests can be routed to any of the servers, without the need to logon. The LTPA can be decrypted because of the shared key.

Does this give a different access?

If MQWEB and z/OSMF share the same encryption key, a client logs on to MQWEB and gets a LTPA. The client then uses the LTPA to logon to z/OSMF. All this does is replace the userid and password. MQWEB and z/OSMF still have to determine what permissions the userid has. LTPA does not affect the access.

What happens when the LTPA expires?

Using CURL to send a request to MQWEB with an expired LTPA, I got

 
< HTTP/2 401 
...
* Replaced cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
* Added cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
* Added cookie LtpaToken2_8443="""" for domain 10.1.1.2, path /, expire 786297600
< set-cookie: LtpaToken2_8443=""; Expires=Thu, 01 Dec 1994 16:00:00 GMT; Path=/; Secure; HttpOnly
...
{"error": [{
  "msgId": "MQWB0112E",
  "action": "Login to the REST API to obtain a valid authentication cookie.",
  "completionCode": 0,
  "reasonCode": 0,
  "type": "rest",
  "message": "MQWB0112E: The 'LtpaToken2_8443' authentication token cookie failed verification.",
  "explanation": "The REST API request cannot be completed because the authentication token failed verification."

Where MQ uses the token name LtpaToken2_${httpsPort}, and so has the https port in it.

Because the Ltpa token had expired, a null token was stored in the cookie.

MQ returned a message giving an explanation. Each product will be different.

Tracing LTPA2

Using a LTPA2 token.

I had problems with LTPA2 – I thought an LTPA2 token would expire – but it continued to work.
A Liberty trace with

com.ibm.ws.security.token.ltpa.internal.LTPAToken2=finest

gave me a lot more data which was not useful – but it did give me

LTPAToken2 3 Current time = Wed Aug 27 17:21:31 GMT 2025, expiration time = Wed Aug 27 19:16:23 GMT 2025

This shows the LTPA was valid from 17:21 to 19:16 or 115 minutes remaining.

Logging on and creating a LTPA token

With

com.ibm.ws.security.token.ltpa.internal.LTPAToken2=finer

I got lots of data including

00000055 LTPAToken2    >  <init> Entry 
  user:ADCDPL/COLIN 
  120

Which shows the userid, and the expiry interval in minutes

When the LTPA had expired

Current time = Thu Aug 28 06:51:35 GMT 2025, expiration time = Wed Aug 27 19:42:20 GMT 2025  
The token has expired: current time = Thu Aug 28 06:51:35 GMT 2025, expire time = Wed Aug 27 19:42:20 GMT 2025.

What RACF audit records are produced with pass tickets?

A pass ticket is a one time password for a userid, valid with the specified application. I’ve blogged Creating and using pass tickets on z/OS.

I’ve also blogged Of course – JCL subroutines is the answer about using ICETOOL to process RACF audit records in SMF.

Create a pass ticket

After I had created the pass ticket, I used the following JCL below to format the RACF SMF PTCREATE record.

//IBMPTICK JOB 1,MSGCLASS=H RESTART=PRINT 
//         JCLLIB ORDER=COLIN.RACF.ICETOOL 
// INCLUDE MEMBER=RACFSMF 
//* INCLUDE MEMBER=PRINT 
// INCLUDE MEMBER=ICETOOL 
//TOOLIN    DD  * 
 COPY    FROM(IN)  TO(TEMP) USING(TEMP) 
 DISPLAY FROM(TEMP) LIST(PRINT) - 
         BLANK - 
         ON(63,8,CH)  HEADER('USER ID') -
         ON(184,8,CH)  HEADER('FROMJOB') -  
         ON(14,8,CH)  HEADER('RESULT') - 
         ON(23,8,CH)  HEADER('TIME') - 
         ON(286,8,CH) HEADER('APPL   ') - 
         ON(295,8,CH) HEADER('FORUSER  ')
//TEMPCNTL DD * 
   INCLUDE COND=(5,8,CH,EQ,C'PTCREATE') 
   OPTION  VLSHRT 
//

to produce

USER ID    FROMJOB    RESULT     TIME       APPL       FORUSER     USER NAME   
--------   --------   --------   --------   --------   ---------   ------------
ZWESVUSR   ZWE1AZ     SUCCESS    15:00:55   MQWEB      COLIN       ZOWE SERVER

Which shows from Job ZWE1AZ running with userid ZWESVUSR; it successfully created a pass ticket for userid COLIN with application MQWEB.

Show where the pass ticket is used

Once the pass ticket had been used, I used the following JCL to display the JOBINIT audit record.

//IBMJOBI  JOB 1,MSGCLASS=H RESTART=PRINT 
//         JCLLIB ORDER=COLIN.RACF.ICETOOL 
// INCLUDE MEMBER=RACFSMF 
//* INCLUDE MEMBER=PRINT 
// INCLUDE MEMBER=ICETOOL 
//TOOLIN    DD  * 
 COPY    FROM(IN)  TO(TEMP) USING(TEMP) 
 DISPLAY FROM(TEMP) LIST(PRINT) - 
         BLANK - 
         ON(63,8,CH)   HEADER('USER ID ') - 
         ON(14,8,CH)   HEADER('RESULT  ') - 
         ON(23,8,CH)   HEADER('TIME    ') - 
         ON(184,8,CH)  HEADER('JOBNAME ') - 
         ON(286,8,CH)  HEADER('APPL    ') - 
         ON(631,8,CH)  HEADER('SESSTYPE')- 
         ON(4604,4,CH) HEADER('PTOEVAL ') - 
         ON(4609,4,CH) HEADER('PSUCC   ') 
//TEMPCNTL DD * 
   INCLUDE COND=(5,8,CH,EQ,C'JOBINIT ') 
   OPTION  VLSHRT 
//

it produced the output

USER ID    RESULT     TIME       JOBNAME    APPL       SESSTYPE   PTOEVAL    PSUCC 
--------   --------   --------   --------   --------   --------   --------   -------- 
COLIN      SUCCESSP   15:01:02   CSQ9WEB    MQWEB      OMVSSRV    YES        YES 
COLIN      RACINITD   15:01:02   CSQ9WEB    MQWEB      OMVSSRV    NO         NO

The first record shows,

  • in job CSQ9WEB,
  • running with APPLication id of MQWEB.
  • Sesstype OMVSSVR is a z/OS UNIX server application. See RACROUTE TYPE=VERIFY under SESSION=type.
  • userid COLIN SUCCCESSfully logged on with Passticket (SUCCESSP)
  • PTOEVAL – YES the supplied password was evaluated as a PassTicket,
  • PSUCC – YES the supplied password was evaluated successfully as a PassTicket.

The second record shows RACINITD (Successful RACINIT deletion) for the userid COLIN in the job CSQ9WEB, and the password was not used.

Zowe: Getting data from Zowe

As part of an effort to trace the https traffic from Zowe, I found there are trace points you can enable.

You can get a list of these from a request like “https://10.1.1.2:7558/application/loggers&#8221;. In the browser it returns one long string like (my formatting)

{"levels":["OFF","ERROR","WARN","INFO","DEBUG","TRACE"],
"loggers":{"ROOT":{"configuredLevel":"INFO","effectiveLevel":"INFO"},
"_org":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework":{"configuredLevel":null,"effectiveLevel":"INFO"},
"_org.springframework.web":{"configuredLevel":null,"effectiveLevel":"INFO"},
...

Once you know the trace point, you can change it. See here.

Using https module

certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
url="https://10.1.1.2:7558/application/loggers"
https GET ${url}  $certs  $verify

This displayed the data, nicely formatted. But if you pipe it, the next stage receives one long character string.

Using Python

#!/usr/bin/env python3

import ssl
import json
import sys
from http.client import HTTPConnection 
import requests
import urllib3
# trace the traffic flow
HTTPConnection.debuglevel = 1

my_header = {  'Accept' : 'application/json' }

urllib3.disable_warnings()
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

certificate="colinpaice.pem"
key="colinpaice.key.pem"
cpcert=(certificate,key)
jar = requests.cookies.RequestsCookieJar()

s = requests.Session()
geturl="https://10.1.1.2:7558/application/loggers"

res = s.get(geturl,headers=my_header,cookies=jar,cert=cpcert,verify=False)

if res.status_code != 200:
    print("error code",res.status_code)
    sys.exit(8)

headers = res.headers

for h in headers:
    print(h,headers[h])

cookies = res.cookies.get_dict()
for c in cookies:
    print("cookie",c,cookies[c])

js = json.loads(res.text)
print("type",js.keys())
print(js['levels'])
print(js['groups'])
loggers = js['loggers']
for ll in loggers:
    print(ll,loggers[ll])

This prints out one line per item.

The command

python3  zloggers.py |grep HTTP

gives

...
org.apache.http {'configuredLevel': 'DEBUG', 'effectiveLevel': 'DEBUG'}
org.apache.http.conn {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
org.apache.http.conn.ssl {'configuredLevel': None, 'effectiveLevel': 'DEBUG'}
...

My WLM definitions were not behaving as I expected.

I had configured WLM so the MQ started tasks (CSQ*) were defined as a low priority STC.

  Subsystem-Type  Xref  Notes  Options  Help                              
--------------------------------------------------------------------------
                 Modify Rules for the Subsystem Type    Row 22 to 25 of 25
Command ===> ___________________________________________ Scroll ===> CSR
                                                                          
Subsystem Type . : STC         Fold qualifier names?   Y  (Y or N)        
Description  . . . All Started Tasks                                      
                                                                          
Action codes:   A=After     C=Copy        M=Move     I=Insert rule        
                B=Before    D=Delete row  R=Repeat   IS=Insert Sub-rule   
                                                             More ===>    
         -------Qualifier--------                 -------Class--------    
Action   Type      Name     Start                  Service     Report     
                                         DEFAULTS: STCLOM      ________   
 ____  1 TN        CSQ9WEB  ___                    STCLOM      MQ         
 ____  1 TN        CSQ9CHIN ___                    STCLOM      MQ         
 ____  1 TN        CSQ9ANG  ___                    STCLOM      MQ

But I could see from SDSF, that the CSQ9CHIN’s SrvClass was STCHIM, and CSQ9WEB’s was STCHIM. It took me a couple of hours digging to find out why.

Higher up the list, the WLM definitions had

         -------Qualifier--------                 -------Class--------    
Action   Type      Name     Start                  Service     Report     
                                         DEFAULTS: STCLOM      ________   
 ____  1 TN        %MASTER% ___                    SYSTEM      MASTER     
 ____  1 SPM       SYSTEM   ___                    SYSTEM      ________   
 ____  1 SPM       SYSSTC   ___                    SYSSTC      ________   
 ____  1 TNG       STCHI    ___                    SYSSTC      ________   
 ____  1 TNG       STCMD    ___                    STCMDM      ________   
 ____  1 TNG       MONITORS ___                    ________    MONITORS   
 ____  1 TNG       SERVERS  ___                    STCMDM      ________   
 ____  1 TNG       ONLPRD   ___                    STCHIM      ________

There is a definition for ONLPRD (online production), a group of transaction names (Transaction Name Group).

From option 5 Classification Groups, of the main WLM panel it displays

                         Classification Group Menu                        
 Select one of the following options.                                       
 __  1.  Accounting Information Groups   14. Plan Name Groups              
     2.  Client Accounting Info Groups   15. Procedure Name Groups         
     3.  Client IP Address Groups        16. Process Name Groups           
     4.  Client Transaction Name Groups  17. Scheduling Environment Groups 
     5.  Client Userid Groups            18. Subsystem Collection Groups   
     6.  Client Workstation Name Groups  19. Subsystem Instance Groups     
     7.  Collection Name Groups          20. Subsystem Parameter Groups    
     8.  Connection Type Groups          21. Sysplex Name Groups           
     9.  Correlation Information Groups  22. System Name Groups            
     10. LU Name Groups                  23. Transaction Class Groups      
     11. Net ID Groups                   24. Transaction Name Groups       
     12. Package Name Groups             25. Userid Groups                 
     13. Perform Groups                  26. Container Qualifier Groups

Most of these had no definition, but option 24. Transaction Name Groups gave me

                           Group Selection List                Row 1 to 5 of 5
Command ===> ____________________________________________________________     
                                                                              
Qualifier type . . . . . . . : Transaction Name                               
                                                                              
Action Codes: 1=Create, 2=Copy, 3=Modify, 4=Browse, 5=Print, 6=Delete,        
              /=Menu Bar                                                      
                                                    -- Last Change ---        
Action  Name      Description                       User      Date            
  __    MONITORS  Online System Activity monitors   TODD      1999/11/16      
  __    ONLPRD    Online Production Subsystems      IBMUSER   2023/01/10      
  __    SERVERS   Server Address Spaces             TODD      1999/11/16      
  __    STCHI     High STC's                        TODD      1999/11/16      
  __    STCMD     Medium STC's                      TODD      1999/11/16

and these names match what is in the classification rules section above.

Option 3 to modify ONLPRD, gave

                              Modify a Group                   Row 1 to 8 of 8
Command ===> ____________________________________________________________     
                                                                              
Enter or change the following information:                                    
                                                                              
Qualifier type . . . . . . . : Transaction Name                               
Group name . . . . . . . . . : ONLPRD                                         
Description  . . . . . . . . . Online Production Subsystems                   
Fold qualifier names?  . . . . Y  (Y or N)                                    
                                                                              
Qualifier Name  Start  Description                                            
%%%%DBM1        ___    DB2 Subsystems                                         
%%%%MSTR        ___    DB2 Subsystems                                         
%%%%DIST        ___    DB2 Subsystems                                         
%%%%SPAS        ___    DB2 Subsystems                                         
CICS*           ___    CICS Online Systems                                    
IMS*            ___    IMS Online Systems                                     
CSQ*            ___    MQ Series

and we can see that MQ started tasks starting with CSQ are in this group.

As this definition is higher in the classification rules list – it will take precedence over any definitions I had defined lower down.

Because there was a definition (within the Started classification)

____  1 TNG       ONLPRD   ___                    STCHIM      ________

Started tasks in the group ONLPRD are classified as STCHIM, and so this explains why the classification of the MQ address spaces were “wrong”.

I had several options

  • Change the groups and put MQ in its own group with STCLOM
  • Move my CSQ9* specific definitions above the group.

What’s the best way of connecting to an HTTPS server. Pass ticket or JWT?

This blog post was written as background to some blog posts on Zowe API-ML. It provides back ground knowledge for HTTPS servers running on z/OS, and I think it is useful on its own. Ive written about an MQWEB server – because I have configured this on my system.

The problem

I want to manage my z/OS queue manager from my Linux machine.I have several ways of doing it.

Which architecture?

  • Use an MQ client. Establish a client connect to the CHINIT, and use MQPUT and MQGET administration messages to the queue manager.
    • You can issue a command string, and get back a response string which you then have to parse
    • You can issue an MQINQ API request to programmatically query attributes, and get the values back in fields. No parsing, but you have to write a program to do the work.
  • Use the REST API. This is an HTTP request in a standard format into the MQWEB server.
    • You can issue a command string, and get back a response string which you then have to parse to extract the values.
    • You can issue a JSON object where the request is encoded in a URL, and get the response back in JSON format. It is trivial to extract individual fields from the returned data.

Connecting to the MQWEB server

If you use REST (over HTTPS) there are several ways of doing this

  • You can connect using userid and password. It may be OK to enter your password when you are at the keyboard, but not if you are using scripts and you may be away from your keyboard. If hackers get hold of the password, they have weeks to use it, before the password expires. You want to give your password once per session, not for every request.
  • You can connect using certificates, without specifying userid and password.
    • It needs a bit of set up at the server to map your certificate to a userid.
    • It takes some work to set up how to revoke your access, if you leave the company, or the certificate is compromised.
    • Your private key could be copied and used by hackers. There is discussion about reducing the validity period from over a year to 47 days. For some people this is still too long! You can have your private certificate on a dongle which you have to present when connecting to a back end. This reduces the risk of hackers using your private key.
  • You can connect with a both certificate and userid and password. The certificate is used to establish the TLS session, and the userid and password are used to logon to the application.
  • You can use a pass ticket. You issue a z/OS service which, if authorised, generates a one time password valid for 10 minutes or less. If hackers get hold of the pass ticket, they do not have long to be able to exploit it. The application generating the pass ticket, does not need the password of the userid, because the application has been set up as trusted.
  • You can use a JSON Web Token (JWT). This has some similarities with certificates. In the payload is a userid value and issuer value . I think of issuer as the domain the JWT has come from – it could be TEST or a company name. From the issuer value, and IP address range, you configure the server to specify a realm value. From the userid and realm you can map this to a userid on the server. This JWT can be valid from minutes to many hours (but under a day). The userid and realm mapping to a userid is different to certificate mapping to a userid.

Setting up a pass ticket

The passticket is used within the sysplex. It cannot be used outside of a sysplex. The pass ticket is a password – so needs to be validated against the RACF database.

The application that generates the pass ticket must be authorised to a profile for the application. For example, define the profile for the application TSO on system S0W1, the profile is TSOS0W1.

 RDEFINE PTKTDATA TSOS0W1

and a profile to allow a userid to create a pass ticket for the application

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE) 

PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(COLIN) ACCESS(UPDATE) 
PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(IBMUSER)ACCESS(UPDATE)

Userids COLIN and IBMUSER can issue the callable service IRRSPK00 to generate a pass ticket for a user for the application TSOS0W1.

The output is a one-use password which has a validity of up to 10 minutes.

As an example, you could configure your MQWEB server to use profile name MQWEB, or CSQ9WEB.

How is it used

A typical scenario is for an application running on a work station to issue a request to an “application” on z/OS, like z/OSMF, to generate a pass ticket for a userid and application name.

The client on the work station then issues a request to the back end server, with the userid and pass ticket. If the back end server matches the application name then the pass ticket will be accepted as a password. The logon will fail if a different application is used, so a pass ticket for TSO cannot be used for MQWEB.
This is more secure than sending a userid and password up with every back end request, but there is additional work in creating the pass ticket, and two network flows.

This solution scales because very little work needs to be done on the work station, and there is some one-off work for the setup to generate the pass tickets.

JSON Web Tokens

See What are JSON Web Tokens and how do they work?

The JWT sent from the client has an expiry time. This can be from seconds to hours. I think it should be less than a day – perhaps a couple of hours at most. If a hacker has a copy of the JWT, they can use it until it expires.

The back end server needs to authenticate the token. It could do this by having a copy of the public certificate in the server’s keyring, or send a request down to the originator to validate it.

If validation is being done with public certificates, because the client’s private key is used to generate the JWT, the server needs a copy of the public certificate in the server’s keyring. This can make it hard to manage if there are many clients.

The Liberty web server has definitions like

<openidConnectClient id="RSCOOKIE" 
  clientId="COLINCOO2" 
  realmName="zOSMF" 
  inboundPropagation="supported" 
  issuerIdentifier="zOSMF" 
  mapIdentityToRegistryUser="false" 
  signatureAlgorithm="RS384" 
  trustAliasName="CONN1.IZUDFLT" 
  trustStoreRef="defaultKeyStore" 
  userIdentifier="sub" 
 > 
<authFilter id="afint"> 
  <remoteAddress id="myAddress" ip="10.1.0.2" matchType="equals" /> 
</authFilter         > 
                                                                                                 
</openidConnectClient>

For this entry to be used various parameters need to match

  • The issuerIdentifier. This string identifies the client. It could be MEGABANK, TEST, or another string of your choice. It has to match what is in the JWT.
  • signatureAlgorithm. This matches the incoming JWT.
  • trustAliasName and trustStoreRef. These identify the certificate used to validate the certificate
  • remoteAddress. This is the address, or address range of the client’s IP addresses.

If you have 1000 client machines, you may need 1000 <openidConnectClient…/> definitions, because of the different certificate and IP addresses.

You may need 1000 entries in the RACMAP mapping of userid + realm to userid to be used on the server.

How is it used

You generate the JWT. There are different ways of doing this.

  • Use a service like z/OSMF
  • Use a service on your work station. I have used Python to do this. The program is 30 lines long and uses the Python jwt package

You get back a long string. You can see what is in the string by pasting the JWT in to jwt.io.
You pass this to the backend as a cookie. The cookie name depends on what the server is expecting. For example

'Authorization': "Bearer " + token

The JWT has limited access

For the server to use the JWT, it needs definitions to recognise it. If you have two back end servers

  • Both servers could be configured to accept the JWT
    • If the server specified a different REALM, then the mapped userid from the JWT could be different for each server because the userid/realm to userid mapping can be different.
  • One server is configured to accept the JWT
    • If only one server has the definitions for the JWT, then trying to use the JWT to logon to another server will fail.

Tracing input and output of the Liberty web server.

The Liberty web server is used by many IBM products on z/OS, for example z/OSMF, MQSeries and z/OSConnect (but not Zowe).

When using Zowe, I struggled finding out what data was input to the server. As usual, when you have found the answer it is easy.

Once it worked, I had

<httpAccessLogging id="accessLogging" 
  logFormat="%a %s ET=%D %r i=%i c=%C " 
  enabled="true" 
/> 
<httpEndpoint id="defaultHttpEndpoint" 
  accessLoggingRef="accessLogging" 
  httpPort="9080" 
  httpsPort="10443" 
  />

Where the httpEndpoint defines the port 10443 , and references httpAccessLogging.

It one point I had two ports defined for https. I separated the output for each port using

filepath="${server.output.dir}/logs/http_10443_access.log"

within the httpAccessLogging definition, to output the data to a specific file to match the port.

What data is output?

You can control what data is output. I used logFormat to output what I was interested in.

logFormat="%a %s ET=%D %r i=%i c=%C "

Where

  • %a is the remote IP address 10.1.0.2
  • %s is the status – if it worked the value is 200.
  • ET=%D. This is the duration of the request in microseconds. It appears as ET=667601
  • %r the first line of the request POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.
  • i=%i the header name from the request. My request did not have one so this comes out as i=-
  • c=%C gives the cookies. You can request a specific cookie. My output had c=jwtToken:eyJraWQiOiJhYVBkRzd5N…. which is the JSON Web Token. To see the contents, I took this token, and pasted it into http:jwt.io.

You can ask for the datetime, but this comes out as a long string with year,month, day hh:mm:ss.uuuuuu. I found the year month and day were not needed, but I could not find how to display just the time.

The output from the above format was

10.1.0.2 200 ET=667601 POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.1 i=- c=jwtToken:eyJraWQiOiJhYVBkRzd5NmZETk5UT....