Secure store aren’t

Applications such as Zowe can store secure information on the end user’s machine. This is not very secure! It is built into the operating systems. It is a bit like securing a door with a bit of string. Joshua Waters pointed out

The fact of the matter is that regardless of whether or not you are storing your credentials on a machine, if there is a virus or malicious actor on it, your credentials are up for grabs while the user is logged in. The only time they wouldn’t be up for grabs is if you were using an application that either require some master key to access the credentials store for it, or every authed request to the server requires user to re-enter credentials.

On Linux

The information is in the gnome-keyring ~/.local/share/keyrings/login.keyring .

You can use the Linux command seahorse to display the contents of the gnome-keyring. The user’s password is used to decrypt the store.
The following python code display the keyring contents

import secretstorage
conn = secretstorage.dbus_init()
collection = secretstorage.get_default_collection(conn)
for item in collection.get_all_items():
    print('='*30)
    print('label:', item.get_label())
    print('attributes:')
    for k,v in item.get_attributes().items():
        print('\t%-12s: %s' % (k,v))
    print('secret:',item.get_secret())

This gave

label: Zowe/secure_config_props
attributes:
account : secure_config_props
service : Zowe
xdg:schema : org.freedesktop.Secret.Generic
secret: b'eyIva...9fQ=='

The secret is based64 encoded. You can decode it (on Linux) with

base64 -d <<<"eyIva...9fQ=="  

This gave

{"/home/colinpaice/ssl/ssl2/zowe.config.json":
{"profiles.project_base.properties.user":"colin",
"profiles.project_base.properties.password":"password"
}
}

Where /home/colinpaice/ssl/ssl2/zowe.config.json is the name of the configuration file it applies to.

You can delete an entry using

import secretstorage
conn = secretstorage.dbus_init()
collection = secretstorage.get_default_collection(conn)
for item in collection.get_all_items():
print('='*30)
print('label:', item.get_label())
if item.get_label() == "Zowe/secure_config_props":
item.delete()
print("delete")
continue

This deletes all of the entries for that component – so all the Zowe data.

Who can see the contents of the store?

Your gnome-keyring is encrypted with your password, so you can access it. Someone one else would need your password to be able to decrypt it and see the contents.

What happens on other platforms?

On Windows and Mac’s it is essentially the same. There is a secure disk, and you need to be running as the owner to access it.

If your machine is infected with a virus, which runs under your userid, it can access the key stores and so get userid and password information store in the “secure store”.

Zowe cli help command is not helpful!

The zowe cli help option does not easily tell you how to get all of the help. In order to get the syntax of the command – you have to know the full command with the and then add the --help option! (This is working as designed!)

There is some online help here in a tree view or a “flat view of all of the commands“.

Whoops profile options not found


Step 1

The command zowe --help gives output including

USAGE
zowe <group>

Where <group> is one of the following:

GROUPS
auth Connect to Zowe API ML authentication service
config Manage JSON project and global configuration
zos-console | console Issue z/OS console commands and collect responses

...

Step 2

Now you know there is a console command….

The command zowe --help console gives output including

 USAGE

zowe zos-console <group>

Where <group> is one of the following:

GROUPS

collect Collect z/OS console command responses
issue Issue z/OS console commands

Step 3

Now you know there is a console issue command…

The command zowe --help console issue finally gives lots of output including

  • OPTIONS
    • --console-name | --cn | -c
    • --include-details | --id | -i
    • --key-only | --ko | -k (boolean)
    • --return-first | --rf | -r (boolean)
    • --solicited-keyword | --sk | -s (string)
  • ZOSMF CONNECTION OPTIONS
    • --host | -H (string) The z/OSMF server host name.
    • --port | -P (number) The z/OSMF server port. Default value: 443
    • --user | -u (string) Mainframe (z/OSMF) user name, which can be the same as your TSO login. Your TSO logon userid
    • --password | --pass | --pw (string) Mainframe (z/OSMF) password, which can be the same as your TSO password. Your TSO userid’s password
    • --reject-unauthorized | --ru (boolean) Reject self-signed certificates. Default value: true
    • --base-path | --bp (string) The base path for your API mediation layer instance. Specify this option to prepend the base path to all z/OSMF resources when making REST requests. Do not specify this option if you are not using an API mediation layer.
    • --protocol (string)
    • --cert-file (local file path) The file path to a certificate file to use for authentication
    • --cert-key-file (local file path) The file path to a certificate key file to use for authentication
    • --completion-timeout | --cto (number) The amount in time, in seconds, a REST operation should wait to complete before timing out
    • --establish-connection-timeout | --ecto (number) The amount of time, in seconds, a REST operation should wait while connecting to the server before timing out.
  • PROFILE OPTIONS
    • --zosmf-profile | --zosmf-p (string) The name of a (zosmf) profile to load for this command execution.
    • --base-profile | --base-p (string) The name of a (base) profile to load for this command execution.
  • BASE CONNECTION OPTIONS
    • --token-type | --tt (string) The type of token to get and use for the API. Omit this option to use the default token type, which is provided by ‘zowe auth login’.
    • --token-value | --tv (string) The value of the token to pass to the API.
  • MQ options
    • --mq-profile | --mq-p (string) The name of a (MQ) profile to load for this command execution.

Now you know what the options are you can search for them. This pointed me to the console command page.

Whoops profile options not found

I fell over trying to specify a nested profile.

For example

...
"profiles": {
"qa_lpar": { // Base profile connection properties are used unless overriden
"type": "base",
"properties": {
}
},
"profiles": {
"mq": {...
},

This is referred to as qa_lpar.mq .

What would I have done?

Personally I would have have a help page which listed all of the common options then list commands for example

  • Common options
    • --host etc

Specific commands

Why did that curl request take so long?

I’ve just discovered that you can get curl to report how long each part of a session took.

I set up a bash script

t=(--write-out "\n DNS Lookup:%{time_namelookup}s\n TCP Connect:%{time_connect}s\n TLS Handshake: %{time_appconnect}s\n Total Time: %{time_total}s\n")
...
curl -X PUT --header 'Content-Type: application/json' "${t[@]}" ....

and in the output it gives

 DNS Lookup:0.000033s
TCP Connect:0.003088s
TLS Handshake: 0.322826s
Total Time: 0.981765s

Note the \n in the text to give new lines.

For more details

see curl man page. –write-out.

You can specify an output file, stderr or stdout.

There are many parameters, the performance timing ones are

  • time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed.
  • time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
  • time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.
  • time_posttransfer:The time it took from the start until the last byte is sent by libcurl. In microseconds.
  • time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
  • time_queue: The time, in seconds, the transfer was queued during its run. This adds the queue time for each redirect step that may have happened. Transfers may be queued for significant amounts of time when connection or parallel limits are in place.
  • time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer and transfer before the final transaction was started. “time_redirect” shows the complete execution time for multiple redirections.
  • time_starttransfer: The time, in seconds, it took from the start until the first byte was received. This includes time_pretransfer and also the time the server needed to calculate the result.
  • time_total: The total time, in seconds, that the full operation lasted.
  • tls_earlydata: The amount of bytes that were sent as TLSv1.3 early data. This is 0 if this TLS feature was not used and negative if the data sent had been rejected by the server. The use of early data is enabled via the command line option “–tls-earlydata“.

z/OSMF console times out with return code 2 reason code 21

When trying to use the z/OSMF console interface to issue a console command and get the response I got after 60 seconds.

{“reason”:”Timeout when creating TSO address space for console COLIN99″,”return-code”:2,”reason-code”:21}

This was caused by PARMLIB(IZUPRMCM) not having a HOSTNAME specified, and was picking up a HOSTNAME of S0W1.DAL-EBIS.IHOST.COM. When I specified HOSTNAME(10.1.1.2) and restarted z/OSMF it all worked!

The HOSTNAME can be in various places in TCP/IP and the value from TSO (and jobs) may be different from a Unix thread.

I had to use the address of the TCP/IP stack for example 10.1.1.2; using the address of 127.0.0.1 didn’t work. I could not connect to z/OSMF from my client.

It was strange that with the HOSTNAME missing, I could create TSO address spaces.

How did I diagnose this problem?

In the file /global/zosmf/data/logs/IZUG0.log were entries like

INFO:Prepare to start new TSO/E address space with acct: ACCT#, proc: IZUFPROC, rsize: 50000, apptag: IZUCONAP 
Ýtx000000000000000E:IBMUSER@10.1.0.2 (PUT) /zosmf/restconsoles/consoles/COLIN99?null¨
2025-06-28T16:44:02.114Z|00000092|com.ibm.zoszmf.consoles.tsoconnect.Connection|run
WARNING:exception when run as server:
java.net.SocketTimeoutException: Connect timed out
...
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
...
at com.ibm.zoszmf.consoles.tsoconnect.Connection$3.run(Connection.java:372)

This shows the TSO account number, the TSO procedure, the region size. I checked these were valid, and the userid had access to them.

The http request was a clue that this was using TCP/IP, and not using the TSO services available through CEA interface.

How did I run the command?

I used a script like

name=”colinpaice”
cert=” –cert ./$name.pem:password –key $name.key.pem”

insecure=”–insecure”
curl -X PUT –header ‘Content-Type: application/json’ $cert –header ‘Accept: application/json’
–insecure -d ‘{ “cmd”: “d a,l”, “sol-key”: “JES” }’
https://10.1.1.2:10443/zosmf/restconsoles/consoles/COLIN99&#8217;

What CEA TSO operator commands are there?

Part of the CEA facility service on z/OS, provides the capability for an application to start TSO address spaces, send it TSO commands, and receive the responses. This is used by products lie z/OSMF. You can have a CEA TSO address spaces for a user, as well as a “normal” TSO userid, where you logon and use ISPF.

More information about the commands

Change the CEA parameters F CEA,CEA=(x1,x2,…xN)

Display the CEA configuration parameters F CEA,D,P

STATUS: ACTIVE-FULL      CLIENTS: 0  INTERNAL: 0            
CEA = (00)
SNAPSHOT = N
HLQLONG = CEA HLQ =
BRANCH = COUNTRYCODE =
CAPTURE RANGE FOR SLIP DUMPS:
LOGREC = 01:00:00 LOGRECSUMMARY= 04:00:00
OPERLOG = 00:30:00
CAPTURE RANGE FOR ABEND DUMPS:
...
CAPTURE RANGE FOR CONSOLE DUMPS:
...
TSOASMGR:
RECONSESSIONS = 0 RECONTIME = 00:00:00
MAXSESSIONS = 50 MAXSESSPERUSER= 10

Display a summary of CEA TSO regions F CEA,D,S

STATUS: ACTIVE-FULL      CLIENTS: 0  INTERNAL: 0         
EVENTS BY TYPE: #WTO: 0 #ENF: 0 #PGM: 0
TSOASMGR: ALLOWED: 50 IN USE: 1 HIGHCNT: 0

Display client summary F CEA,D,CLIENTSUMMARY and D CEA,CLIENT=*

STATUS: ACTIVE-FULL      CLIENTS: 0  INTERNAL: 0                   
EVENTS BY TYPE: #WTO: 0 #ENF: 0 #PGM: 0
TSOASMGR: ALLOWED: 50 IN USE: 1 HIGHCNT: 0
NO CLIENTS KNOWN TO CEAS AT THIS TIME
12I CN=L700 DEVNUM=0700 SYS=S0W1

Display the session information F CEA,DIAG,SESSTABLE

INDEX=0001 USERID=COLIN    APPID=IZUCONAP ASID=004E MSGQID=00060018                       
COUNT=0001 ASCBADDR=FC3B80 STOKEN=0000013800000009 STTIME=15:34:43.966
LRTIME=15:34:43.967 LOGONPROC=IZUFPROC GROUP= REGION=50000
CODEPG=1047 CHARSET=697 ROWS=204 COLS=160 RECONN=N RCTIME=00:00:00.000
ACCT=ACCT#
HOST REMOTESYS= REMOTEQID=00000000 CALLERSYS=

This shows information like the TSO LOGON procedure used, the screen size,the region size and the account number.

Mapping a certificate to a userid and so avoid needing a password is good – but…

You can use the RACDCERT MAP command to map a certificate to a userid, and so avoid the need for specifying a password. Under the covers code uses the pthread_security_np and pass a certificate, or a userid and password, and if validated, the thread becomes that userid, just the same as if the userid was logged on.

Is this secure?

If you store a userid and password on your laptop, even though the data may be “protected” someone who has access to your machine may be able to copy the file and so impersonate you.

With a public certificate and private key, if someone can access your machine, they may be able to copy these files and so impersonate you.

You can get dongles which you plug into your laptop on which you can store protected data. In order to use the data, you need the physical device.

You need to protect the RACF command

Because the RACFCERT command has the power to be dangerous, you need to protect it.

You do not want someone to specify their certificate maps to a powerful userid, such as SYS1. The documentation says

To issue the RACDCERT MAP command, you must have the SPECIAL attribute or sufficient authority to the IRR.DIGTCERT.MAP resource in the FACILITY class for your intended purpose.

For a general user to create a mapping associated with their own user ID they need READ access to IRR.DIGTCERT.MAP.

For a general user to create a mapping associated with another user ID or MULTIID, they need need UPDATE access to IRR.DIGTCERT.MAP.

What’s the best way to set this up?

I think that as part of your process for setting up userids, the process should create the mapping for the certificate to a userid. This way you do not have people creating the mapping. If a mapping already exists, you cannot create another mapping.

You may want an automated process which checks the approval, and issues the commands, and so you do not have humans with the authority to issue the commands.

Of course you’ll have a break-glass all powerful userid in case of emergencies.

But….


Even though the password had expired, I could logon using the certificate. If I revoked the userid the logon failed.

I used certificate logon from z/OSMF and issued console commands. The starts a TSO address space, and z/OSMF passes the commands and responses to the tso address space.

Once a TSO address space has been started, there are no more checks to see if the userid is still valid.

If you want to inactivate the userid, you’ll need to revoke it, and then cancel all the TSO address spaces running on behalf of the userid. Walking someone off site is not good enough. There may be scripts which are automated, and will logon with no human intervention.
TSO address spaces may be configured to be cancelled if there is no activity. If the TSO address space is kept busy, (for example by sending it requests) it may never be forced off.

Wow – sdsf can remember thousands of console commands

By default SDSF can remember up to 50 commands.

SDSF can support 1000’s of commands in its history list – and can also group them by user specified name. I have a group called zosmf, and it has all the commands for managing z/OSMF. I have another group for zowe, and this has the commands I use for Zowe.

This is documented here.

Customisation

This is documented here.

I allocated the dataset COLIN.SDSF.ISFTABL as a PDS FB LRECL80, 1 Cyl primary, 1 Cyl secondary.

I blogged on how to create a logon script for allocating your definitions.

The important part is you need to allocate the data set as ISFTABL

SET &SDSFTAB= &STR(&SYSUID..SDSF.ISFTABL)
ALLOC DA('&SDSFTAB') SHR FILE(ISFTABL)

Using it

Use the / command to display the pop up the System Command Extension window.

This has

   Edit  Options  Help                                                
────────────────────────────────────────────────────────────────────
System Command Extension

===> p izusvr1______________________________________________________
===> _______________________________________________________________

Comment

Group ZOSMF Show ZOSMF (F4 for list)

=> p izusvr1
=> s izusvr1
=>
=>

F5=FullScr F6=Details F7=Up F8=Down F10=Save F11=Clear F12=Cancel

Type a command, specify a group, and press PF10 for save.

To display the commands for a group, either

  • tab down to Show and type the group name, or
  • tab down to Show and press PF4, then select the group

You use the Group field when saving commands, and Show when displaying commands.

Full screen display

You can press PF5 to make it full screen (or not full screen).

Zowe: How to save connection parameters: zowe CLI client configuration files

A Zowe client needs information to be able to invoke a service on Zowe on z/OS. Typically these are

  • host address
  • the host port
  • the service
  • the client public key .pem file
  • the client private key .pem file
  • whether to not allow self signed certificates sent from the host
  • the file containing the CA from the host servers.

These options can be given on the zowe cli command, or can be stored in profile files.

You can have multiple profiles, for example, tso, ssh, z/OSMF, production z/OSMF and test z/OSMF.

The above information may be different for different Zowe server instances- such as development, test and production, and for different applications within a Zowe instance.

Some configuration files are stored in the directory pointed to by the environment variable ZOWE_CLI_HOME, on Linux this defaults to ~/.zowe. These files are available where ever your working directory is. You can also have a configuration file in your working directory so is “application specific”.

Zowe supports a hierarchy of configuration files.

  • zowe.config.json within the current directory. If you change directory you will not use it.
  • zowe.config.user.json , the global user file in the ZOWE_CLI_HOME environment variable ( or it might default to ~/.zowe ). This is used for any user specific options (if any)
  • zowe.config.json the global file in the ZOWE_CLI_HOME environment variable ( or it might default to ~/.zowe ). This is used for project wide options.

The Zowe CLI command will start at the top of the hierarchy and go down until it finds the attributes. If it cannot find an attribute it may have a default.

Within the configuration file are different sections, for example

  • a base section with common parameters,
  • ftp, with specific parameters such as the FTP port number,
  • zOSMF, with with specific parameters such as the z/OSMF port number

You can edit the files manually, or use the Zowe config command to make changes.

Sigh

To specify a Certificate Authority(CA) certificate to a zowe cli command, you have to use an environment variable, for example

export NODE_EXTRA_CA_CERTS=/home/colinpaice/ssl/ssl2/doczosca.pem

You need to allow for this when writing your scripts, because different backend servers may have different CA certificates.

See here.

Getting started

Once you are familiar with Zowe and have a better idea of your Zowe configuration, you can develop the various configuration files; what information is user specific, what information is project specific, and what information is Zowe instance specific. Some files are stored in the direction in the ZOWE_CLI_HOME environment variable which defaults to ~/.zowe (on Linux).

If you may have more than one Zowe instance, you may decide NOT to use ~/.zowe, but have directories like ~/.zoweproduction and ~/.zowetest. This means you have to explicitly specify the location.

Get started by creating zowe.config.json in your working directory. See below, or use cut and paste from here, step 3. If you use an IDE such as slickedit or vscode, these have syntax assist, and can highlight errors as you type.

This is a good blog post on initial configuration. But does not cover use of certificates.

The Zowe config command

You can use the zowe config command to create and update the configuration files. The command has options including

This page has a good description of the options.

  • –dry-run : display the changes, but do not apply the changes
  • –edit and –editor name : this opens the file in the specified editor.

Create a zowe.config.json in the current directory

The command

zowe config init

will create a file zowe.config.json in the current directory.

I created a minimum (hand crafted) file to illustrate how the zowe config commands work.

{
"$schema": "./zowe.schema.json",
"profiles": {
"zosmfcp": {
"type": "zosmf",
"properties": {
"port": 443
},
"secure": []
},

"project_base": {
"type": "base",
"properties": {
"host": "10.1.1.2",
"rejectUnauthorized": true
},
"secure": [
"user",
"password"
]
}
},
"defaults": {
"zosmf": "zosmfcp",
"base": "project_base"
},
"autoStore": true
}

Key elements of this file

  • “defaults”:
    • zosmf : For a z/OSMF request,if --base-profile is not specified, use the referenced profile – zosmfcp.
    • common parameters are defined in the base: profile -> project_base
  • “profiles”: There are two profiles defined
    • zosmfcp with the zosmf specific parameters
    • project_base with the common parameters.

When a zowe command is used, the parameter –base-profile can specify which profile to use.

Commands to display and set profile information

My working directory is /home/colinpaice/ssl/ssl2/ .

zowe config list –locations –root

This lists the only the file names used by Zowe CLI.

/home/colinpaice/ssl/ssl2/zowe.config.json
/home/colinpaice/.zowe/zowe.config.json

zowe config list –locations

This shows the file names and their contents

/home/colinpaice/ssl/ssl2/zowe.config.json: 
$schema: ./zowe.schema.json
profiles:
zosmf:
type: zosmf
properties:
port: 10443
secure:
(empty array)
project_base:
...
/home/colinpaice/.zowe/zowe.config.json:
$schema: ./zowe.schema.json
profiles:
zosmf:
type: zosmf
properties:
port: 10443
secure:
(empty array)
tso:
type: tso
....
global_base:
type: base
properties:
host: 10.1.1.2
rejectUnauthorized: true
user: (secure value)
password: (secure value)
secure:
- user
- password
defaults:
zosmf: zosmf
tso: tso
ssh: ssh
base: global_base
autoStore: true

Change a parameter in the file in the current directory

zowe config set “profiles.zosmf.properties.port “30443”

Change/Add a profile (and content) to the the file in the current directory

zowe config set “profiles.zosmf2.properties.port” “40443”

If the profile zosmf2 did not exist before, it will create it, and add the property port

{
"$schema": "./zowe.schema.json",
"profiles": {
...
"zosmf2": {
"properties": {
"port": 40443
}
}
},
...
}

Set a parameter in the file in the user configuration file

zowe config set “profiles.zosmf2.properties.port” “40443” –user-config

set a parameter in the file in the global configuration file

zowe config set “profiles.zosmf2.properties.port” “50443” –global-config

Example of using the set commands

zowe config set “profiles.zosmf2.properties.port” “50443” –global-config
zowe config set “profiles.zosmf2.properties.port” “4444” –user-config
zowe config set “profiles.zosmf2.properties.port” “1111”

  • file zowe.config.user.json has zosmf2.properties.port:4444
  • file zowe.config.json: has zosmf2.properties.port: 1111
  • file ~/.zowe/zowe.config.json: has zosmf2.properties.port: 50443

Using the list commands, and displaying only the above fields I have

zowe config list gives

profiles: 
...
zosmf2:
properties:
port: 4444

zowe config list –locations gives

/home/colinpaice/ssl/ssl2/zowe.config.user.json: 
profiles:
zosmf2:
properties:
port: 4444
defaults:
/home/colinpaice/ssl/ssl2/zowe.config.json:
profiles:
...
zosmf2:
properties:
port: 1111
...
/home/colinpaice/.zowe/zowe.config.json:
$schema: ./zowe.schema.json
profiles:
...
zosmf2:
properties:
port: 50443

So we can see the profile /home/colinpaice/ssl/ssl2/zowe.config.user.json: provided the port value 444.

If I use –base-profile zosmf2, the port used will be 4444.

If I add

zowe config set “profiles.zosmf2.properties.ca” “MYCA”

This create property zosmf2:properties:ca:”MYCA” in file zowe.config.json.

zowe config list gives

profiles:
zosmf2:
properties:
port: 4444
ca: MYCA

so we can see what is being used.

If I use zowe config list –locations I can see the definition is in /home/colinpaice/ssl/ssl2/zowe.config.json

display profiles

zowe config profiles gave

zosmf
tso
ssh
global_base
zosmf2

What are the valid options?

If you used the zowe config init command, then this generated a schema file.

For zosmf this schema has

  • “host”: “The z/OSMF server host name.”
  • “port”: “default”: 443
  • “user”: “Mainframe (z/OSMF) user name, which can be the same as your TSO login.”
  • “password”: “Mainframe (z/OSMF) password, which can be the same as your TSO password.”
  • “rejectUnauthorized”:”Reject self-signed certificates.”, “default”: true
  • “certFile”: “The file path to a certificate file to use for authentication” “certKeyFile”: “The file path to a certificate key file to use for authentication”
  • “basePath”: “The base path for your API mediation layer instance. Specify this option to prepend the base path to all z/OSMF resources when making REST requests. Do not specify this option if you are not using an API mediation layer.”
  • “protocol”: “The protocol used (HTTP or HTTPS)” “default”: “https”, “http”, “https”
  • “encoding”: “The encoding for download and upload of z/OS data set and USS files. The default encoding if not specified is IBM-1047.”
  • “responseTimeout”: “The maximum amount of time in seconds the z/OSMF Files TSO servlet should run before returning a response. Any request exceeding this amount of time will be terminated and return an error. Allowed values: 5 – 600”

The CA certificate to use is specified by an environment variable, and is not a parameter in the profiles.


Zowe: Changing trace on z/OS

Trace within Zowe is a bit confusing. Different components, such as Gateway and Application-Server seems to have different ways of tracing.

As I understand it, you set a global switch for a components trace, restart Zowe, then use a REST API to set or reset subcomponents.

It took me a day or so to be able to change the trace of Zowe components!

There is Zowe documentation on Troubleshooting Zowe API Mediation Layer. Which is incomplete, and missing useful documentation.

The syntax of the logging commands is here

List the existing traces

The documentation says

GET scheme://hostname:port/application/loggers

This didn’t work for me because I was using port 7554. After I tried all of the Zowe ports, the following worked

"https://10.1.1.2:7558/application/loggers"

Where 7558 is the port for zaas.

This returned one long string, with many logical lines of output. For example:

{
"levels": [
"OFF",
"ERROR",
"WARN",
"INFO",
"DEBUG",
"TRACE"
],
"loggers": {
"ROOT": {
"configuredLevel": "INFO",
"effectiveLevel": "INFO"
},
"_org": {
"configuredLevel": null,
"effectiveLevel": "INFO"
},
"_org.springframework": {
"configuredLevel": null,
"effectiveLevel": "INFO"
},
...

},
"groups": {
"web": {
"configuredLevel": null,
"members": [
"org.springframework.core.codec",
"org.springframework.http",
"org.springframework.web",
"org.springframework.boot.actuate.endpoint.web",
"org.springframework.boot.web.servlet.ServletContextInitializerBeans"
]
},
"sql": {
"configuredLevel": null,
"members": [
"org.springframework.jdbc.core",
"org.hibernate.SQL",
"org.jooq.tools.LoggerListener"
]
}
}
}
  • 434 lines for org.springframework….
  • 36 lines for com.netflix….
  • 286 line for io….
  • 63 lines for org.apache…
  • 57 lines for org.hibernate
  • 68 lines for org.ehcache
  • 109 lines for org.zowe.apiml.

Selecting one record type

I used a Bash script

#!/bin/bash 
url="https://10.1.1.2:7558/application/loggers/org.zowe.apiml.zaas.security"
certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
https GET ${url} $certs $verify

gave

{
"configuredLevel": "null",
"effectiveLevel": "DEBUG"
}

This means org.zowe.apiml.zaas.security was configured as null, but one of org.zowe, org.zowe.apiml, or org.zowe.apiml.zaas was set to DEBUG, and all of the definitions below it were set to DEBUG.

If I set the value of org.zowe.apiml.zaas.security to WARN, then set the value of org.zowe.apiml.zaas to DEBUG, the value of org.zowe.apiml.zaas.security stayed at WARN.

{
"configuredLevel": "WARN",
"effectiveLevel": "WARN"
}

Updating a value

It took me a while to get a script to update the value.

url="https://10.1.1.2:7558/application/loggers/org.zowe.apiml.zaas.security"
certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
echo -n '{"configuredLevel": "WARN"}' | https POST ${url} $certs $verify
echo "================="
https GET ${url} $certs $verify

This pipes the update {“configuredLevel”: “WARN”} into the https command. This returned no data. If it fails it may return a message.

Changing the hierarchy

Logically changing a value for a.b.c to ZZZZ works as follows

for every element under a.b.c
   if the element's value is null 
   then set element's value = ZZZZ
   else do nothing 

z/OSMF: what traces are available

I could find no z/OSMF documentation on how to turn on traces. The documentation says “Contact IBM”. Someone pointed out the z/OSMF Diagnostic Assistant. This is an icon on the z/OSMF main screen (https://10.1.1.2:10443/zosmf/ for me).

You need to click on “Add Service” to list the components, before you can change the log level.

Fill in the details, use the Tick box at the start of the line, and the Log Level pull down. Once you have chose them, click on the “Set” button.

You can issue an operator command.

f server-name,logging='trace_specification'

Below is a list of the trace commands from the diagnostic Assistant . The granularity of the trace is warning, info, fine, finer, finest

For SAF security calls use zos.native=finest.

Ive seen this as zos.native.03=…

I also found these – but I dont know what they mean!

  • com.ibm.ccc.=ALL:
  • com.ibm.crypto.=all:
  • com.ibm.websphere.security.*=all:
  • com.ibm.ws.security.=ALL:
  • com.ibm.ws.webcontainer.=all:
  • com.ibm.wsspi.webcontainer.*=all:
  • HTTPChannel=all:
  • GenericBNF=all:
  • zos.native.03=all – this give information about a subset of z/OS calls
  • com.ibm.websphere.security.jwt=all

SAF security calls: zos.native=finest

Produces output like (formatted for display)

6/23/25, 16:39:20:192 GMT?] 00000028 id=00000000 zos.native.03.001 
Trace: 6/23/25, 16:39:20:192 GMT? t=8c8140 key=S2 (300100f)
Description: RACROUTE REQUEST=FASTAUTH call
racrouteArea_p: 000000007e4b0c10
6/23/25, 16:39:20:193 GMT?] 00000028 id=00000000 zos.native.03.001
Trace: 6/23/25, 16:39:20:193 GMT? t=8c8140 key=S2 (3001010)
Description: RACROUTE REQUEST=FASTAUTH return
returnCode: 0
safReturnCode: 0
racfReturnCode: 0

racfReasonCode: 4
6/23/25, 16:39:20:194 GMT?] 00000028 id=00000000 zos.native.03.001
Trace: 6/23/25, 16:39:20:194 GMT? t=8c8140 key=S2 (3001012)
Description: Exit: checkAuthorizationFast
returnCode: 0
6/23/25, 16:39:20:195 GMT?] 00000028 id=00000000 zos.native.02.008
Trace: 6/23/25, 16:39:20:195 GMT? t=8c8140 key=S2 (2008005)
Description: Entry: registrySetUnused
alreadyVerified: true
token: data_address=00000051_2a6103b8, data_length=64
+--------------------------------------------------------------------------+
|OSet| A=000000512a6103b8 Length=0000040 | EBCDIC | ASCII |
+----+-----------------------------------+----------------+----------------+
|0000|C2C2C7E9 D9C5C7E3 00000001 00000040|BBGZREGT....... |...............@|
|0010|00000000 7DC72100 00000007 00000000|....'G..........|....}.!.........|
|0020|01000000 00000000 00000000 00000000|................|................|
|0030|00000000 00000000 00000000 00000000|................|................|
+--------------------------------------------------------------------------+

It is not very well written, because there are “safReturnCode:”, “SAF return code:”,”return code:” and “returnCode:”, and “rc:”. I found it easiest to use the following in an edit session to find non zero return codes.

  • x all
  • f ‘code: 0’ all
  • del all x
  • f ‘code:’ all

The trace entry ” token: data_address=00000051_2a6103b8, data_length=64 ” shows the value with name “token” at the specified address, and length. It is displayed in hex, EBCDIC and ASCII.

Trace points.

Taken from a trace file

  • zos.native.02.002 getConsoleCommand
  • zos.native.02.006 CommandProcessor.ntv_issueCommandResponse
  • zos.native.02.00d DeleteWorkUnit*
  • zos.native.02.00e wlm_enclave_*
  • zos.native.02.00e wlm_enclave_create leave
  • zos.native.03.001 RACROUTE REQUEST=FASTAUTH
  • zos.native.03.003 CertificateCredential…
  • zos.native.03.004 ntv_createCertificateCredential
  • zos.native.03.005 checkAccess
  • zos.native.03.006 ntv_checkAccess
  • zos.native.03.007 invokeIRRSIA00 createACEE
  • zos.native.03.008 PenaltyBox
  • zos.native.03.00b getGroupsForUser
  • zos.native.04.002 write_to_operator_response
  • zos.native.04.004 Latch
  • zos.native.04.007 getStck