Oh p*x, it didn’t copy across some files.

I had managed to mess up the files for a product, so I wanted to copy them across from an older system.

This worked for some of the files – but when I came to start the subsystem – it was missing some files! For example /u/my/zosmf/liberty/lib/native/zos/s390x/bbgzsrv

I copied the files across again – and they were still not there!

Once you know the answer it is obvious…

There is a directory /usr/lpp/zosmf/liberty – and it was this directory that was missing.

Once I looked into it more carefully – this was not a directory, but a symbolic link to another directory liberty -> ../liberty_zos/current

To fix this I used

# go to my version of zosmf
cd /u/my/zosmf
# remove the symbolic link
rm liberty
#make the new link
ln -s /usr/lpp/liberty_zos/current liberty

and now I could use ls /u/tmp/zosmfp/liberty/lib/native/zos/s390x/bbgzsrv and it found the file.

If I had checked this before I started, I would have save myself a half day of IPLing older systems!

Zowe: What should I used on my client to talk to z/OS?

There are different ways of accessing the z/OS server.

  • You can use a web server. You will need to configure the certificates.
  • You can write your own program, for example in C or Java.
  • cUrl is a very common tool. Handy for small tasks.
  • HTTPie is an easy tool to generate one of commands. The output comes back nicely formatted, with colour. HTTPie is written in Python.
  • Python has good support for TLS and working with back send-servers. You can capture the data in a dictionary to make it easy to process.
  • You can use zowe Command Level Interface (CLI). This uses the best bits of all of the above.

Which is best?

The answer is the old phrase “horses for courses”.

If you want to do a one-of then the above tools are all very similar.

If you want to use multiple back ends, for example production and test, then you need to have code which selects the host, port, certificates, and applications within a z/OSMF or Zowe instance, then you will need to have code which handles this. Zowe CLI has this code built in.

I could write some basic code in about 100 lines of Python to do this – but why bother?

A Zowe developer’s view

I asked a developer what is good about zowe CLI.

The most common use case I see with Zowe CLI is for use in automation/Jenkins. It saves you the trouble of having to write code to handle the REST requests. It also supports the use of plug-ins for other products or services running on z/OS, like Endeavor, ChangeMan, Db2, CICS, etc. to do more than just download/upload files or submit jobs (e.g., you could also interact with a version control system, issue Db2 queries, deploy changes to a CICS region, and so on). 

Regarding output/data types, Zowe CLI commands can also provide json output (by specifying the `–response-format-json` or `–rfj` flag on commands), which is pretty easy to work with in bash scripts using something jq on Linux to parse the output and extract fields that you need.

For example

With –response-format-json

zowe zos-files list data-set ‘COLIN.Z*’ … –response-format-json

Generated

{
"success": true,
"exitCode": 0,
"message": "",
"stdout": "COLIN.ZCONNECT.BETA.ZFS\nCOLIN.ZCONNECT.BETA.ZFS.DATA\nCOLIN.ZFS\n...
"stderr": "",
"data": {
"success": true,
"commandResponse": null,
"apiResponse": {
"items": [
{
"dsname": "COLIN.ZCONNECT.BETA.ZFS"
},
{
"dsname": "COLIN.ZCONNECT.BETA.ZFS.DATA"
},
...
"returnedRows": 44,
"JSONversion": 1
}
}

Without –response-format-json

COLIN.ZCONNECT.BETA.ZFS
COLIN.ZCONNECT.BETA.ZFS.DATA
COLIN.ZFS
COLIN.ZFS.DATA
COLIN.ZFS.DSFS
COLIN.ZFS.DSFS.DATA
...

The first output is easier to post-process.

Some articles on Zowe CLI

Zowe: What does this message mean?

You can use your favourite search engine to look for the message. If it is not found, you can search within the open source.

Thanks to Martin Zeitham for the following.

Each message has prefix, which indicates the server or component. For example ZWEL* is Launcher etc.
There is a command ./zwe diagnose -e messageID to get more details:

./zwe diagnose -e ZWED0020I

gave

This code corresponds to the errors related to the Zowe Desktop and the App Server.

To find the description of this error code, refer to the:

Zowe documentation for Application framework
https://docs.zowe.org/stable/troubleshoot/app-framework/appserver-error-codes

You may also explore reports from other users experiencing the same error by searching
https://github.com/search?q=org%3Azowe+ZWED0020I&type=discussions

I found it quicker to look in the Zowe bin/commands/diagnose/index.js file, which has

And note the ability to see any discussion on the message ZWED0020I in the git repository using

https://github.com/search?q=org%3Azowe+ZWED0020I&type=discussions

Giving a started task userid a password should be a sackable offence.

Last week I was going through some product documentation, and I got to the part where it said “Now change the started task userid, and give it a password”. I made a note to raise a defect on this, because this is a no-no.
This week someone asked on IBM-MAIN, the impact of giving a started task userid a password. There were many well informed comments covering things I didn’t know about, so I thought I’d make a blog post of of the comments.

Best practices dictate that passwords only be provided when a userid will be used by a specific person and that person needs a password for that userid. All other userids should never have a password.

A userid can be revoked because of too many invalid password attempts, or revoked because of inactivity. If a userid is revoked it cannot logon. If the userid of a started task is revoked, the started task may start, but it may be restricted as to what it can do – because it cannot logon.

You can alter a userid to have NOPASSWORD (and have no PHRASE). This means started tasks can start, but the userid cannot be used to logon to the system. This is known as making the userid PROTECTED.

Often started task userids have special capabilities, such as running as different userids, being able to set security options, or modify system storage. This means you do not want your Help Desk staff from adding or resetting passwords for protected userid. Changes to these protected userids should be done from a secure, limited access userid.

If you think through the impact of using started tasks. It may be better for all routine production jobs to be run as started tasks. This has the advantage that there are no passwords involved, and you can use automation to issue the start command based on a timer.

You might have a CICS started task userid for all CICS regions or just for a subset of regions. You might have one started task userid for all started tasks, or a started task userid for each logical instance, eg CICS, MQ, DB2, Zowe, TCP/IP etc.

System jobs

System jobs should be run as started tasks, and the started tasks should be protected

Personal jobs

You can submit jobs from your userid, (and not specify a password) and the job will run under your userid.

You can put USER=name,PASSWORD=… on a job card, and if these validate the job will run with the specified userid. This is not a good idea, as the password may be visible in the dataset.

Departmental userids

You can put USER=name and omit the password, and use surrogate checking. The documentation says

You can allow the use of surrogate users. A surrogate user is a RACF-defined user who has been authorized to submit jobs on behalf of another user (the execution user) without specifying the execution user’s password. Jobs submitted by a surrogate user run with the identity of the execution user. For example, if user JOE submits a job with the following JOB statement, JOE is the surrogate user and TOM is the execution user:

JOE can submit userid containing

//jobname JOB 'accounting-information',USER=TOM

You set up a security profile (see the documentation) to control which userid can specify a userid on the JOB USER= statement.

All access checks are done with TOM’s user ID.

The TOM userid can be a protected userid – without a password, if surrogates are used.

To set up surrogates, defined the profile, and give a group access to the profile, rather than give userids access. You are likely to have a group already defined. Administration, such as when someone leaves the department, is much easier, as you just need to remove the persons userid from the group.

Thanks to

Robert S. Hansel, Seymour J Metz, Steve Beaver, Jack Zukt, Mike Schwab, Jon Perryman for their comments.

Is making a backup easy ? Perhaps. Is it easy to restore? It depends

I was asked about backing up a database before doing a migration to the next version of the software. The answer was yes – take a backup. But you need to think it through.

I have a security database (RACF). If I backup it up today, ready for a migration next week, the backup may be missing some entries. For example tomorrow, someone changes their password, or defines a new resource. If I need to restore from the backup, the restored database will not have these changes.

It gets worse. If I backup the database while is is in use, I may get inconsistent data. I start my backup, and define two resources. One resource is in the part of the database already backed up, the other resource is the part of the database which is just about to be backed up.

I restore the database from the backup. It will have the second change but not the first change.

How to backup

You need to ensure that the database is read only for users, during the work. People cannot defined resources, change passwords etc., but the migration progress can make its changes.

An easy way of doing this is to start the system in single user mode.

You backup the database just before you do the migration. You might do a backup the week before to ensure the process works, for example you have enough space, and permissions, for a backup.

Restore

Restoring is a harder problem.

If you do a migration and the migration fails, you should be able to restore the database as it was before the migration started.

If the migration seemed to work, but a week later you discover there are problems. If you restore from the backup – any changes made by end users after the migration will be missing from the backup.

You need a process to handle this, for example compare the active database with the backed up copy of the data base, and note any difference, then make the same changes.

Making changes

Someone said that planning for changes when it all works is easy – just push the button.

The hard, and often forgotten, part is planning for when things go wrong, for example you find your backup was not successful, or the migration takes longer than planned, and you have an availability deadline to meet.

Zowe: Problem determination

This is a work in progress as I learn more about debugging Zowe

Startup and processing the zowe.yaml file.

Troubleshooting Zowe Launcher. Creates a trace of the parsing of the Zowe.yaml file.

It is verbose;only use when necessary.

For example

It displays the contents of the zowe.yaml file

MapStart                                                 
zowe:
MapStart
runtimeDirectory:
Scalar: (len=13)/u/tmp/zowep/
logDirectory:
Scalar: (len=17)/u/tmp/zowec/logs
workspaceDirectory:
Scalar: (len=22)/u/tmp/zowec/workspace
extensionDirectory:
Scalar: (len=23)/u/tmp/zowec/extensions

and

path:                                                             
validate JSON value->type=4 specTypeMask=0x4
typeCode=2 shifted=0x4 mask=0x4
validateJSONObject required=[]
accessPath (top is blank):
validate object pname=zowe

Component loggers

I found this doc, I haven’t used it

Customizing Zowe API Mediation Layer logging for Specifies one of the following services: zaas, gateway, discovery, api-catalog, caching-service.

ZSS https trace

In the zowe.yaml

zss: 
enabled: true
port: 7557
crossMemoryServerName: ZWESIS_STD
agent:
jwt:
fallback: true
64bit: true
https:
trace: true

produces a trace in the logs directory

zssServer-2025-05-29-08-39.log.tlstrace

use gsktrace zssServer-2025-05-29-08-39.log.tlstrace >gsktrace.txt

What are my Java options?

See Specifying Java overrides

Zowe tracing.

See Enabling tracing for app-server and zss.

Zowe: Specifying Java overrides

As part of my configuring Zowe, and using Java shared classes, I’ve needed to change the Java parameters in Zowe. I know there is a plan to provide this support – but this is what I’ve done.

Find the start up script.

If you use the Unix command

find zowe/components -name startup.sh

This will list the scripts for the components.

I edited zowe/components/gateway/bin/start.sh and inserted at the top

 COL="/u/tmp/zowec/" 

TAG="GW"
fn="${COL}/java.options.${TAG}"
if [ -f "${fn}" ]; then
set -x
export
exec 1>>${COL}start.log1${TAG} 2>>${COL}start.log2${TAG}
COLIN="-Xoptionsfile=${fn} -XshowSettings "
else
COLIN=""
fi

and at the bottom of the file

    -Dloader.path=${GATEWAY_LOADER_PATH} \ 
${COLIN} \
-jar ${JAR_FILE} &

This code

  • checks for a file java.options.GW in the Zowe instance directory.
  • If this file exists
    • Create log files in the directory
    • Create an override variable (COLIN), specifying the file name, and telling Java to print out the settings.
  • If it doesn’t exist, set the variable (COLIN) to blank
  • When the Java command is issued, substitute the variable COLIN – which is either blank, or the options file and display Java options command.

Create the java options file.

You can use

touch java.options.GW

or edit the the and add content.

The output

It produced output like

VM settings: 
Min. Heap Size: 32.00M
Max. Heap Size: 512.00M
Using VM: IBM J9 VM

Property settings:
XcompilationThreads3 =
Xhealthcenter =
Xjit:count = 0
...
JRE 21 z/OS s390x-64-Bit Compressed References 20241024_16 (JIT enabled, AOT enabled)
OpenJ9 - f45de8e9eb0
OMR - 55ddfd47ab0
IBM - 3c87141
JCL - df334d2be4a based on jdk-21.0.4+7
java.home = /Z31B/usr/lpp/java/J21.0_64
java.io.tmpdir = /tmp
....
xcom.ibm.java.diagnostics.healthcenter.headless.output.directory = /u/tmp/zowec

Locale settings:
default locale = English (United States)
default display locale = English (United States)
...
Security settings summary:
See "java -X" for verbose security settings options
Security provider static configuration: (in order of preference)
Provider name: OpenJCEPlus
Provider name: IBMZSecurity
Provider name: SUN
Provider name: SunRsaSign
...
May 29, 2025 9:30:25 AM sun.security.ssl.SSLLogger log
WARNING: No AlgorithmParameters for sect163k1
May 29, 2025 9:30:27 AM sun.security.ssl.SSLLogger log
WARNING: No AlgorithmParameters for sect163r1
...
Security TLS configuration (SunJSSE provider):
Enabled Protocols:
TLSv1.3
TLSv1.2

Something for the weekend. Which is heavier and ounce of gold or an ounce of paper?

I remember being asked this question at my grandfather’s knee. Of course the popular answer, which is wrong, is they both weight the same. Yes, they have different weights, an ounce of gold is heavier than an ounce of paper.

An ounce of gold weights 31 grams, and an ounce of paper weight 28 grams. Gold is measured in ounces troy, and most other things, including paper, are measured in ounces avoirdupois.

A similar question is why do cars in Britain have a better miles per gallon than in the US – it is not because Americans have bigger cars. A US gallon is 3.7 litres and a UK gallon is 4.5 litres, Americans have smaller litres.

Jump up and down: Do not give userids access to resources!

I am doing some work connecting subsystems together. The documentation for all the products involved, describe giving a userid permission to access to a resource.

This is not best practice. Like many things it will be obvious once you understand it. You need to look at the bigger pictures.
The documentation says for userids who want to use “this”, give them access to the profile “…”. What’s wrong with that?

  • If you have a department of 1000 people, giving them all access to the resource will be tedious
  • There are likely to be several resources people need access to, so connecting these 1000 userids to multiple resources will be even more tedious.
  • Someone joins your department – so you need to connect their userid to the long list of groups.
  • Someone leaves your department – you cannot trivially ask what resources can this userid access – you have to look at the access list for each resource and remove the userid from the group.

You most probably have groups set up already. Rather than give the userid access, give the group access to the resource.

If someone joins your group – you connect their userid to the groups – and they have access. If someone leaves your department, remove them from the groups – and they no longer have access to the resources.

You my want to support a new product… That’s easy – give the group(s) access to the resources – and the people will auto-magically get access.

As I said it is obvious once you see it.

Do not give userids access to resources – give groups access, and connect the userid to the group. I’ll go and raise documentation comments on the products’ documentation.