Is making a backup easy ? Perhaps. Is it easy to restore? It depends

I was asked about backing up a database before doing a migration to the next version of the software. The answer was yes – take a backup. But you need to think it through.

I have a security database (RACF). If I backup it up today, ready for a migration next week, the backup may be missing some entries. For example tomorrow, someone changes their password, or defines a new resource. If I need to restore from the backup, the restored database will not have these changes.

It gets worse. If I backup the database while is is in use, I may get inconsistent data. I start my backup, and define two resources. One resource is in the part of the database already backed up, the other resource is the part of the database which is just about to be backed up.

I restore the database from the backup. It will have the second change but not the first change.

How to backup

You need to ensure that the database is read only for users, during the work. People cannot defined resources, change passwords etc., but the migration progress can make its changes.

An easy way of doing this is to start the system in single user mode.

You backup the database just before you do the migration. You might do a backup the week before to ensure the process works, for example you have enough space, and permissions, for a backup.

Restore

Restoring is a harder problem.

If you do a migration and the migration fails, you should be able to restore the database as it was before the migration started.

If the migration seemed to work, but a week later you discover there are problems. If you restore from the backup – any changes made by end users after the migration will be missing from the backup.

You need a process to handle this, for example compare the active database with the backed up copy of the data base, and note any difference, then make the same changes.

Making changes

Someone said that planning for changes when it all works is easy – just push the button.

The hard, and often forgotten, part is planning for when things go wrong, for example you find your backup was not successful, or the migration takes longer than planned, and you have an availability deadline to meet.

Zowe: Problem determination

This is a work in progress as I learn more about debugging Zowe

Startup and processing the zowe.yaml file.

Troubleshooting Zowe Launcher. Creates a trace of the parsing of the Zowe.yaml file.

It is verbose;only use when necessary.

For example

It displays the contents of the zowe.yaml file

MapStart                                                 
  zowe:                                                  
    MapStart                                             
      runtimeDirectory:                                  
        Scalar: (len=13)/u/tmp/zowep/                    
      logDirectory:                                      
        Scalar: (len=17)/u/tmp/zowec/logs                
      workspaceDirectory:                                
        Scalar: (len=22)/u/tmp/zowec/workspace           
      extensionDirectory:                                
        Scalar: (len=23)/u/tmp/zowec/extensions

and

path:                                                             
validate JSON value->type=4 specTypeMask=0x4                      
    typeCode=2 shifted=0x4 mask=0x4                               
    validateJSONObject required=[]                                
    accessPath (top is blank):                                    
    validate object pname=zowe

Component loggers

I found this doc, I haven’t used it

Customizing Zowe API Mediation Layer logging for Specifies one of the following services: zaas, gateway, discovery, api-catalog, caching-service.

ZSS https trace

In the zowe.yaml

zss: 
  enabled: true 
  port: 7557 
  crossMemoryServerName: ZWESIS_STD 
  agent: 
    jwt: 
      fallback: true 
    64bit: true 
    https: 
      trace: true

produces a trace in the logs directory

zssServer-2025-05-29-08-39.log.tlstrace

use gsktrace zssServer-2025-05-29-08-39.log.tlstrace >gsktrace.txt

What are my Java options?

See Specifying Java overrides

Zowe tracing.

See Enabling tracing for app-server and zss.

Zowe: Specifying Java overrides

As part of my configuring Zowe, and using Java shared classes, I’ve needed to change the Java parameters in Zowe. I know there is a plan to provide this support – but this is what I’ve done.

Find the start up script.

If you use the Unix command

find zowe/components -name startup.sh

This will list the scripts for the components.

I edited zowe/components/gateway/bin/start.sh and inserted at the top

 COL="/u/tmp/zowec/" 
 
 TAG="GW" 
 fn="${COL}/java.options.${TAG}" 
 if [ -f "${fn}" ]; then 
  set -x 
   export 
   exec 1>>${COL}start.log1${TAG} 2>>${COL}start.log2${TAG} 
   COLIN="-Xoptionsfile=${fn} -XshowSettings " 
 else 
  COLIN="" 
 fi

and at the bottom of the file

    -Dloader.path=${GATEWAY_LOADER_PATH} \ 
    ${COLIN} \ 
    -jar ${JAR_FILE} &

This code

  • checks for a file java.options.GW in the Zowe instance directory.
  • If this file exists
    • Create log files in the directory
    • Create an override variable (COLIN), specifying the file name, and telling Java to print out the settings.
  • If it doesn’t exist, set the variable (COLIN) to blank
  • When the Java command is issued, substitute the variable COLIN – which is either blank, or the options file and display Java options command.

Create the java options file.

You can use

touch java.options.GW

or edit the the and add content.

The output

It produced output like

VM settings: 
    Min. Heap Size: 32.00M 
    Max. Heap Size: 512.00M 
    Using VM: IBM J9 VM 
                                                          
Property settings: 
    XcompilationThreads3 = 
    Xhealthcenter = 
    Xjit:count = 0 
...
JRE 21 z/OS s390x-64-Bit Compressed References 20241024_16 (JIT enabled, AOT enabled) 
OpenJ9   - f45de8e9eb0 
OMR      - 55ddfd47ab0 
IBM      - 3c87141 
JCL      - df334d2be4a based on jdk-21.0.4+7 
    java.home = /Z31B/usr/lpp/java/J21.0_64 
    java.io.tmpdir = /tmp 
....
    xcom.ibm.java.diagnostics.healthcenter.headless.output.directory = /u/tmp/zowec 
                                                                                             
Locale settings: 
    default locale = English (United States) 
    default display locale = English (United States) 
...
Security settings summary: 
    See "java -X" for verbose security settings options 
    Security provider static configuration: (in order of preference) 
        Provider name: OpenJCEPlus 
        Provider name: IBMZSecurity 
        Provider name: SUN 
        Provider name: SunRsaSign 
...
May 29, 2025 9:30:25 AM sun.security.ssl.SSLLogger log 
WARNING: No AlgorithmParameters for sect163k1 
May 29, 2025 9:30:27 AM sun.security.ssl.SSLLogger log 
WARNING: No AlgorithmParameters for sect163r1 
...
    Security TLS configuration (SunJSSE provider): 
        Enabled Protocols: 
            TLSv1.3 
            TLSv1.2

Something for the weekend. Which is heavier and ounce of gold or an ounce of paper?

I remember being asked this question at my grandfather’s knee. Of course the popular answer, which is wrong, is they both weight the same. Yes, they have different weights, an ounce of gold is heavier than an ounce of paper.

An ounce of gold weights 31 grams, and an ounce of paper weight 28 grams. Gold is measured in ounces troy, and most other things, including paper, are measured in ounces avoirdupois.

A similar question is why do cars in Britain have a better miles per gallon than in the US – it is not because Americans have bigger cars. A US gallon is 3.7 litres and a UK gallon is 4.5 litres, Americans have smaller litres.

Jump up and down: Do not give userids access to resources!

I am doing some work connecting subsystems together. The documentation for all the products involved, describe giving a userid permission to access to a resource.

This is not best practice. Like many things it will be obvious once you understand it. You need to look at the bigger pictures.
The documentation says for userids who want to use “this”, give them access to the profile “…”. What’s wrong with that?

  • If you have a department of 1000 people, giving them all access to the resource will be tedious
  • There are likely to be several resources people need access to, so connecting these 1000 userids to multiple resources will be even more tedious.
  • Someone joins your department – so you need to connect their userid to the long list of groups.
  • Someone leaves your department – you cannot trivially ask what resources can this userid access – you have to look at the access list for each resource and remove the userid from the group.

You most probably have groups set up already. Rather than give the userid access, give the group access to the resource.

If someone joins your group – you connect their userid to the groups – and they have access. If someone leaves your department, remove them from the groups – and they no longer have access to the resources.

You my want to support a new product… That’s easy – give the group(s) access to the resources – and the people will auto-magically get access.

As I said it is obvious once you see it.

Do not give userids access to resources – give groups access, and connect the userid to the group. I’ll go and raise documentation comments on the products’ documentation.

What are good tools for working with a backend server?

I’ve used a web browser, cURL, openssl client, and Python for getting to a back end REST server. I’ve recently discovered HTTPie. For example

https GET https://127.0.0.1:7554/application/loggers –cert colinpaice.pem –cert-key colinpaice.key.pem –verify ca.pem2

This invokes some Python code which does all the work. It produced

The documentation is here.

Keep a session

By default each request is a single shot, nothing is retained (such as a Java Web Token) so you need to enter your credentials every time.

However, HTTPie also supports persistent sessions via the --session=SESSION_NAME_OR_PATH option. In a session, custom HTTP headers (except for the ones starting with Content- or If-), authentication, and cookies (manually specified or sent by the server) persist between requests to the same host.

Named sessions

You can create one or more named session per host. For example, this is how you can create a new session named user1

http --session=user1 -a user1:password ....

From now on, you can refer to the session by its name (user1). When you choose to use the session again, all previously specified authentication or HTTP headers will automatically be set.

Plugins

There are plugins available see here. For example authentication

Not for humans – AT-TLS and security messages

EZD1287I TTLS Error RC: 6 Initial Handshake

6 Key label is not found

My server certificate was in the keyring, but it had expired. I renewed it, and got past this.

EZD1287I TTLS 403 No certificate received from partner.

403 No certificate received from partner.

In my curl request I did not have –cert ./colinpaice.pem:password –key ./colinpaice.key.pem, when the server expected a certificate (gpmserve had CLIENT_CERT(ACCEPT)).

My z/OS had HandshakeRole ServerWithClientAuth, but the client did not provide a certificate.

EZD1287I TTLS Error RC: 428 Initial Handshake

The private key cannot be obtained from the certificate.

The server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

EZD1287I TTLS Error RC: 435 Initial Handshake

435 Certification authority is unknown.

I got this having replaced the CA certificate. Deleting a certificate removes it from any keyring. When you recreate the CA, you need to add it to every keyring it was in. Before deleting a certificate it is worth listing it to see where it is used. I added it to my keyring and it worked!

EZD1287I TTLS Error RC: 510 No acceptable key labels found

510 No acceptable key labels found

My server certificate was in the keyring, but it did not have the capabilities to support the handshake. For example client only supported RSA, but server was ECC.

pthread_security_np, pthread_security_app_np

This function call can be used to map a certificate to a userid.

I got

ESRCH (143): The user ID provided as input is not defined to the security product or does not have an OMVS segment defined.

errno2 0be8044c. Code 044c is JRNoCertforUser, There is no userid defined for this certificate. Action: Ensure the userid is known to the SAF service.


You also get this message if the mapping from certificate to userid is missing. For example

RACDCERT DELMAP(LABEL('IBMUSER1Label))ID(IBMUSER)
RACDCERT MAP ID(IBMUSER) -
   WITHLABEL('IBMUSER1Label') -
   SDNFILTER('CN=colinpaice.O=cpwebuser.C=GB')
RACDCERT LISTMAP ID(IBMUSER)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

R_ticketserv (IRRSPK00): Parse or extract 8, 16, 28

I was using the service to generate a pass token, and got SAF 8 RACF 16 RS 28.

Unable to generate PassTicket. Verify that the secured signon (PassTicket) function and
application ID is configured properly by referring to Using PassTickets in z/OS Security Server RACF Security Administrator’s Guide.

Action

I needed to define

RALTER  PTKTDATA MYAPPL  SSIGNON(KEYMASKED(7E4304D681920260)) - 
    APPLDATA('NO REPLAY PROTECTION')

Where MYAPPL is the application name.

R_ticketserv (IRRSPK00): Parse or extract 8, 8, 16

With userid = “ADCBD” and APPL = ” MVSS0W1″ I got SAF 0 RACF 0 RS 0.

With userid = “ADCBD” and APPL = ” TSOS0W1″ I got SAF 8 RACF 8 RS 16. Not authorized to use this service.
I used

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE)

and it worked

EZD1287I TTLS Error RC: 467 Initial Handshake

Running from Python – a request worked. Running from Chrome gave me the above message.
I solved it by making the connection TLS 1.3 ON and TLS 1.2 OFF

The doc (Cryptographic Services System Secure Sockets Layer Programming)

says

467 Signature algorithm not in signature algorithm pairs list.
Explanation
A signature algorithm that is used to sign a local or peer certificate is not included in the signature algorithm pairs list or the certificate signature algorithm pairs list. The server certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the client during the TLS handshake. The client certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the server during the TLS handshake. If remote partner specifies the signature algorithm pairs and the certificate signature algorithm pairs, the specified certificate signature algorithm pairs take precedence over the signature algorithm pairs. For TLS V1.2 handshakes, the GSK_TLS_CERT_SIG_ALG_PAIRS setting is only used on the client
side to indicate the signature algorithms that the client supports in the server’s certificate chain.
User response
Ensure that the signatures of the local and peer certificates in the certificate chain use signature algorithms that are present in the signature algorithm pairs list or the certificate signature algorithm pairs list that is presented by the session partner. If the certificate chain is correct, configure the client or server or both to specify all necessary signature algorithms pairs in the GSK_TLS_SIG_ALG_PAIRS or SK_TLS_CERT_SIG_ALG_PAIRS
settings to allow use of the certificate chain. If GSK_TLS_CERT_SIG_ALG_PAIRS is specified, it takes precedence while checking the signature algorithms used in the certificate chain.

Zowe: setup: configuring end user userids

Connect the userid to the right groups.

It is better to give permissions to groups, rather than to individual userids.

If someone changes jobs, within your organisation, you just remove the userid of the person leaving from the group, and they lose all access.

If you have given access to a userid, you have to go through all resources (for example APPLs) and remove access.

When a new person joins, you just connect them to the group, and they should have access.

Zowe groups

If you are going to use Zowe, then user userid needs access to the Zowe group.

  • If class(APPL) OMVSAPPL is defined, the Zowe group needs read access to it
  • If users will be using z/OMSF they will need read access to the class(APPL) IZUDFLT (or what ever is specified)
  • If certificate logon is being used, the certificate will need to map to a userid

Access z/OSMF

If userid will be using z/OSMF they need access to the z/OSMF group.

In the z/OSMF logs, I got

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

I connected userid COLIN2 to the z/OSMF group IZUUSER. I think the change is available immediately.

Access to OMVSAPPL

An application can switch userid by using the pthread_security_np, and passing in a userid and password or a certificate (which maps to a userid).

The application can control which userid can be used by means of an resource in the APPL class.

If the resource was called MYAPPL, then pthread_security_np would use “MYAPPL”, and the userid being switched to much gave read access to MYAPPL.

The default (and only supported) resource is called OMVSAPPL.

If OMVSAPPL in class(APPL) is defined, the userid must have read access to it. z/OSMF also uses OMVSAPPL.
If OMVSAPPL in class(APPL) is not defined, no checks are done.

With Zowe you cannot specify the resource name.

Certificate logon

You can use a digital certificate from a web browser ( curl, or other tools) to authenticate to z/OS.  You need to map the certificate to a userid.

A certificate coming in can have a Distinguished Name like CN=adcdd.O=cpwebuser.C=GB  (Note the ‘.’not ‘,’ between elements).

Your userid needs to have SPECIAL define to be able to use the RACDCERT command (SPECIAL, not just GROUP-SPECIAL).

You will need a definition like (see here for the command)

RACDCERT MAP ID(ADCDD ) - 
    SDNFILTER('CN=adcdd.O=cpwebuser.C=GB') - 
    WITHLABEL('adcdd')

or a general definition for those certificate with  O=cpwebuser.C=GB, ignoring the CN part

RACDCERT MAP ID(ADCDB ) - 
   SDNFILTER('O=cpwebuser.C=GB') - 
   WITHLABEL('cpwerbusergb')

or using the Issuing Distinguished Name (the Certificate Authority)

IDNFILTER(‘CN=TESTCA.OU=SSSCA.C=GB)

Using a generic

SDNFILTER(‘CN=a*.O=cpwebuser.C=GB’)

does not work.

If you attempt to use a certificate which is not mapped you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(COLIN)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0163)  SUBJECT(CN=adcdd.O=cpwebuser.C=GB) ISSUER(CN=SSCA8.OU=CA.O=SSS.C=GB).

It is worth defining these using JCL, because if you try to add it, and it already exists then you get a message saying it exists already.  If you know the userid, you can list the maps associated with it.   If you do not know the userid, there is no practical way of finding out – you have to logon with the certificate, and display the userid from the web browser, or extract the list of all users, and use LISTMAP on all of them.

Zowe: Planning: How do I protect what Zowe users can do on z/OS?

As part of my planning for Zowe, I wanted to know how I could control what Zowe users can do on z/OS. I could not find any definitions for security profiles, so how do I do it?

It took a few days thinking about this to realise I was looking a the problem the wrong way. The correct way of looking at it, is that Zowe is a transport system for getting requests from a user’s work station to z/OS. This is similar to a 3270 emulator connection to z/OS. You control what the userid can do, and do not try to control what the 3270 emulator can do.

When a userid logs on to TSO through a 3270 emulator, z/OS knows the userid of the address space, and can control access to what resources the userid can access.

When a userid logs on to Zowe there are two paths that can be taken:

  • Zowe can create a TSO address space for the userid, using the CEA facilities. The z/OS® CEA TSO/E address space manager provides services to programmatically start and manage TSO/E address spaces and provides a communications mechanism for use between the caller and the programs running in these managed address spaces.
  • A thread within Zowe can use the pthread_security and change the userid of that thread. It can use your certificate, or a userid and password to validate the user. At the end of the request it resets the userid back.

From a permissions perspective, it does not matter if request came into a TSO address space or as a result of the pthread_security request. The userid is extracted and normal SAF processing is used to manage access to a resource.

But…

Your system may have IPSEC rules which police traffic into and out of TCP/IP on z/OS, for example allow traffic from these external IP address during 0900 to 1700 Monday to Friday, and deny access at any other time.

You might need to have similar rules for connectivity through Zowe. There are several ports used by Zowe and z/OSMF. You need to review what controls you need for these ports to stop unwanted traffic from accessing your system.

Zowe: Which messages to automate

In the Zowe.yaml file is a section sysMessages. This is a list of the messages that are displayed on syslog. You can add messages to these.

The provided list is

  • ZWEAM000I %s started in %s seconds, giving the component and the duration of the startup
  • ZWED0031I The appServer is ready at ipAddress, Plugins successfully loaded: percentage% (successful/total)
  • ZWEL0001I component %s started. A start request was issued for the component
  • ZWEL0002I component %s stopped. A request was made to stop the component
  • ZWEL0006I starting components. A request was made to start the component specified as enable: true in the zowe.yaml file.
  • ZWEL0008I stopping components. A request was made to stop Zowe, and Zowe is shutting down.
  • ZWEL0018I Zowe instance prepared successfully. The pre-starting checks have been completed. This message is followed by ZWEL0006I starting components.
  • ZWEL0021I component %s stopped
  • ZWEL0022I Zowe Launcher stopped. This is the last message that Zowe emits.
  • ZWES1013I ZSS Server has started. Version ‘%s’ ‘%s’
  • ZWES1601I ZSS Server is ready to accept JWT with|without fallback to legacy tokens