Zowe: Planning – certificates

A private key is used for encryption and should be kept private. If someone has access to the private key they can impersonate you.

A public key is the opposite of the private key. It is used for decrypting data encrypted with the private key. The public key can (and should) be generally available.

Certificates are used in TLS for authentication and encryption, and can be used for identification. They include a public key.

A Certificate Authority(CA) certificate is used to validate other certificates. It involves doing a checksum of a certificate, and encrypting it with the CA private key – a process known as signing the certificate

To validate a certificate, the recipient needs a copy of the CA which was used to sign the original certificate. It can the decrypt the encrypted checksum, and compare it with the certificates checksum.

If you create a Certificate Authority certificate you will need to distribute this to all machines that might communicate with your server(possibly thousands of machines), and installed into the keystore on those machines. This can be a big task. Most system have one site wide CA certificate which is distributed to all machines. You might have a second CA to limit access to a system. This CA is used to sign the server certificates.

Creating certificates

As part of creating a certificate you create the private/public keys. There are different algorithms for these keys. Some are stronger than others (it takes more time and CPU to break them) Keys with elliptic curve algorithms are generally stronger than using RSA techniques, and there other techniques resistant to Quantum Computing.

You have to use a server certificate with RSA and keysize 2048.

I found that authentication with JWT (Java Web Tokens) only worked with RSA keys and not Elliptic Curves. This is because of encryption with JWT.

Key stores

You should use keyrings rather than .pem files, as they are more secure. .pem files can be copied, and anyone with authority can copy them. You have to give explicit permission to be allowed to access a keyring. Certificate and the private keys within a keyring can be stored in cryptographic hardware, and the private keys are never exposed in clear text.

Many systems uses a keystore for storing the private key used by the server, and a trust store for the Certificate Authority keys needed to validate any client certificate sent to the server. This can, but is not recommended, be the same as the keystore.

The keystore will need the server key. You can specify which key should be used.

  • the Certificate Authority keys needed to validate the server key.
  • the server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

The trust store needs all the CA certificates which may be needed to verify a client certificate.

The client machine will have one or more certificates. A copy of the CA used to create these needs to be installed in the trust store on the server.

I understand that if you add a certificate to the trust store, you need to restart Zowe for it to be picked up, so try to get a list of all the CAs you will need before you start.

Validating certificates

The trust store needs the certificates to validate any client certificates sent to the server. This will usually just be Certificate Authority certificates.

Zowe works with z/OSMF. They communicate with certificates. The z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key. If they are not set up properly you can get messages “LoadBalancer does not contain an instance for the service zaas”.

If you have a site wide CA, which is the same for both of the server keys, then you do not need to do any more work, as both trust store keyrings will already have the CA. If Zowe and z/OSMF have different CA certificates, the CA certificates need to be connected to the other keyrings.

Subject Alternative Name

The Subject Alternative Name within a certificate provides the IP addresses, or IP Names of the server. A client can check this address with the IP address of the session, and terminate the session if they do not match. This is considered best practice. This check can be disabled in Zowe by using verifyCertificates NOSTRICT in the zowe.yaml file.

RACF allows one name or address when the certificate is defined, for example ALTNAME(IP(127.0.0.1))

On my z/OS system the command TSO NETSTAT HOME gives three addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6.

You can configure your sysplex, so all systems in the sysplex have the same IP address, and traffic gets routed internally to the correct system. Without this, if you start a server on a different LPAR it will have a different IP address, and so the validation will fail.

On my system, the zOSMF certificate did not have an ALTNAME specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work, until I fixed the certificate.

If the z/OSMF certificate has an ALTNAME(IP…) specified, use the IP address value when you configure zOSMF for example

zOSMF: 
host: 127.0.0.1
port: 10443
applId: IZUDFLT

Mapping certificates

If you are using client certificate to authenticate rather than a userid and password, then you’ll need to map certificates to userids, for example with the RACDCERT MAP command. You can specify which CA the certificates were signed with, and fields from the subject Distinguished Name. The question of which is better: using userid and password, or client certificate to logon has no easy answer

  • It is easy for a hacker to get a password (lost handbag, yellow sticky stuck to the screen, a couple of pints down the pub) It is easy to change a password once is is compromised.
  • It is harder for a hacker to get a certificate – but it is harder to change and re-issue a certificate. You have to get the updated certificate down to the client’s machine. It could be stolen if a hacker has access to the machine.
  • Using Biometric data to logon is the ultimate limit in this area. Hackers could steal it – but there is no way of changing it if it is compromised!

Decisions

You need to decide

  • Are you going to have a CA just for Zowe? or reuse the site CA.
    • If you are going to have a Zowe specific CA – how are you going to distribute it to all the client machines.
    • You’ll need to ensure Zowe and z/OSMF have the other’s CA certificates in their trust store.
  • Are you going to use Subject Alternative Name ALTNAME(IP(10.1.1.2))
    • What value of verifyCertificates STRICT|NOSTRICT are you going to specify in the zowe.yaml file.
  • Are you going to authenticate using certificates. You will need to set up mapping from certificate to userid. RACDCERT MAP
  • If you will be using JWT, and so need an RSA key in the server certificate

Zowe: Planning

See Preparing for installation in Zowe publications for detailed information on the structure of Zowe files and data sets.

Topic covered


Required products

  • Z/OS 2.5 or later
  • Java 17 IBM® Semeru Runtime Certified Edition for z/OS® version 17
  • Node.js version v18. or later. See here.

Optional products

  • z/OSMF. This is used for Authentication and REST API services.
  • SDSF. This is used to issue console commands, and capture the output

Disk storage

  • For the Zowe product. This typically is in a directory like /usr/lpp/zowe. It needs about 1300 MB, or 1600 CYL.
  • For each instance use Primary 118, Secondary 18 Cylinders.
  • For the .pax file 440 MB
  • Some PDSs are generated. Together they use about 20 Cyl of DASD

CPU

There is a CPU spike when Zowe is started up. It starts several address spaces running Java. Once active the CPU costs depend on how much Zowe is used, plus a steady low overhead.

TCP/IP Interfaces

A TCP/IP configuration can have multiple IP addresses. When Zowe initiates a connection to one of its components, it needs the correct IP address.

This is configured in zowe.externalDomains and can be a domain name or IP address.

Ports

The default ports in the configuration are

  • gateway 7554
  • zaas 7552
  • discovery 7553
  • caching services 7555
  • app server 7556
  • zss 7557
  • infinispan in memory caching, port 7600
  • infinispan in memory caching key exchange, port 7600

These ports can be used by clients outside of z/OS, and between the Zowe started tasks. You need to configure your network and firewalls to allow traffic on these ports.

High level directory structures

The product libraries have a name including the release, such as /usr/lpp/zowe/zowe32. The directory is used within the zowe.yaml file and shell scripts such as the bash .profile. You might want to consider an alias for the product libraries. Such as /global/zoweprod -> /usr/lpp/zowe/zowe32. To upgrade all users, just change the alias, and they will not need to change their scripts.

By default

  • log files are stored in /global/zowe/logs
  • instance specific internal information is stored in /global/zowe/workspace
  • extensions are stored in /global/zowe/extensions.

The /global directory, also known as the global filesystem, serves as a centralized location for storing and sharing files across multiple systems and users. There may be a different file system for /u on each system and so a Zowe instance configured in /u/… may not be visible across systems. If you may run Zowe instance on more than one machine you need to ensure the file system is available on all potential machines.

If you are going to use more than one Zowe instance, you need to plan for this. For example use the Zowe instance name as part of the directory. Perhaps use /global/zowe/logs/ZOWE1, or /u/zowec/ZOWE1/logs.

Having the system name SYS1 as part of the file path can be confusing if you will start the Zowe instance on more than one LPAR.

Where to store extensions

You may want to share the same instance of /global/zowe/extensions across all Zowe instances, or have each Zowe instances to use their own directory.

Automation and monitoring

Zowe can be configured so that important messages like Zowe has started and Zowe is stopping are written to the system log. You may need to change your automation to react to these messages. For example

+ZWEL0018I Zowe instance prepared successfully                        
+ZWEL0006I starting components
+ZWEL0001I component gateway started

+ZWEL0008I stopping components
+ZWEL0002I component gateway stopped
+ZWEL0022I Zowe Launcher stopped

See Which messages to automate?

Understanding the configuration process

z/OS system configuration

As part of the installation, one of the first steps is to run a script which extracts PDS from the Zowe files.

There are two load libraries which will need to be APF authorised.

Two started task procedures will be created. One started task is a server which can run authorised code. The other runs a Java program.
When Zowe starts up it spawns threads that run as BPXAS address spaces. The names of the spawned threads are like ZWE1AD. The prefix ZWE1 can be changed in the configuration file.

The cross memory server started task needs parameters. These can be put in the SYS1.PARMLIB concatenation, or in a Zowe PDS.
An update is needed to parmlib for PPT entries to set the storage key for Zowe.

The Zowe default dataset names have a prefix IBMUSER.ZOWEV3. You can use your own names, such as PP.ZOWEV3 (for Program Products).

Workload manager definitions

You should set up WorkLoad Manager definitions for the address spaces.
Note: If a WLM definition has discretionary, any Java work will not run on GCPs if the ZIIPS are busy, so you need care in specifying which rule applies to the address spaces.

Security definitions

You should protect the data sets produced by Zowe configuration; the sample library is not sensitive, but the load libraries are APF authorised, and these APF authorised libraries should be protected from being changed by an unauthorised user.

What global FACILITY profiles are used

As part of the configuration, security definitions are created.

  • Some changes are global, enabling some system wide facilities. Most systems will typically have these already enabled, but if not, you may want to configure them early to give plenty of time to test them.
    • Activate classes STARTED and FACILITY
    • Define FACILITY BPX.DAEMON
    • Define FACILITY BPX.SERVER
    • Define FACILITY BPX.JOBNAME
    • Define FACILITY IRR.RUSERMAP
    • Define FACILITY IRR.IDIDMAP
    • Define FACILITY IRR.RAUDITX
  • Some changes are Zowe related, such as userid for Zowe. This defaults to ZWEUSR
    and group ZWEGRP.
    • Add group ZWEADMIN.
    • Add userid definitions for ZWESVUSR and ZWESIUSR.
    • Defines started task profiles for STARTED; ZWESLSTC, ZWESISTC, and ZWESASTC* (where ZWE is configurable).
  • Other changes
    • A data set profile IBMUSER.ZWEV3..* for the data sets that Zowe install creates.

You should consider how user’s will authenticate.
As part of the TLS session initialisation the server can request the client’s certificate and use information from the certificate to determine the userid to be used.
You can authenticate using:

  • Userid and password
  • using z/OSMF
  • Multi Factor authentication – such as having a widget or mobile phone
    application which generates a code as a one off, time limited code as a password.

You can use AT-TLS.

How do I protect what Zowe users can do on z/OS?

Zowe is a transport mechanism like a 3270 emulator. You control what a userid can do, not what the transport mechanism can do. See here

TLS levels

The levels from the example .yaml file is

tls: 
attls: false
# TLS settings only apply when attls=false
# Else you must use AT-TLS configuration for TLS customization.
minTls: TLSv1.2
maxTls: TLSv1.3

If you are not at TLS v1.3 you may want to start at TLS 1.2, and convert to TLS 1.3 at a later date.

Certificates

See a whole blog post on planning for certificates.

PPT entries

Two modules have to be defined in a SCHED member with properties KEY(4) NOSWAP.

Operations

You need to start the two started tasks. The authorised server is active and does not do much. You can start it at IPL and leave it running.

You start the Zowe main task when you want to use Zowe. This spawns several threads in BPXAS address spaces. These are called ZWE1xx – where the ZWE1 can be customised.
You use the stop z/OS command (P …) to terminate the main task.
When it shuts down, it terminates the services running in BPXAS address spaces.

Data management

Zowe does not hold any state information. The configuration files in the Zowe instance home directory should be backed up regularly, as well as the proclib and parmlib members.

Running in a sysplex

The Zowe product libraries can be shared across all LPARs in a Sysplex. You may want to consider mounting it read only.
A Zowe instance can be started on any LPAR, provided the instance home directory, and other directories it uses, are available and writable.

You can create more Zowe instances on one LPAR ( why would you?) or for availability you can create a Zowe instance on more than one LPAR.

High availability

If you have a Zowe instance running on more than one LPAR, if one instance is shutdown, you can (re)connect to a different solution, and logon.
You can also configure a Zowe HA environment, where you can reconnect to an alternate instance, without needing to logon.

Migrating to a new level of Zowe

Install the new level of Zowe into a different directory to your current system.
The process will update the Zowe PDS datasets:

  • …SZWEAUTH,
  • …SZWEEXEC,
  • …SZWELOAD,
  • …SZWESAMP

so you should back these up, or change the config file to use a different data set prefix.
To overwrite these datasets you need to explicitly set an option.

Backup the configuration files, so if you need to go back to the current level of code, you just rename the files and data sets and restart Zowe.

Before you start – what configuration path to take?

I was talking to someone about installing software on z/OS and how it could be made easier, and they said the biggest challenge is getting changes past the grey haired person in the corner office whose job is to say No. As in

  1. No: You cannot “move quickly and break things”. You must move carefully and keep safe.
  2. No: You cannot install untested stuff on production machines, without installing it first on test or development machines
  3. No: You cannot install products where you have to type commands or information onto the production machines. You have to provide files of information, or a document where commands or information is reviewed and can be copied using cut-and-paste. For example people have had problems with the letter o (for Orange) and zero, and i (for India) 1’s and l (for Lima).
  4. No: You cannot just run a script or command which changes z/OS configuration. All changes have to be reviewed. For example some changes to the security configuration affect all address spaces.
  5. No: You cannot run something that is “one way” for example adding a column to a data base table, to sort data, or delete something without a tested process for restoring and recovering the data.
  6. No: You cannot have the authority to issue commands, because you may issue commands outside of the documented install script, without going through change control.
  7. No: You cannot install something until we know the impact of running it. For example, the amount of CPU, real, virtual, disk storage.

Zowe

Zowe is a project which provides a command level interface from outside of z/OS to z/OS services. This means people do not need to know about ISPF, or the operator console to be able to use z/OS. Zowe has several Java address spaces which provide the facilities.

As part of the aim to make z/OS more accessible, the Zowe installation and configuration has command scripts which can be used to define the resources needed for Zowe to run.

I feel the “recommended” processes for configuring Zowe is fit for a test or development system, not for an important system, and so I recommend using a more traditional approach, which fits in the “No:… above”

Not all the configuration changes needed are listed in the documentation, for example you should configure WLM to give the address spaces appropriate priority. You need to backup data sets and files.

The security definitions enable a global facility which can impact many subsystems. This change alone would need careful roll out to all systems.

APF definitions are made using dynamic APF command interface on the console. They are not saved across IPL, so the changes have to be made to the SYS1.PARMLIB concatenation at some point. My view is just do the parmlib definitions once.