Zowe: Planning: How do I protect what Zowe users can do on z/OS?

As part of my planning for Zowe, I wanted to know how I could control what Zowe users can do on z/OS. I could not find any definitions for security profiles, so how do I do it?

It took a few days thinking about this to realise I was looking a the problem the wrong way. The correct way of looking at it, is that Zowe is a transport system for getting requests from a user’s work station to z/OS. This is similar to a 3270 emulator connection to z/OS. You control what the userid can do, and do not try to control what the 3270 emulator can do.

When a userid logs on to TSO through a 3270 emulator, z/OS knows the userid of the address space, and can control access to what resources the userid can access.

When a userid logs on to Zowe there are two paths that can be taken:

  • Zowe can create a TSO address space for the userid, using the CEA facilities. The z/OS® CEA TSO/E address space manager provides services to programmatically start and manage TSO/E address spaces and provides a communications mechanism for use between the caller and the programs running in these managed address spaces.
  • A thread within Zowe can use the pthread_security and change the userid of that thread. It can use your certificate, or a userid and password to validate the user. At the end of the request it resets the userid back.

From a permissions perspective, it does not matter if request came into a TSO address space or as a result of the pthread_security request. The userid is extracted and normal SAF processing is used to manage access to a resource.

But…

Your system may have IPSEC rules which police traffic into and out of TCP/IP on z/OS, for example allow traffic from these external IP address during 0900 to 1700 Monday to Friday, and deny access at any other time.

You might need to have similar rules for connectivity through Zowe. There are several ports used by Zowe and z/OSMF. You need to review what controls you need for these ports to stop unwanted traffic from accessing your system.

Zowe: Which messages to automate

In the Zowe.yaml file is a section sysMessages. This is a list of the messages that are displayed on syslog. You can add messages to these.

The provided list is

  • ZWEAM000I %s started in %s seconds, giving the component and the duration of the startup
  • ZWED0031I The appServer is ready at ipAddress, Plugins successfully loaded: percentage% (successful/total)
  • ZWEL0001I component %s started. A start request was issued for the component
  • ZWEL0002I component %s stopped. A request was made to stop the component
  • ZWEL0006I starting components. A request was made to start the component specified as enable: true in the zowe.yaml file.
  • ZWEL0008I stopping components. A request was made to stop Zowe, and Zowe is shutting down.
  • ZWEL0018I Zowe instance prepared successfully. The pre-starting checks have been completed. This message is followed by ZWEL0006I starting components.
  • ZWEL0021I component %s stopped
  • ZWEL0022I Zowe Launcher stopped. This is the last message that Zowe emits.
  • ZWES1013I ZSS Server has started. Version ‘%s’ ‘%s’
  • ZWES1601I ZSS Server is ready to accept JWT with|without fallback to legacy tokens

Zowe: messages on z/OS when using Zowe

ZWEAO503E The server is not ready to handle the request: /apicatalog/api/v1/

I got this when the ZAAS service was stopped, use the f zowe,appl=DISPLAY to display the status of the various tasks.

You can use f zowe,appl=start(ZAAS) to start it. Check the zowe sysprint data set for any error messages.

I also got the following message in the z/OSMF logs

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user adcdb has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

BPXTLS failed: rc=-1, return code=143, reason code=0x0be80000


Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required
Other platforms: impersonation is not supported

This is actually from BPX1TLS, phread_security_np. There are not many cases where 0be80000 is mentioned. This link mentions PTKTDATA profiles.

LoadBalancer does not contain an instance for the service zaas

This was a tricky one to track down.

Zowe works with z/OSMF. They communicate with certificates. This means the z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key.

Not only that , if verifyCertificates STRICT is specified in the zowe.yaml file, then Zowe will check the server’s certificate sent from z.OSMF. This includes the altname or altIP, the altIP is the IP address of the connection. An internal IP address is 127.0.0.0. An external IP address is almost anything else. On my system I have addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6. You can display them using the TSO NETSTAT HOME command.

The zOSMF certificate did not have an altip specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work.

You can also get this message when the system is partially up.

ZWEAM701E; InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name

The request to the URL ‘/zaas/api/v1/auth/login’ has failed: SignatureException: Unable to compute RS256 signature. Cause: Unrecognized RSA or RSASSA-PSS key algorithm name. caused by: InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name. messageAction:Refer to specific exception details for troubleshooting.

It looks you Zowe server needs an RSA key for use with JWT. I changed from an Elliptic key to an RSA and it worked.

Using the web browser

403 Impersonation error

  • Error: response status is 403
  • Download
  • Impersonation error

In the logs/zssServer….log it has

BPXTLS failed: rc=-1, return code=139, reason code=0x0be803d1
Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required

This is because Zowe does not have access to the profiles.

Action:
Check the profiles are defined, and the Zowe userid has UPDATE access to them.

Header X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I actually got the message ZWEAG160E No authentication provided in the request 5 times.

The header is collecting all errors during the treatment of the request. Because the response was 503 the Gateway retries it 5 times. This is strange, but expected behaviour.

I also got an error message in the z/OSMF log /global/zosmf/data/logs/zosmfServer/logs/trace.log

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

Give the userid access to the profile. It may be easier to connect the userid to the group. In my case to group IZUUSER.

I gave the userid access and then got

IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request

{
“additionalInfo”: null,
“debug”: null,
“httpStatus”: 403,
“messageID”: “IYUCM0008W”,
“messageText”: “IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services.”,
“requestMethod”: “GET”,
“requestUri”: “/zosmf/provisioning/rest/1.0/scr”
}

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I’m not sure about this. Certificate authentication should work, but I needed to specify

–basic –user colin2:password

Python

Fatal (HANDSHAKE_FAILURE): Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

This post said

I got the Insufficient buffer remaining for AEAD cipher fragment error when trying to use the Python client (browser and curl worked normally). After lots of trial and error, it seems the error was related to the SSL certificates not having properly configured extension fields, especially keyUsage and extendedKeyUsage, and the fact that urllib3 doesn’t include /etc/ssl/certs/ca-certificates.crt by default.

Another post said “This is an OpenJDK issue with TLS 1.3 “occurring between the OpenShift default route and Mule applications with re-encypt route enabled.”

Keystore file message is confusing

I saw the following in a GW trace record

[35mZWESVUSR [0;39m [36mINFO [0;39m((o.a.t.u.n.N.certificate)) Connector
[https-jsse-nio-0.0.0.0-7554], TLS virtual host [default], certificate type [UNDEFINED] configured from keystore [/u/tmp/zowec/.keystore] using alias [CONN1.IZUDFLT] with trust store [null]

it was confusing because I was not using a keystore file. This looks like a bug in Spring, and can be ignored.

ZWEAG121E Authorization header is missing, or the request body is missing or invalid for URL ‘/zaas/api/v1/auth/login’

  • messageAction: Provide valid authentication.
  • messageReason: The authorization header is missing, or the request body is missing or invalid.

I got this when trying to do a certificate login – and no password.

  • The certificate used did not have a mapping to a userid.
  • The userid was revoked

ZWEAO500E: The service has encountered a situation it doesn’t know how to handle.

Please contact support for further assistance. More details are available in the log under the provided message instance ID.

Case 1

I got this when I changed the a logging value using the REST API. I had

“configuredLevel”: “WARN2″‘

which is invalid.

Case 2

In the log I got

java.lang.ClassCastException: com.ibm.crypto.plus.provider.ECPublicKey incompatible with java.security.interfaces.RSAPublicKey

This is because the key type in my keyring was an Elliptic curve, but the code assumesit is RSA, and uses an RSA function – which does not work with an Elliptic key.

HTTPIE: Https error: ConnectionError: HTTPSConnectionPool(host=…, port=…)

Max retries exceeded with url: … (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’)) while doing a GET request to URL: https://…

The back end was not active so the TLS hand shake failed.

How do you get into the well protected castle? – you tunnel

Before I retired, if I wanted access to the corporate systems, I would start up a tool (vpn/dialer) which set up an encrypted session to the corporate front end, and tunnel through this to get to the back end. It was transparent and was like being directly attached.

How do you set up your castle to allow authorised people in – but deny access to unauthorised access?

If you logon directly to a back-end server from a cafe or other public WiFi, your credentials etc may be in the clear. You need to use “a secure app” such as banking app, or provide a vpn or tunnel to access your system to encrypt the traffic.

I’ve been logging on to someone else’s system and most of the useful connections to this are disabled. I could access via 3270 but could not use FTP etc.

I was trying to debug a TLS problem and found suspicious evidence in the trace. A connection was being made to the TLS port on my server, and failing because it did not speak TLS. Someone had clearly found the IP address of my system, and was trying all ports to get in!

How do you protect the system and keep unwanted people out – but allow authorised people to connect. You need to do both of…

Secure tunnelling

You could use TLS to protect the conversation between client and server, but this means you have allowed the connection to get through TCP/IP and to your application before checking to see if the connection is permitted.

Port forwarding or tunnelling controls access to your system at the outer edge of your system, logically within TCP/IP, before it gets to your application. It is well described here.

I used the SSH command

ssh -N -L 9876:12.23.34.45:8765 colin@23.34.45.56 -p 2222

If I used address https://localhost:9876 in my web browser, I am connected to 12.23.34.45:8765 through system 23.34.45.56 .

Where

  • -N says port forwarding
  • my local port 9876 is linked to 12.23.34.45 port 8765
  • via IP address 23.34.45.56. Think of this as the guard house.
  • colin is the userid. You get prompted for your password.
  • use port 2222 at the guard house. Think of multiple entrances to the guard house. “General public” “special guest” “people who work here”. The port says go to the “special guest” door. The first time, you have to successfully logon with your userid and password to get your details recorded as a valid visitor to the site. You are given a token which is saved for next time you want access (think of it as a visitor badge).

What is IPSEC?

IPSEC is part of Communications Server on z/OS. It provides

  • IP filtering to control which packets can enter the system
  • IP filtering to control which packets can leave the system

You can filter on

  • packet information – allow a ping, but not FTP
  • network attributes – only allow from a list of IP address
  • time – at certain times of day. For example a normal working day – so if someone is trying to access this system at two o’clock in the morning, this would be worth investigating.

You can

  • allow
  • deny
  • log

You need to be careful setting the rules up – if you want to allow traffic in from 1.2.3.4, then you need to allow traffic out to address 1.2.3.4

My rule

I would set up an input rule like

For input port 8765 deny all access, all packet types, all hours of the day.

Using the tunnel needs a little care

The browser validates the certificate sent from the server. If there ALTNAME, the client(browser) should check that the IP address specified in the ALTNAME matches the IP address used in the original URL, and your connection has not been high jacked.

In my browser I used https://127.0.0.1:9876, where 9876 was specified in the ssl command above. The certificate at the server had ALTNAME with IP address of 127.0.0.1, and so this was considered valid. The IP ALTNAME IP adddress is usually the IP address of the server (or one of them if there are more than one) so you may get messages saying an insecure certificate is being used. You can accept this, or configure your browser to ignore these checks (which is not a good idea)

Thanks

Thanks to Lionel Dyck and Randy Rackov for their help in this blog post.

Zowe: Setting up the Zowe side for a web browser

If you want to use your web browser to connect to a Zowe server, it is just like connecting to any other back end server.

  • The browser need the CA certificate used to sign the Zowe server’s certificate.
  • The server needs the CA used to sign the browser’s certificate, in the server’s trust store.

The browser need the CA certificate used to sign the Zowe server certificate.

If you do not have a copy of the exported certificate you can export it

//COLINEXP JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CERTAUTH EXPORT(LABEL('DOCZOSCA')) -
DSN('COLIN.CERT.DOC.CA.PEM')
//

This will delete the dataset, recreate it and export the certificate.

This exported certificate will look like

-----BEGIN CERTIFICATE-----                                          
MIIBtzCCAV6gAwIBAgIUb0N8Xwb/WNulhgjzRcIwVBmfAXwwCgYIKoZIzj0EAwMw
...
S9QFQcy5vKGFFAQ=
-----END CERTIFICATE-----

Copy this file to where ever it is needed.

On Linux you can use the command

openssl x509 -in cbt.ca.pem -text -noout|less

to display the contents

Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = COLIN, OU = CA, CN = DOCZOSCA
Validity
Not Before: May 2 05:00:00 2025 GMT
Not After : Jul 3 04:59:59 2029 GMT
Subject: O = COLIN, OU = CA, CN = DOCZOSCA
...

Note: The base64 command does not decode the base64 content properly.

Import this file into your browser

For example with Chrome. Settings, search for certificates, click on security, click on Manage Certificates. Click on Customised, Trusted certificates, Import. Select the file name. You may have to select All Files at the bottom to get your file included in the list.

The file should now be installed, and be listed under Trusted certificates. Click on the pen icon to display it.

The server needs the CA used to sign the browser’s certificate it the server’s trust store.

Create a VB data set on the Zowe z/OS machine. Copy the contents of the client’s CA into this data set. I used cut and paste.

//COLINADD JOB 1,MSGCLASS=H 
//* ADD ORKENY CA CERTIFICATE
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.CERT.DOC.CA.ORK')
RACDCERT DELETE -
(LABEL('ORKCA256')) CERTAUTH
RACDCERT CERTAUTH ADD('COLIN.CERT.DOC.CA.ORK')-
WITHLABEL('ORKCA256') TRUST
RACDCERT CERTAUTH LISTCHAIN(LABEL('ORKCA256'))

RACDCERT ID(COLIN) CONNECT(RING(ZOWE) -
CERTAUTH -
LABEL('ORKCA256'))
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

When this is successful, you will need to shutdown and restart Zowe.

Zowe: Planning – certificates

A private key is used for encryption and should be kept private. If someone has access to the private key they can impersonate you.

A public key is the opposite of the private key. It is used for decrypting data encrypted with the private key. The public key can (and should) be generally available.

Certificates are used in TLS for authentication and encryption, and can be used for identification. They include a public key.

A Certificate Authority(CA) certificate is used to validate other certificates. It involves doing a checksum of a certificate, and encrypting it with the CA private key – a process known as signing the certificate

To validate a certificate, the recipient needs a copy of the CA which was used to sign the original certificate. It can the decrypt the encrypted checksum, and compare it with the certificates checksum.

If you create a Certificate Authority certificate you will need to distribute this to all machines that might communicate with your server(possibly thousands of machines), and installed into the keystore on those machines. This can be a big task. Most system have one site wide CA certificate which is distributed to all machines. You might have a second CA to limit access to a system. This CA is used to sign the server certificates.

Creating certificates

As part of creating a certificate you create the private/public keys. There are different algorithms for these keys. Some are stronger than others (it takes more time and CPU to break them) Keys with elliptic curve algorithms are generally stronger than using RSA techniques, and there other techniques resistant to Quantum Computing.

You have to use a server certificate with RSA and keysize 2048.

I found that authentication with JWT (Java Web Tokens) only worked with RSA keys and not Elliptic Curves. This is because of encryption with JWT.

Key stores

You should use keyrings rather than .pem files, as they are more secure. .pem files can be copied, and anyone with authority can copy them. You have to give explicit permission to be allowed to access a keyring. Certificate and the private keys within a keyring can be stored in cryptographic hardware, and the private keys are never exposed in clear text.

Many systems uses a keystore for storing the private key used by the server, and a trust store for the Certificate Authority keys needed to validate any client certificate sent to the server. This can, but is not recommended, be the same as the keystore.

The keystore will need the server key. You can specify which key should be used.

  • the Certificate Authority keys needed to validate the server key.
  • the server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

The trust store needs all the CA certificates which may be needed to verify a client certificate.

The client machine will have one or more certificates. A copy of the CA used to create these needs to be installed in the trust store on the server.

I understand that if you add a certificate to the trust store, you need to restart Zowe for it to be picked up, so try to get a list of all the CAs you will need before you start.

Validating certificates

The trust store needs the certificates to validate any client certificates sent to the server. This will usually just be Certificate Authority certificates.

Zowe works with z/OSMF. They communicate with certificates. The z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key. If they are not set up properly you can get messages “LoadBalancer does not contain an instance for the service zaas”.

If you have a site wide CA, which is the same for both of the server keys, then you do not need to do any more work, as both trust store keyrings will already have the CA. If Zowe and z/OSMF have different CA certificates, the CA certificates need to be connected to the other keyrings.

Subject Alternative Name

The Subject Alternative Name within a certificate provides the IP addresses, or IP Names of the server. A client can check this address with the IP address of the session, and terminate the session if they do not match. This is considered best practice. This check can be disabled in Zowe by using verifyCertificates NOSTRICT in the zowe.yaml file.

RACF allows one name or address when the certificate is defined, for example ALTNAME(IP(127.0.0.1))

On my z/OS system the command TSO NETSTAT HOME gives three addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6.

You can configure your sysplex, so all systems in the sysplex have the same IP address, and traffic gets routed internally to the correct system. Without this, if you start a server on a different LPAR it will have a different IP address, and so the validation will fail.

On my system, the zOSMF certificate did not have an ALTNAME specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work, until I fixed the certificate.

If the z/OSMF certificate has an ALTNAME(IP…) specified, use the IP address value when you configure zOSMF for example

zOSMF: 
host: 127.0.0.1
port: 10443
applId: IZUDFLT

Mapping certificates

If you are using client certificate to authenticate rather than a userid and password, then you’ll need to map certificates to userids, for example with the RACDCERT MAP command. You can specify which CA the certificates were signed with, and fields from the subject Distinguished Name. The question of which is better: using userid and password, or client certificate to logon has no easy answer

  • It is easy for a hacker to get a password (lost handbag, yellow sticky stuck to the screen, a couple of pints down the pub) It is easy to change a password once is is compromised.
  • It is harder for a hacker to get a certificate – but it is harder to change and re-issue a certificate. You have to get the updated certificate down to the client’s machine. It could be stolen if a hacker has access to the machine.
  • Using Biometric data to logon is the ultimate limit in this area. Hackers could steal it – but there is no way of changing it if it is compromised!

Decisions

You need to decide

  • Are you going to have a CA just for Zowe? or reuse the site CA.
    • If you are going to have a Zowe specific CA – how are you going to distribute it to all the client machines.
    • You’ll need to ensure Zowe and z/OSMF have the other’s CA certificates in their trust store.
  • Are you going to use Subject Alternative Name ALTNAME(IP(10.1.1.2))
    • What value of verifyCertificates STRICT|NOSTRICT are you going to specify in the zowe.yaml file.
  • Are you going to authenticate using certificates. You will need to set up mapping from certificate to userid. RACDCERT MAP
  • If you will be using JWT, and so need an RSA key in the server certificate

Zowe:Operating Zowe

Starting and stopping Zowe

You can start and stop the main server using z/OS start and stop commands
or you can start it from Unix Services using the zwe start –config … command.

I find the zwe command slow, as it has to spawn a process, to issue the command.

You will need authority to issue the commands whether you use sdsf or the zwe commands.

Using the start and stop commands

S ZWESLSTC

and stop it using

P ZWESLSTC

Using the Unix Services command zwe start –config …

zwe start --config /path/to/my/zowe.yaml

This issues a command on the console like

S ZWESLSTC,HAINST=S0W1,JOBNAME=ZWE1SV           

Where ZWESLSTC is the name of the started task procedure. This runs as job name ZWE1SV, the value in the zowe.job.name zowe.yaml file. The output from the job has name ZWE1SV in the spool.

You can stop it using the P… operator command or use

zwe start --config /path/to/my/zowe.yaml

Displaying active components

You can display the active components using

f ZWE1SLSTC,APPL=DISPLAY

This gives output like

ZWEL0011I     name =          gateway, PID = 33620200         
ZWEL0011I name = zaas, PID = 83952054
ZWEL0011I name = api-catalog, PID = 16843191
ZWEL0011I name = discovery, PID = 65976
ZWEL0011I name = caching-service, PID = 65977
ZWEL0011I name = app-server, PID = -1
ZWEL0011I name = zss, PID = 65979

Where a PID of -1 indicates the service is not running.

Starting and stopping components

You can stop and start components using

F ZWE1SV,APPL=STOP(app-server)
F ZWE1SV,APPL=START(app-server)

The output is like

F ZWE1SLSTC,APPL=STOP(APP-SERVER)                       
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
IEE342I MODIFY REJECTED-TASK BUSY
+ZWEL0002I component app-server stopped
+ZWEL0002I component app-server stopped

Where

  • ZWE1SLSTC is the started task, or jobname.
  • the IEE342I messages are because some BPXAS tasks are started with the same jobname, and they do not respond to the Modify broadcast.

If you configure your Zowe started task procedure to be 7 characters or less, the additional BPXAS jobs created will be called like ZWE1SLST1, ZWE1SLST2, ZWE1SLST3… and you will not get the IEE342I MODIFY REJECTED-TASK BUSY messages

Zowe: Planning

See Preparing for installation in Zowe publications for detailed information on the structure of Zowe files and data sets.

Topic covered


Required products

  • Z/OS 2.5 or later
  • Java 17 IBM® Semeru Runtime Certified Edition for z/OS® version 17
  • Node.js version v18. or later. See here.

Optional products

  • z/OSMF. This is used for Authentication and REST API services.
  • SDSF. This is used to issue console commands, and capture the output

Disk storage

  • For the Zowe product. This typically is in a directory like /usr/lpp/zowe. It needs about 1300 MB, or 1600 CYL.
  • For each instance use Primary 118, Secondary 18 Cylinders.
  • For the .pax file 440 MB
  • Some PDSs are generated. Together they use about 20 Cyl of DASD

CPU

There is a CPU spike when Zowe is started up. It starts several address spaces running Java. Once active the CPU costs depend on how much Zowe is used, plus a steady low overhead.

TCP/IP Interfaces

A TCP/IP configuration can have multiple IP addresses. When Zowe initiates a connection to one of its components, it needs the correct IP address.

This is configured in zowe.externalDomains and can be a domain name or IP address.

Ports

The default ports in the configuration are

  • gateway 7554
  • zaas 7552
  • discovery 7553
  • caching services 7555
  • app server 7556
  • zss 7557
  • infinispan in memory caching, port 7600
  • infinispan in memory caching key exchange, port 7600

These ports can be used by clients outside of z/OS, and between the Zowe started tasks. You need to configure your network and firewalls to allow traffic on these ports.

High level directory structures

The product libraries have a name including the release, such as /usr/lpp/zowe/zowe32. The directory is used within the zowe.yaml file and shell scripts such as the bash .profile. You might want to consider an alias for the product libraries. Such as /global/zoweprod -> /usr/lpp/zowe/zowe32. To upgrade all users, just change the alias, and they will not need to change their scripts.

By default

  • log files are stored in /global/zowe/logs
  • instance specific internal information is stored in /global/zowe/workspace
  • extensions are stored in /global/zowe/extensions.

The /global directory, also known as the global filesystem, serves as a centralized location for storing and sharing files across multiple systems and users. There may be a different file system for /u on each system and so a Zowe instance configured in /u/… may not be visible across systems. If you may run Zowe instance on more than one machine you need to ensure the file system is available on all potential machines.

If you are going to use more than one Zowe instance, you need to plan for this. For example use the Zowe instance name as part of the directory. Perhaps use /global/zowe/logs/ZOWE1, or /u/zowec/ZOWE1/logs.

Having the system name SYS1 as part of the file path can be confusing if you will start the Zowe instance on more than one LPAR.

Where to store extensions

You may want to share the same instance of /global/zowe/extensions across all Zowe instances, or have each Zowe instances to use their own directory.

Automation and monitoring

Zowe can be configured so that important messages like Zowe has started and Zowe is stopping are written to the system log. You may need to change your automation to react to these messages. For example

+ZWEL0018I Zowe instance prepared successfully                        
+ZWEL0006I starting components
+ZWEL0001I component gateway started

+ZWEL0008I stopping components
+ZWEL0002I component gateway stopped
+ZWEL0022I Zowe Launcher stopped

See Which messages to automate?

Understanding the configuration process

z/OS system configuration

As part of the installation, one of the first steps is to run a script which extracts PDS from the Zowe files.

There are two load libraries which will need to be APF authorised.

Two started task procedures will be created. One started task is a server which can run authorised code. The other runs a Java program.
When Zowe starts up it spawns threads that run as BPXAS address spaces. The names of the spawned threads are like ZWE1AD. The prefix ZWE1 can be changed in the configuration file.

The cross memory server started task needs parameters. These can be put in the SYS1.PARMLIB concatenation, or in a Zowe PDS.
An update is needed to parmlib for PPT entries to set the storage key for Zowe.

The Zowe default dataset names have a prefix IBMUSER.ZOWEV3. You can use your own names, such as PP.ZOWEV3 (for Program Products).

Workload manager definitions

You should set up WorkLoad Manager definitions for the address spaces.
Note: If a WLM definition has discretionary, any Java work will not run on GCPs if the ZIIPS are busy, so you need care in specifying which rule applies to the address spaces.

Security definitions

You should protect the data sets produced by Zowe configuration; the sample library is not sensitive, but the load libraries are APF authorised, and these APF authorised libraries should be protected from being changed by an unauthorised user.

What global FACILITY profiles are used

As part of the configuration, security definitions are created.

  • Some changes are global, enabling some system wide facilities. Most systems will typically have these already enabled, but if not, you may want to configure them early to give plenty of time to test them.
    • Activate classes STARTED and FACILITY
    • Define FACILITY BPX.DAEMON
    • Define FACILITY BPX.SERVER
    • Define FACILITY BPX.JOBNAME
    • Define FACILITY IRR.RUSERMAP
    • Define FACILITY IRR.IDIDMAP
    • Define FACILITY IRR.RAUDITX
  • Some changes are Zowe related, such as userid for Zowe. This defaults to ZWEUSR
    and group ZWEGRP.
    • Add group ZWEADMIN.
    • Add userid definitions for ZWESVUSR and ZWESIUSR.
    • Defines started task profiles for STARTED; ZWESLSTC, ZWESISTC, and ZWESASTC* (where ZWE is configurable).
  • Other changes
    • A data set profile IBMUSER.ZWEV3..* for the data sets that Zowe install creates.

You should consider how user’s will authenticate.
As part of the TLS session initialisation the server can request the client’s certificate and use information from the certificate to determine the userid to be used.
You can authenticate using:

  • Userid and password
  • using z/OSMF
  • Multi Factor authentication – such as having a widget or mobile phone
    application which generates a code as a one off, time limited code as a password.

You can use AT-TLS.

How do I protect what Zowe users can do on z/OS?

Zowe is a transport mechanism like a 3270 emulator. You control what a userid can do, not what the transport mechanism can do. See here

TLS levels

The levels from the example .yaml file is

tls: 
attls: false
# TLS settings only apply when attls=false
# Else you must use AT-TLS configuration for TLS customization.
minTls: TLSv1.2
maxTls: TLSv1.3

If you are not at TLS v1.3 you may want to start at TLS 1.2, and convert to TLS 1.3 at a later date.

Certificates

See a whole blog post on planning for certificates.

PPT entries

Two modules have to be defined in a SCHED member with properties KEY(4) NOSWAP.

Operations

You need to start the two started tasks. The authorised server is active and does not do much. You can start it at IPL and leave it running.

You start the Zowe main task when you want to use Zowe. This spawns several threads in BPXAS address spaces. These are called ZWE1xx – where the ZWE1 can be customised.
You use the stop z/OS command (P …) to terminate the main task.
When it shuts down, it terminates the services running in BPXAS address spaces.

Data management

Zowe does not hold any state information. The configuration files in the Zowe instance home directory should be backed up regularly, as well as the proclib and parmlib members.

Running in a sysplex

The Zowe product libraries can be shared across all LPARs in a Sysplex. You may want to consider mounting it read only.
A Zowe instance can be started on any LPAR, provided the instance home directory, and other directories it uses, are available and writable.

You can create more Zowe instances on one LPAR ( why would you?) or for availability you can create a Zowe instance on more than one LPAR.

High availability

If you have a Zowe instance running on more than one LPAR, if one instance is shutdown, you can (re)connect to a different solution, and logon.
You can also configure a Zowe HA environment, where you can reconnect to an alternate instance, without needing to logon.

Migrating to a new level of Zowe

Install the new level of Zowe into a different directory to your current system.
The process will update the Zowe PDS datasets:

  • …SZWEAUTH,
  • …SZWEEXEC,
  • …SZWELOAD,
  • …SZWESAMP

so you should back these up, or change the config file to use a different data set prefix.
To overwrite these datasets you need to explicitly set an option.

Backup the configuration files, so if you need to go back to the current level of code, you just rename the files and data sets and restart Zowe.

Before you start – what configuration path to take?

I was talking to someone about installing software on z/OS and how it could be made easier, and they said the biggest challenge is getting changes past the grey haired person in the corner office whose job is to say No. As in

  1. No: You cannot “move quickly and break things”. You must move carefully and keep safe.
  2. No: You cannot install untested stuff on production machines, without installing it first on test or development machines
  3. No: You cannot install products where you have to type commands or information onto the production machines. You have to provide files of information, or a document where commands or information is reviewed and can be copied using cut-and-paste. For example people have had problems with the letter o (for Orange) and zero, and i (for India) 1’s and l (for Lima).
  4. No: You cannot just run a script or command which changes z/OS configuration. All changes have to be reviewed. For example some changes to the security configuration affect all address spaces.
  5. No: You cannot run something that is “one way” for example adding a column to a data base table, to sort data, or delete something without a tested process for restoring and recovering the data.
  6. No: You cannot have the authority to issue commands, because you may issue commands outside of the documented install script, without going through change control.
  7. No: You cannot install something until we know the impact of running it. For example, the amount of CPU, real, virtual, disk storage.

Zowe

Zowe is a project which provides a command level interface from outside of z/OS to z/OS services. This means people do not need to know about ISPF, or the operator console to be able to use z/OS. Zowe has several Java address spaces which provide the facilities.

As part of the aim to make z/OS more accessible, the Zowe installation and configuration has command scripts which can be used to define the resources needed for Zowe to run.

I feel the “recommended” processes for configuring Zowe is fit for a test or development system, not for an important system, and so I recommend using a more traditional approach, which fits in the “No:… above”

Not all the configuration changes needed are listed in the documentation, for example you should configure WLM to give the address spaces appropriate priority. You need to backup data sets and files.

The security definitions enable a global facility which can impact many subsystems. This change alone would need careful roll out to all systems.

APF definitions are made using dynamic APF command interface on the console. They are not saved across IPL, so the changes have to be made to the SYS1.PARMLIB concatenation at some point. My view is just do the parmlib definitions once.

Zowe: Create the z/OS security definitions

The Zowe documentation for security definitions is here. I consider the documentation is dangerous, as it defines some global profiles which can affect other address spaces, and change their behaviour.

Before you start

You need to know

  • Which userid you want the main Zowe started tasks to run under.
  • Which user you want the cross memory server to run under.
  • The name of an admin group these userids will be a member of. You can connect your existing userids to this group to make them administrators.
  • The name of the started task for Zowe, and for the cross memory server.
  • The name of the Zowe APF authorised data set.

System wide definitions

If you use RACF you need the following RACF profiles defined – they may already be defined.

SETROPTS LIST                                        
RLIST FACILITY BPX.SERVER ALL
RLIST FACILITY BPX.DAEMON ALL
RLIST FACILITY BPX.JOBNAME ALL
RLIST FACILITY IRR.RUSERMAP ALL
RLIST FACILITY IRR.RAUDITX ALL
RLIST FACILITY IRR.IDIDMAP.QUERY ALL
/* this is optional
RLIST APPL OMVSAPPL ALL

If you are missing the BPX.SERVER or the BPX.DAEMON profiles, then you need to carefully test your existing applications, because their behaviour may change when these profiles are defined.

 /* permit Zowe main server to create a user's security environment   */      
/* ATTENTION: Defining the BPX.DAEMON or BPX.SERVER profile makes */
/* z/OS UNIX switch to z/OS UNIX level security. This is */
/* more secure, but it can impact operation of existing */
/* applications. Test this thoroughly before activating */
/* it on a production system. */
RDEFINE FACILITY BPX.DAEMON UACC(NONE)

RDEFINE FACILITY BPX.SERVER UACC(NONE)

/* permit Zowe main server to set job name */
RDEFINE FACILITY BPX.JOBNAME UACC(NONE)

/* permit Zowe main server to use client certificate mapping service */
RDEFINE FACILITY IRR.RUSERMAP UACC(NONE)

/* permit Zowe main server to use distributed identity mapping */
/* service
RDEFINE FACILITY IRR.IDIDMAP.QUERY UACC(NONE)

/* permit Zowe main server to cut SMF records */
RDEFINE FACILITY IRR.RAUDITX UACC(NONE)

SETROPTS RACLIST(APPL) REFRESH
SETROPTS RACLIST(FACILITY) REFRESH

Userid and group definitions

You need to have a userid to run the Zowe started tasks. Zowe also defines a different userid for the ZIS cross memory server. It is not clear to me why this is needed.

The userids need an admin group.

The definitions provided by Zowe for userids ZWESVUSR and ZWESIUSR; and the admin group are

/* - The sample commands assume automatic generation of GID is       */    
/* enabled. */

/* group for administrators */
/* replace AUTOGID with GID(&ADMINGID.) if AUTOGID not enabled */
LISTGRP ZWEADMIN OMVS
ADDGROUP ZWEADMIN OMVS(AUTOGID) -
DATA('ZOWE ADMINISTRATORS')

/* uncomment to add existing user IDs to the ADMINGRP group */
/* CONNECT (userid,userid,...) - */
/* GROUP(ZWEADMIN) AUTH(USE) */

/* userid for ZOWE main server */
/* replace AUTOUID with UID(&ZOWEUID.) if AUTOUID not enabled */
LISTUSER ZWESVUSR OMVS
ADDUSER ZWESVUSR -
NOPASSWORD -
DFLTGRP(ZWEADMIN) -
OMVS(HOME(/tmp) PROGRAM(/bin/sh) AUTOUID) -
NAME('ZOWE SERVER') -
DATA('ZOWE MAIN SERVER')
/* userid for ZIS cross memory server */
/* replace AUTOUID with UID(&ZISUID.) if AUTOUID not enabled */
LISTUSER ZWESIUSR OMVS
ADDUSER ZWESIUSR -
NOPASSWORD -
DFLTGRP(ZWEADMIN) -
OMVS(HOME(/tmp) PROGRAM(/bin/sh) AUTOUID) -
NAME('ZOWE ZIS SERVER') -
DATA('ZOWE ZIS CROSS MEMORY SERVER')

/* show results .................................................... */
LISTGRP ZWEADMIN OMVS
LISTUSER ZWESVUSR OMVS
LISTUSER ZWESIUSR OMVS

Create a profile to protect the APF data set

  LISTDSD PREFIX(IBMUSER.ZWEV3A) ALL                                      
ADDSD 'IBMUSER.ZWEV3A.*.**' UACC(READ) -
DATA('Zowe')
PERMIT 'IBMUSER.ZWEV3A.*.**' CLASS(DATASET) -
ACCESS(ALTER) ID(ZWEADMIN)

SETROPTS GENERIC(DATASET) REFRESH

/* show results .................................................... */
LISTGRP IBMUSER.ZWEV3A
LISTDSD PREFIX(IBMUSER.ZWEV3A) ALL

Create the definitions for the started tasks

/* started task for ZOWE main server                               
RLIST STARTED ZWESLSTC* ALL STDATA
/*RDELETE STARTED ZWESLSTC*
RDEFINE STARTED ZWESLSTC* -
STDATA(USER(ZWESVUSR) -
GROUP(ZWEADMIN) -
TRUSTED(NO)) DATA('ZOWE MAIN SERVER')

/* started task for ZIS cross memory server
RLIST STARTED ZWESISTC* ALL STDATA
/*RDELETE STARTED ZWESISTC*
RDEFINE STARTED ZWESISTC* -
STDATA(USER(ZWESIUSR) -
GROUP(ZWEADMIN) -
TRUSTED(NO)) DATA('ZOWE ZIS CROSS MEMORY SERVER')

/* started task for ZIS Auxiliary cross memory server
RLIST STARTED ZWESASTC* ALL STDATA
/*RDELETE STARTED ZWESASTC*
RDEFINE STARTED ZWESASTC* -
STDATA(USER(ZWESIUSR) -
GROUP(ZWEADMIN) -
TRUSTED(NO)) DATA('ZOWE ZIS AUX CROSS MEMORY SERVER')

SETROPTS RACLIST(STARTED) REFRESH

RLIST STARTED ZWESLSTC* ALL STDATA
RLIST STARTED ZWESISTC* ALL STDATA
RLIST STARTED ZWESASTC* ALL STDATA

Define a profile and permissions to the cross memory server

/* DEFINE ZIS SECURITY RESOURCES ................................... */      

/* define ZIS security profile */
RLIST FACILITY ZWES.IS ALL
RDEFINE FACILITY ZWES.IS UACC(NONE)

/* DEFINE AUX SERVER PERMISIONS .................................... */

/* permit AUX STC to use ZIS cross memory server */
PERMIT ZWES.IS CLASS(FACILITY) ACCESS(READ) -
ID(ZWESIUSR)
SETROPTS RACLIST(FACILITY) REFRESH

/* DEFINE ZOWE SERVER PERMISIONS ................................... */

/* permit Zowe main server to use ZIS cross memory server */
PERMIT ZWES.IS CLASS(FACILITY) ACCESS(READ) -
ID(ZWESVUSR)
SETROPTS RACLIST(FACILITY) REFRESH

Define other profiles

/* permit Zowe main server to create a user's security environment   */      
/* comment out the following 2 lines if the OMVSAPPL is not defined */
/* in your environment */
PERMIT OMVSAPPL CLASS(APPL) ID(ZWESVUSR) -
ACCESS(READ)
SETROPTS RACLIST(APPL) REFRESH

PERMIT IRR.RUSERMAP CLASS(FACILITY) ACCESS(READ) -
ID(ZWESVUSR)

/* permit Zowe main server to use distributed identity mapping */
/* service
PERMIT IRR.IDIDMAP.QUERY CLASS(FACILITY) ACCESS(READ) -
ID(ZWESVUSR)

/* permit Zowe main server to cut SMF records */
PERMIT IRR.RAUDITX CLASS(FACILITY) ACCESS(READ) -
ID(ZWESVUSR)

Define another profile

/* DEFINE ZOWE RESOURCE PROTECTION ................................. */      

/* - Defines resource APIML.SERVICES that controls access to */
/* detailed information about API services to Zowe users. */

/* define resource for information about API services */
RDEFINE ZOWE APIML.SERVICES UACC(NONE)

/* uncomment and replace "user" to permit Zowe users to access */
/* the resource: */
/* PERMIT APIML.SERVICES CLASS(ZOWE) ID(user) ACCESS(READ) */

/* show results */
RLIST ZOWE *

z/OSMF changes

Same origin policy. Wikipedia says the same-origin policy (SOP) is a concept in the web-app application security model. Under the policy, a web browser permits scripts contained in a first web page to access data in a second web page, but only if both web pages have the same origin.

Having a REST request go from Zowe to z/OSMF counts as a cross-origin resource sharing (CORS)

The z/OSMF documentation says

Enabling cross-origin resource sharing (CORS) for REST services

Your installation can allow browser applications from certain, trusted sites to access z/OSMF REST services on the host system. If so, you must enable cross-origin resource sharing (CORS) on the host system. This work involves creating an “allow list” of exceptions (the trusted sites), and enabling those exceptions in your external security manager.

Identify which sites are to be allowed, and which REST interfaces are to be made available for cross-site access. Then, work with your security administrator to create the appropriate authorizations in your external security manager. In a RACF installation, for example, define generic or discrete profiles for the remote sites in the ZMFAPLA class, and permit the profiles to the z/OSMF REST interfaces.

For example

RDEFINE ZMFAPLA IZUDFLT.REST.JOBS.COM.IBM.LAB2 UACC(NONE)

Zowe Explorer access

A userid using the Zowe explorer access into z/OSMF needs

  • Access to the ISPF procedure
  • Access to the TSO Account number
  • Read access to CSFRNGL Class(CSFSERV )