Month: May 2025

How do you get into the well protected castle? – you tunnel

Before I retired, if I wanted access to the corporate systems, I would start up a tool (vpn/dialer) which set up an encrypted session to the corporate front end, and tunnel through this to get to the back end. It was transparent and was like being directly attached.

How do you set up your castle to allow authorised people in – but deny access to unauthorised access?

If you logon directly to a back-end server from a cafe or other public WiFi, your credentials etc may be in the clear. You need to use “a secure app” such as banking app, or provide a vpn or tunnel to access your system to encrypt the traffic.

I’ve been logging on to someone else’s system and most of the useful connections to this are disabled. I could access via 3270 but could not use FTP etc.

I was trying to debug a TLS problem and found suspicious evidence in the trace. A connection was being made to the TLS port on my server, and failing because it did not speak TLS. Someone had clearly found the IP address of my system, and was trying all ports to get in!

How do you protect the system and keep unwanted people out – but allow authorised people to connect. You need to do both of…

Secure tunnelling

You could use TLS to protect the conversation between client and server, but this means you have allowed the connection to get through TCP/IP and to your application before checking to see if the connection is permitted.

Port forwarding or tunnelling controls access to your system at the outer edge of your system, logically within TCP/IP, before it gets to your application. It is well described here.

I used the SSH command

ssh -N -L 9876:12.23.34.45:8765 colin@23.34.45.56 -p 2222

If I used address https://localhost:9876 in my web browser, I am connected to 12.23.34.45:8765 through system 23.34.45.56 .

Where

  • -N says port forwarding
  • my local port 9876 is linked to 12.23.34.45 port 8765
  • via IP address 23.34.45.56. Think of this as the guard house.
  • colin is the userid. You get prompted for your password.
  • use port 2222 at the guard house. Think of multiple entrances to the guard house. “General public” “special guest” “people who work here”. The port says go to the “special guest” door. The first time, you have to successfully logon with your userid and password to get your details recorded as a valid visitor to the site. You are given a token which is saved for next time you want access (think of it as a visitor badge).

What is IPSEC?

IPSEC is part of Communications Server on z/OS. It provides

  • IP filtering to control which packets can enter the system
  • IP filtering to control which packets can leave the system

You can filter on

  • packet information – allow a ping, but not FTP
  • network attributes – only allow from a list of IP address
  • time – at certain times of day. For example a normal working day – so if someone is trying to access this system at two o’clock in the morning, this would be worth investigating.

You can

  • allow
  • deny
  • log

You need to be careful setting the rules up – if you want to allow traffic in from 1.2.3.4, then you need to allow traffic out to address 1.2.3.4

My rule

I would set up an input rule like

For input port 8765 deny all access, all packet types, all hours of the day.

Using the tunnel needs a little care

The browser validates the certificate sent from the server. If there ALTNAME, the client(browser) should check that the IP address specified in the ALTNAME matches the IP address used in the original URL, and your connection has not been high jacked.

In my browser I used https://127.0.0.1:9876, where 9876 was specified in the ssl command above. The certificate at the server had ALTNAME with IP address of 127.0.0.1, and so this was considered valid. The IP ALTNAME IP adddress is usually the IP address of the server (or one of them if there are more than one) so you may get messages saying an insecure certificate is being used. You can accept this, or configure your browser to ignore these checks (which is not a good idea)

Thanks

Thanks to Lionel Dyck and Randy Rackov for their help in this blog post.

Zowe: Setting up the Zowe side for a web browser

If you want to use your web browser to connect to a Zowe server, it is just like connecting to any other back end server.

  • The browser need the CA certificate used to sign the Zowe server’s certificate.
  • The server needs the CA used to sign the browser’s certificate, in the server’s trust store.

The browser need the CA certificate used to sign the Zowe server certificate.

If you do not have a copy of the exported certificate you can export it

//COLINEXP JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//STEPLIB  DD  DISP=SHR,DSN=SYS1.MIGLIB 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CERTAUTH   EXPORT(LABEL('DOCZOSCA'))   - 
       DSN('COLIN.CERT.DOC.CA.PEM') 
//

This will delete the dataset, recreate it and export the certificate.

This exported certificate will look like

-----BEGIN CERTIFICATE-----                                          
MIIBtzCCAV6gAwIBAgIUb0N8Xwb/WNulhgjzRcIwVBmfAXwwCgYIKoZIzj0EAwMw     
...
S9QFQcy5vKGFFAQ=                                                     
-----END CERTIFICATE-----

Copy this file to where ever it is needed.

On Linux you can use the command

openssl x509 -in cbt.ca.pem -text -noout|less

to display the contents

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = COLIN, OU = CA, CN = DOCZOSCA
        Validity
            Not Before: May  2 05:00:00 2025 GMT
            Not After : Jul  3 04:59:59 2029 GMT
        Subject: O = COLIN, OU = CA, CN = DOCZOSCA
...

Note: The base64 command does not decode the base64 content properly.

Import this file into your browser

For example with Chrome. Settings, search for certificates, click on security, click on Manage Certificates. Click on Customised, Trusted certificates, Import. Select the file name. You may have to select All Files at the bottom to get your file included in the list.

The file should now be installed, and be listed under Trusted certificates. Click on the pen icon to display it.

The server needs the CA used to sign the browser’s certificate it the server’s trust store.

Create a VB data set on the Zowe z/OS machine. Copy the contents of the client’s CA into this data set. I used cut and paste.

//COLINADD JOB 1,MSGCLASS=H 
//* ADD ORKENY CA CERTIFICATE 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.CERT.DOC.CA.ORK') 
RACDCERT DELETE - 
  (LABEL('ORKCA256')) CERTAUTH 
RACDCERT CERTAUTH     ADD('COLIN.CERT.DOC.CA.ORK')- 
  WITHLABEL('ORKCA256')  TRUST 
RACDCERT CERTAUTH LISTCHAIN(LABEL('ORKCA256')) 
                                                                  
RACDCERT ID(COLIN) CONNECT(RING(ZOWE) - 
                            CERTAUTH   - 
                            LABEL('ORKCA256')) 
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

When this is successful, you will need to shutdown and restart Zowe.

Zowe: Planning – certificates

A private key is used for encryption and should be kept private. If someone has access to the private key they can impersonate you.

A public key is the opposite of the private key. It is used for decrypting data encrypted with the private key. The public key can (and should) be generally available.

Certificates are used in TLS for authentication and encryption, and can be used for identification. They include a public key.

A Certificate Authority(CA) certificate is used to validate other certificates. It involves doing a checksum of a certificate, and encrypting it with the CA private key – a process known as signing the certificate

To validate a certificate, the recipient needs a copy of the CA which was used to sign the original certificate. It can the decrypt the encrypted checksum, and compare it with the certificates checksum.

If you create a Certificate Authority certificate you will need to distribute this to all machines that might communicate with your server(possibly thousands of machines), and installed into the keystore on those machines. This can be a big task. Most system have one site wide CA certificate which is distributed to all machines. You might have a second CA to limit access to a system. This CA is used to sign the server certificates.

Creating certificates

As part of creating a certificate you create the private/public keys. There are different algorithms for these keys. Some are stronger than others (it takes more time and CPU to break them) Keys with elliptic curve algorithms are generally stronger than using RSA techniques, and there other techniques resistant to Quantum Computing.

You have to use a server certificate with RSA and keysize 2048.

I found that authentication with JWT (Java Web Tokens) only worked with RSA keys and not Elliptic Curves. This is because of encryption with JWT.

Key stores

You should use keyrings rather than .pem files, as they are more secure. .pem files can be copied, and anyone with authority can copy them. You have to give explicit permission to be allowed to access a keyring. Certificate and the private keys within a keyring can be stored in cryptographic hardware, and the private keys are never exposed in clear text.

Many systems uses a keystore for storing the private key used by the server, and a trust store for the Certificate Authority keys needed to validate any client certificate sent to the server. This can, but is not recommended, be the same as the keystore.

The keystore will need the server key. You can specify which key should be used.

  • the Certificate Authority keys needed to validate the server key.
  • the server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

The trust store needs all the CA certificates which may be needed to verify a client certificate.

The client machine will have one or more certificates. A copy of the CA used to create these needs to be installed in the trust store on the server.

I understand that if you add a certificate to the trust store, you need to restart Zowe for it to be picked up, so try to get a list of all the CAs you will need before you start.

Validating certificates

The trust store needs the certificates to validate any client certificates sent to the server. This will usually just be Certificate Authority certificates.

Zowe works with z/OSMF. They communicate with certificates. The z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key. If they are not set up properly you can get messages “LoadBalancer does not contain an instance for the service zaas”.

If you have a site wide CA, which is the same for both of the server keys, then you do not need to do any more work, as both trust store keyrings will already have the CA. If Zowe and z/OSMF have different CA certificates, the CA certificates need to be connected to the other keyrings.

Subject Alternative Name

The Subject Alternative Name within a certificate provides the IP addresses, or IP Names of the server. A client can check this address with the IP address of the session, and terminate the session if they do not match. This is considered best practice. This check can be disabled in Zowe by using verifyCertificates NOSTRICT in the zowe.yaml file.

RACF allows one name or address when the certificate is defined, for example ALTNAME(IP(127.0.0.1))

On my z/OS system the command TSO NETSTAT HOME gives three addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6.

You can configure your sysplex, so all systems in the sysplex have the same IP address, and traffic gets routed internally to the correct system. Without this, if you start a server on a different LPAR it will have a different IP address, and so the validation will fail.

On my system, the zOSMF certificate did not have an ALTNAME specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work, until I fixed the certificate.

If the z/OSMF certificate has an ALTNAME(IP…) specified, use the IP address value when you configure zOSMF for example

zOSMF: 
  host: 127.0.0.1 
  port: 10443 
  applId: IZUDFLT

Mapping certificates

If you are using client certificate to authenticate rather than a userid and password, then you’ll need to map certificates to userids, for example with the RACDCERT MAP command. You can specify which CA the certificates were signed with, and fields from the subject Distinguished Name. The question of which is better: using userid and password, or client certificate to logon has no easy answer

  • It is easy for a hacker to get a password (lost handbag, yellow sticky stuck to the screen, a couple of pints down the pub) It is easy to change a password once is is compromised.
  • It is harder for a hacker to get a certificate – but it is harder to change and re-issue a certificate. You have to get the updated certificate down to the client’s machine. It could be stolen if a hacker has access to the machine.
  • Using Biometric data to logon is the ultimate limit in this area. Hackers could steal it – but there is no way of changing it if it is compromised!

Decisions

You need to decide

  • Are you going to have a CA just for Zowe? or reuse the site CA.
    • If you are going to have a Zowe specific CA – how are you going to distribute it to all the client machines.
    • You’ll need to ensure Zowe and z/OSMF have the other’s CA certificates in their trust store.
  • Are you going to use Subject Alternative Name ALTNAME(IP(10.1.1.2))
    • What value of verifyCertificates STRICT|NOSTRICT are you going to specify in the zowe.yaml file.
  • Are you going to authenticate using certificates. You will need to set up mapping from certificate to userid. RACDCERT MAP
  • If you will be using JWT, and so need an RSA key in the server certificate

Zowe:Operating Zowe

Starting and stopping Zowe

You can start and stop the main server using z/OS start and stop commands…
or you can start it from Unix Services using the zwe start –config … command.

I find the zwe command slow, as it has to spawn a process, to issue the command.

You will need authority to issue the commands whether you use sdsf or the zwe commands.

Using the start and stop commands

S ZWESLSTC

and stop it using

P ZWESLSTC

Using the Unix Services command zwe start –config … 

zwe start --config /path/to/my/zowe.yaml

This issues a command on the console like

S ZWESLSTC,HAINST=S0W1,JOBNAME=ZWE1SV

Where ZWESLSTC is the name of the started task procedure. This runs as job name ZWE1SV, the value in the zowe.job.name zowe.yaml file. The output from the job has name ZWE1SV in the spool.

You can stop it using the P… operator command or use

zwe start --config /path/to/my/zowe.yaml

Displaying active components

You can display the active components using

f ZWE1SLSTC,APPL=DISPLAY

This gives output like

ZWEL0011I     name =          gateway, PID = 33620200         
ZWEL0011I     name =             zaas, PID = 83952054         
ZWEL0011I     name =      api-catalog, PID = 16843191         
ZWEL0011I     name =        discovery, PID = 65976            
ZWEL0011I     name =  caching-service, PID = 65977            
ZWEL0011I     name =       app-server, PID = -1               
ZWEL0011I     name =              zss, PID = 65979

Where a PID of -1 indicates the service is not running.

Starting and stopping components

You can stop and start components using 

F ZWE1SV,APPL=STOP(app-server)
F ZWE1SV,APPL=START(app-server)

The output is like

F ZWE1SLSTC,APPL=STOP(APP-SERVER)                       
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
IEE342I MODIFY   REJECTED-TASK BUSY                    
+ZWEL0002I component app-server stopped                
+ZWEL0002I component app-server stopped

Where

  • ZWE1SLSTC is the started task, or jobname.
  • the IEE342I messages are because some BPXAS tasks are started with the same jobname, and they do not respond to the Modify broadcast.

If you configure your Zowe started task procedure to be 7 characters or less, the additional BPXAS jobs created will be called like ZWE1SLST1, ZWE1SLST2, ZWE1SLST3… and you will not get the IEE342I MODIFY REJECTED-TASK BUSY messages

Zowe: Planning

See Preparing for installation in Zowe publications for detailed information on the structure of Zowe files and data sets.

Topic covered

Required products

  • Z/OS 2.5 or later
  • Java 17 IBM® Semeru Runtime Certified Edition for z/OS® version 17
  • Node.js version v18. or later. See here.

Optional products

  • z/OSMF. This is used for Authentication and REST API services.
  • SDSF. This is used to issue console commands, and capture the output

Disk storage

  • For the Zowe product. This typically is in a directory like /usr/lpp/zowe. It needs about 1300 MB, or 1600 CYL.
  • For each instance use Primary 118, Secondary 18 Cylinders.
  • For the .pax file 440 MB
  • Some PDSs are generated. Together they use about 20 Cyl of DASD

CPU

There is a CPU spike when Zowe is started up. It starts several address spaces running Java. Once active the CPU costs depend on how much Zowe is used, plus a steady low overhead.

TCP/IP Interfaces

A TCP/IP configuration can have multiple IP addresses. When Zowe initiates a connection to one of its components, it needs the correct IP address.

This is configured in zowe.externalDomains and can be a domain name or IP address.

Ports

The default ports in the configuration are

  • gateway 7554
  • zaas 7552
  • discovery 7553
  • caching services 7555
  • app server 7556
  • zss 7557
  • infinispan in memory caching, port 7600
  • infinispan in memory caching key exchange, port 7600

These ports can be used by clients outside of z/OS, and between the Zowe started tasks. You need to configure your network and firewalls to allow traffic on these ports.

High level directory structures

The product libraries have a name including the release, such as /usr/lpp/zowe/zowe32. The directory is used within the zowe.yaml file and shell scripts such as the bash .profile. You might want to consider an alias for the product libraries. Such as /global/zoweprod -> /usr/lpp/zowe/zowe32. To upgrade all users, just change the alias, and they will not need to change their scripts.

By default

  • log files are stored in /global/zowe/logs
  • instance specific internal information is stored in /global/zowe/workspace
  • extensions are stored in /global/zowe/extensions.

The /global directory, also known as the global filesystem, serves as a centralized location for storing and sharing files across multiple systems and users. There may be a different file system for /u on each system and so a Zowe instance configured in /u/… may not be visible across systems. If you may run Zowe instance on more than one machine you need to ensure the file system is available on all potential machines.

If you are going to use more than one Zowe instance, you need to plan for this. For example use the Zowe instance name as part of the directory. Perhaps use /global/zowe/logs/ZOWE1, or /u/zowec/ZOWE1/logs.

Having the system name SYS1 as part of the file path can be confusing if you will start the Zowe instance on more than one LPAR.

Where to store extensions

You may want to share the same instance of /global/zowe/extensions across all Zowe instances, or have each Zowe instances to use their own directory.

Automation and monitoring

Zowe can be configured so that important messages like Zowe has started and Zowe is stopping are written to the system log. You may need to change your automation to react to these messages. For example

+ZWEL0018I Zowe instance prepared successfully                        
+ZWEL0006I starting components                                        
+ZWEL0001I component gateway started                                  
                                            
+ZWEL0008I stopping components                                        
+ZWEL0002I component gateway stopped                                  
+ZWEL0022I Zowe Launcher stopped

See Which messages to automate?

Understanding the configuration process

z/OS system configuration

As part of the installation, one of the first steps is to run a script which extracts PDS from the Zowe files.

There are two load libraries which will need to be APF authorised.

Two started task procedures will be created. One started task is a server which can run authorised code. The other runs a Java program.
When Zowe starts up it spawns threads that run as BPXAS address spaces. The names of the spawned threads are like ZWE1AD. The prefix ZWE1 can be changed in the configuration file.

The cross memory server started task needs parameters. These can be put in the SYS1.PARMLIB concatenation, or in a Zowe PDS.
An update is needed to parmlib for PPT entries to set the storage key for Zowe.

The Zowe default dataset names have a prefix IBMUSER.ZOWEV3. You can use your own names, such as PP.ZOWEV3 (for Program Products).

Workload manager definitions

You should set up WorkLoad Manager definitions for the address spaces.
Note: If a WLM definition has discretionary, any Java work will not run on GCPs if the ZIIPS are busy, so you need care in specifying which rule applies to the address spaces.

Security definitions

You should protect the data sets produced by Zowe configuration; the sample library is not sensitive, but the load libraries are APF authorised, and these APF authorised libraries should be protected from being changed by an unauthorised user.

What global FACILITY profiles are used

As part of the configuration, security definitions are created.

  • Some changes are global, enabling some system wide facilities. Most systems will typically have these already enabled, but if not, you may want to configure them early to give plenty of time to test them.
    • Activate classes STARTED and FACILITY
    • Define FACILITY BPX.DAEMON
    • Define FACILITY BPX.SERVER
    • Define FACILITY BPX.JOBNAME
    • Define FACILITY IRR.RUSERMAP
    • Define FACILITY IRR.IDIDMAP
    • Define FACILITY IRR.RAUDITX
  • Some changes are Zowe related, such as userid for Zowe. This defaults to ZWEUSR
    and group ZWEGRP.
    • Add group ZWEADMIN.
    • Add userid definitions for ZWESVUSR and ZWESIUSR.
    • Defines started task profiles for STARTED; ZWESLSTC, ZWESISTC, and ZWESASTC* (where ZWE is configurable).
  • Other changes
    • A data set profile IBMUSER.ZWEV3..* for the data sets that Zowe install creates.

You should consider how user’s will authenticate.
As part of the TLS session initialisation the server can request the client’s certificate and use information from the certificate to determine the userid to be used.
You can authenticate using:

  • Userid and password
  • using z/OSMF
  • Multi Factor authentication – such as having a widget or mobile phone
    application which generates a code as a one off, time limited code as a password.

You can use AT-TLS.

How do I protect what Zowe users can do on z/OS?

Zowe is a transport mechanism like a 3270 emulator. You control what a userid can do, not what the transport mechanism can do. See here

TLS levels

The levels from the example .yaml file is

tls: 
  attls: false 
  # TLS settings only apply when attls=false 
  # Else you must use AT-TLS configuration for TLS customization. 
  minTls:  TLSv1.2 
  maxTls:  TLSv1.3

If you are not at TLS v1.3 you may want to start at TLS 1.2, and convert to TLS 1.3 at a later date.

Certificates

See a whole blog post on planning for certificates.

PPT entries

Two modules have to be defined in a SCHED member with properties KEY(4) NOSWAP.

Operations

You need to start the two started tasks. The authorised server is active and does not do much. You can start it at IPL and leave it running.

You start the Zowe main task when you want to use Zowe. This spawns several threads in BPXAS address spaces. These are called ZWE1xx – where the ZWE1 can be customised.
You use the stop z/OS command (P …) to terminate the main task.
When it shuts down, it terminates the services running in BPXAS address spaces.

Data management

Zowe does not hold any state information. The configuration files in the Zowe instance home directory should be backed up regularly, as well as the proclib and parmlib members.

Running in a sysplex

The Zowe product libraries can be shared across all LPARs in a Sysplex. You may want to consider mounting it read only.
A Zowe instance can be started on any LPAR, provided the instance home directory, and other directories it uses, are available and writable.

You can create more Zowe instances on one LPAR ( why would you?) or for availability you can create a Zowe instance on more than one LPAR.

High availability

If you have a Zowe instance running on more than one LPAR, if one instance is shutdown, you can (re)connect to a different solution, and logon.
You can also configure a Zowe HA environment, where you can reconnect to an alternate instance, without needing to logon.

Migrating to a new level of Zowe

Install the new level of Zowe into a different directory to your current system.
The process will update the Zowe PDS datasets:

  • …SZWEAUTH,
  • …SZWEEXEC,
  • …SZWELOAD,
  • …SZWESAMP

so you should back these up, or change the config file to use a different data set prefix.
To overwrite these datasets you need to explicitly set an option.

Backup the configuration files, so if you need to go back to the current level of code, you just rename the files and data sets and restart Zowe.

Before you start – what configuration path to take?

I was talking to someone about installing software on z/OS and how it could be made easier, and they said the biggest challenge is getting changes past the grey haired person in the corner office whose job is to say No. As in

  1. No: You cannot “move quickly and break things”. You must move carefully and keep safe.
  2. No: You cannot install untested stuff on production machines, without installing it first on test or development machines
  3. No: You cannot install products where you have to type commands or information onto the production machines. You have to provide files of information, or a document where commands or information is reviewed and can be copied using cut-and-paste. For example people have had problems with the letter o (for Orange) and zero, and i (for India) 1’s and l (for Lima).
  4. No: You cannot just run a script or command which changes z/OS configuration. All changes have to be reviewed. For example some changes to the security configuration affect all address spaces.
  5. No: You cannot run something that is “one way” for example adding a column to a data base table, to sort data, or delete something without a tested process for restoring and recovering the data.
  6. No: You cannot have the authority to issue commands, because you may issue commands outside of the documented install script, without going through change control.
  7. No: You cannot install something until we know the impact of running it. For example, the amount of CPU, real, virtual, disk storage.

Zowe

Zowe is a project which provides a command level interface from outside of z/OS to z/OS services. This means people do not need to know about ISPF, or the operator console to be able to use z/OS. Zowe has several Java address spaces which provide the facilities.

As part of the aim to make z/OS more accessible, the Zowe installation and configuration has command scripts which can be used to define the resources needed for Zowe to run.

I feel the “recommended” processes for configuring Zowe is fit for a test or development system, not for an important system, and so I recommend using a more traditional approach, which fits in the “No:… above”

Not all the configuration changes needed are listed in the documentation, for example you should configure WLM to give the address spaces appropriate priority. You need to backup data sets and files.

The security definitions enable a global facility which can impact many subsystems. This change alone would need careful roll out to all systems.

APF definitions are made using dynamic APF command interface on the console. They are not saved across IPL, so the changes have to be made to the SYS1.PARMLIB concatenation at some point. My view is just do the parmlib definitions once.

Zowe: Create the z/OS security definitions

The Zowe documentation for security definitions is here. I consider the documentation is dangerous, as it defines some global profiles which can affect other address spaces, and change their behaviour.

Before you start

You need to know

  • Which userid you want the main Zowe started tasks to run under.
  • Which user you want the cross memory server to run under.
  • The name of an admin group these userids will be a member of. You can connect your existing userids to this group to make them administrators.
  • The name of the started task for Zowe, and for the cross memory server.
  • The name of the Zowe APF authorised data set.

System wide definitions

If you use RACF you need the following RACF profiles defined – they may already be defined.

SETROPTS LIST                                        
RLIST   FACILITY BPX.SERVER ALL                      
RLIST   FACILITY BPX.DAEMON ALL                      
RLIST   FACILITY BPX.JOBNAME ALL                     
RLIST   FACILITY IRR.RUSERMAP ALL                    
RLIST   FACILITY IRR.RAUDITX ALL                     
RLIST   FACILITY IRR.IDIDMAP.QUERY ALL               
/* this is optional                                  
RLIST   APPL     OMVSAPPL          ALL

If you are missing the BPX.SERVER or the BPX.DAEMON profiles, then you need to carefully test your existing applications, because their behaviour may change when these profiles are defined.

 /* permit Zowe main server to create a user's security environment   */      
 /* ATTENTION: Defining the BPX.DAEMON or BPX.SERVER profile makes    */      
 /*            z/OS UNIX switch to z/OS UNIX level security. This is  */      
 /*            more secure, but it can impact operation of existing   */      
 /*            applications. Test this thoroughly before activating   */      
 /*            it on a production system.                             */      
  RDEFINE FACILITY BPX.DAEMON UACC(NONE)                                     
                                                                             
  RDEFINE FACILITY BPX.SERVER UACC(NONE)                                     
                                                                             
 /* permit Zowe main server to set job name                           */      
  RDEFINE FACILITY BPX.JOBNAME UACC(NONE)                                    
                                                                             
 /* permit Zowe main server to use client certificate mapping service */      
  RDEFINE FACILITY IRR.RUSERMAP UACC(NONE)                                   
                                                                             
 /* permit Zowe main server to use distributed identity mapping       */      
 /* service                                                                   
  RDEFINE FACILITY IRR.IDIDMAP.QUERY UACC(NONE)                              
                                                                             
 /* permit Zowe main server to cut SMF records                        */      
  RDEFINE FACILITY IRR.RAUDITX UACC(NONE)                                    
                                                                             
  SETROPTS RACLIST(APPL) REFRESH                                             
  SETROPTS RACLIST(FACILITY) REFRESH

Userid and group definitions

You need to have a userid to run the Zowe started tasks. Zowe also defines a different userid for the ZIS cross memory server. It is not clear to me why this is needed.

The userids need an admin group.

The definitions provided by Zowe for userids ZWESVUSR and ZWESIUSR; and the admin group are

/* - The sample commands assume automatic generation of GID is       */    
/*   enabled.                                                        */    
                                                                           
/* group for administrators                                          */    
/* replace AUTOGID with GID(&ADMINGID.) if AUTOGID not enabled       */    
  LISTGRP  ZWEADMIN OMVS                                                   
  ADDGROUP ZWEADMIN OMVS(AUTOGID) -                                        
   DATA('ZOWE ADMINISTRATORS')                                             
                                                                           
/* uncomment to add existing user IDs to the ADMINGRP group          */    
/* CONNECT (userid,userid,...) -                                     */    
/* GROUP(ZWEADMIN) AUTH(USE)                                        */     
                                                                           
/* userid for ZOWE main server                                       */    
/* replace AUTOUID with UID(&ZOWEUID.) if AUTOUID not enabled */           
  LISTUSER ZWESVUSR OMVS                                                   
  ADDUSER  ZWESVUSR -                                                      
   NOPASSWORD -                                                            
   DFLTGRP(ZWEADMIN) -                                                     
   OMVS(HOME(/tmp) PROGRAM(/bin/sh) AUTOUID) -                             
   NAME('ZOWE SERVER') -                                                   
   DATA('ZOWE MAIN SERVER')  
 /* userid for ZIS cross memory server                                */    
 /* replace AUTOUID with UID(&ZISUID.) if AUTOUID not enabled */            
   LISTUSER ZWESIUSR OMVS                                                   
   ADDUSER  ZWESIUSR -                                                      
    NOPASSWORD -                                                            
    DFLTGRP(ZWEADMIN) -                                                     
    OMVS(HOME(/tmp) PROGRAM(/bin/sh) AUTOUID) -                             
    NAME('ZOWE ZIS SERVER') -                                               
    DATA('ZOWE ZIS CROSS MEMORY SERVER')                                  
                                                                            
 /* show results .................................................... */    
   LISTGRP  ZWEADMIN OMVS                                                   
   LISTUSER ZWESVUSR OMVS                                                   
   LISTUSER ZWESIUSR OMVS

Create a profile to protect the APF data set

  LISTDSD PREFIX(IBMUSER.ZWEV3A) ALL                                      
  ADDSD  'IBMUSER.ZWEV3A.*.**' UACC(READ) -                               
    DATA('Zowe')                                                          
  PERMIT 'IBMUSER.ZWEV3A.*.**' CLASS(DATASET) -                           
    ACCESS(ALTER) ID(ZWEADMIN)                                            
                                                                          
  SETROPTS GENERIC(DATASET) REFRESH                                       
                                                                          
/* show results .................................................... */   
  LISTGRP IBMUSER.ZWEV3A                                                  
  LISTDSD PREFIX(IBMUSER.ZWEV3A) ALL

Create the definitions for the started tasks

/* started task for ZOWE main server                               
  RLIST   STARTED ZWESLSTC* ALL STDATA                             
/*RDELETE STARTED ZWESLSTC*                                        
  RDEFINE STARTED ZWESLSTC* -                                      
   STDATA(USER(ZWESVUSR) -                                         
     GROUP(ZWEADMIN) -                                             
   TRUSTED(NO)) DATA('ZOWE MAIN SERVER')                           
                                                                   
/* started task for ZIS cross memory server                        
  RLIST   STARTED ZWESISTC* ALL STDATA                             
/*RDELETE STARTED ZWESISTC*                                        
  RDEFINE STARTED ZWESISTC* -                                      
   STDATA(USER(ZWESIUSR) -                                         
   GROUP(ZWEADMIN) -                                               
   TRUSTED(NO)) DATA('ZOWE ZIS CROSS MEMORY SERVER')               
                                                                   
/* started task for ZIS Auxiliary cross memory server              
  RLIST   STARTED ZWESASTC* ALL STDATA                             
/*RDELETE STARTED ZWESASTC*                                        
  RDEFINE STARTED ZWESASTC* -                                      
   STDATA(USER(ZWESIUSR) -                                         
   GROUP(ZWEADMIN) -                                               
   TRUSTED(NO)) DATA('ZOWE ZIS AUX CROSS MEMORY SERVER')           
                                                                   
  SETROPTS RACLIST(STARTED) REFRESH                                
                                                                   
  RLIST STARTED ZWESLSTC* ALL STDATA                               
  RLIST STARTED ZWESISTC* ALL STDATA                               
  RLIST STARTED ZWESASTC* ALL STDATA

Define a profile and permissions to the cross memory server

/* DEFINE ZIS SECURITY RESOURCES ................................... */      
                                                                             
/* define ZIS security profile                                       */      
  RLIST   FACILITY ZWES.IS ALL                                               
  RDEFINE FACILITY ZWES.IS UACC(NONE)                                        
                                                                             
/* DEFINE AUX SERVER PERMISIONS .................................... */      
                                                                             
/* permit AUX STC to use ZIS cross memory server                     */      
  PERMIT ZWES.IS CLASS(FACILITY) ACCESS(READ) -                              
    ID(ZWESIUSR)                                                             
  SETROPTS RACLIST(FACILITY) REFRESH                                         
                                                                             
/* DEFINE ZOWE SERVER PERMISIONS ................................... */      
                                                                             
/* permit Zowe main server to use ZIS cross memory server            */      
  PERMIT ZWES.IS CLASS(FACILITY) ACCESS(READ) -                              
    ID(ZWESVUSR)                                                             
  SETROPTS RACLIST(FACILITY) REFRESH

Define other profiles

/* permit Zowe main server to create a user's security environment   */      
/* comment out the following 2 lines if the OMVSAPPL is not defined  */      
/* in your environment                                               */      
  PERMIT OMVSAPPL CLASS(APPL) ID(ZWESVUSR) -                                 
    ACCESS(READ)                                                             
  SETROPTS RACLIST(APPL) REFRESH                                             
                                                                             
  PERMIT IRR.RUSERMAP CLASS(FACILITY) ACCESS(READ) -                         
    ID(ZWESVUSR)                                                             
                                                                             
/* permit Zowe main server to use distributed identity mapping       */      
/* service                                                                   
  PERMIT IRR.IDIDMAP.QUERY CLASS(FACILITY) ACCESS(READ) -                    
   ID(ZWESVUSR)                                                              
                                                                             
/* permit Zowe main server to cut SMF records                        */      
  PERMIT IRR.RAUDITX CLASS(FACILITY) ACCESS(READ) -                          
    ID(ZWESVUSR)

Define another profile

/* DEFINE ZOWE RESOURCE PROTECTION ................................. */      
                                                                             
/* - Defines resource APIML.SERVICES that controls access to         */      
/*   detailed information about API services to Zowe users.          */      
                                                                             
/* define resource for information about API services                */      
  RDEFINE ZOWE APIML.SERVICES UACC(NONE)                                     
                                                                             
/* uncomment and replace "user" to permit Zowe users to access       */      
/* the resource:                                                     */      
/* PERMIT APIML.SERVICES CLASS(ZOWE) ID(user) ACCESS(READ)           */      
                                                                             
/* show results                                                      */      
  RLIST ZOWE *

z/OSMF changes

Same origin policy. Wikipedia says the same-origin policy (SOP) is a concept in the web-app application security model. Under the policy, a web browser permits scripts contained in a first web page to access data in a second web page, but only if both web pages have the same origin.

Having a REST request go from Zowe to z/OSMF counts as a cross-origin resource sharing (CORS)

The z/OSMF documentation says

Enabling cross-origin resource sharing (CORS) for REST services

Your installation can allow browser applications from certain, trusted sites to access z/OSMF REST services on the host system. If so, you must enable cross-origin resource sharing (CORS) on the host system. This work involves creating an “allow list” of exceptions (the trusted sites), and enabling those exceptions in your external security manager.

Identify which sites are to be allowed, and which REST interfaces are to be made available for cross-site access. Then, work with your security administrator to create the appropriate authorizations in your external security manager. In a RACF installation, for example, define generic or discrete profiles for the remote sites in the ZMFAPLA class, and permit the profiles to the z/OSMF REST interfaces.

For example

RDEFINE ZMFAPLA IZUDFLT.REST.JOBS.COM.IBM.LAB2 UACC(NONE)

Zowe Explorer access

A userid using the Zowe explorer access into z/OSMF needs

  • Access to the ISPF procedure
  • Access to the TSO Account number
  • Read access to CSFRNGL Class(CSFSERV )

Zowe: Installing and configuring z/OS system definitions

Installing Zowe – getting the libraries to z/OS

You can install Zowe using SMP/E or using a PAX file.

Follow the instructions.

  • From SMP/E
  • From a .pax file Where it says pax -ppx -rf .pax
    • cd to the product library for example cd /usr/lpp/zowe
    • within the directory, unpax using pax -ppx -rf path-to-pax/.pax such as pax -ppx -rf /tmp/zowe3.0.pax
    • you can remove the pax file if you no longer need it. Once you have the libraries installed, you edit a zowe.yaml file to define Zowe parameters, then you run a step to create data sets and other objects.

General comment

In the following sections, it refers to configuration keys by using the concatenation of key names and dots.
For example, if you want to update the configuration key zowe.certificate.keystore.type with the value PKCS12, you should set the value for this entry in the zowe.yaml:

zowe:
  # Specify the certificate keystore type
  certificate:
    keystore:
      type: PKCS12

where each level is indented two more spaces than the preceding level

If you search for “zowe.certificate.keystore.type”, you will not find it in the file.

Within the configuration is a data set prefix ZWEV3. You may want to use a name like PP.ZWE3 which fits with your data set naming convention.

My set up

/u/tmp/zowep contains my installed libraries

/u/tmp/zowec contains my configuration

Configuring Zowe

The Zowe documentation on the customisation process is here.

  • You should have Zowe and the prereqs installed, for example Zowe
    installed in /u/tmp/zowep.
  • Create the directory for the Zowe instance configuration file,
    for example /u/tmp/zowec

The instructions below configure the minimal function Zowe instance. This makes it quicker to resolve set up issues, such as certificate and keyring problems.
The instructions then add more function, until it is fully configured.

How to configure Zowe

The Zowe documentation takes the approach of using a large .yaml file for all configuration information. I found this unwieldy, and hard to use.
The customising information has logically separate sections covering

  • Creating the PDS
  • Creating proclib and parmlib members
    • APF definitions
  • Creating security definitions
  • Creating certificates
  • Configuring what runs when Zowe is started.

I found it easier to have a configuration file for each

Creating the PDS

The Zowe documentation describes the official way.

Use /u/tmp/zowep/bin/zwe install –ds-prefix value

Note: That is – – ds-prefix with two dashes.

Where value is the high level qualifier to use. For example PP.ZOWE31

If you use –ds-prefix you do not need to use a zowe.yaml configuration file.

This creates some data sets with the specified High Level Qualifier.

If you are rerunning the step, or upgrading use

/u/tmp/zowep/bin/zwe install –ds-prefix value –allow-overwrite

Create instance specific data sets and members

See the Zowe documentation .

DescriptionParameterDefault value
Where Zowe MVS data sets will be installedprefix:IBMUSER.ZWEV3A
PROCLIB where Zowe STCs will be copied overproclib:USER.PROCLIB
Zowe PARMLIBparmlib:IBMUSER.ZWEV3A.CUST.PARMLIB
PARMLIB members for pluginsparmlibMembers: zisZWESIP00
JCL library where Zowe will store temporary JCLs during initializationjcllib:IBMUSER.ZWEV3A.CUST.JCLLIB
Utilities for use by Zowe and extensionsloadlib:IBMUSER.ZWEV3A.SZWELOAD
APF authorized LOADLIB for ZoweauthLoadlib:IBMUSER.ZWEV3A.SZWEAUTH
APF authorized LOADLIB for Zowe ZIS PluginsauthPluginLib:IBMUSER.ZWEV3A.CUST.ZWESAPL

I created a ins.yaml file with 

zowe: 
  setup: 
    dataset: 
      prefix: COLIN.ZOWE 
      parmlib: COLIN.ZOWE.CUST.PARMLIB 
      proclib: COLIN.ZOWE.CUST.PROCLIB  
      jcllib: COLIN.ZOWE.CUST.JCLLIB 
      authLoadlib: COLIN.ZOWE.SZWEAUTH 
      authPluginLib: COLIN.ZOWE.CUST.ZWESAPL

and use the command

/u/tmp/zowep/bin/zwe init mvs -c ins.yaml

APF authorise the data sets

The official way of APF defining the Zowe data sets is to use the command

/u/colin/zowep/bin/zwe init apfauth -c in.yaml

This issues some operator commands to dynamically APF authorise two data sets.

SETPROG APF,ADD,DSNAME=COLIN.ZOWE.SZWEAUTH,VOLUME=CBTU07                    
...      
SETPROG APF,ADD,DSNAME=COLIN.ZOWE.CUST.ZWESAPL,VOLUME=CBTU01                
...

This does not save definitions across IPL, so you might just as well as define the parameters manually

You can create or update a PROGXX member in the parmlib concatentation, such as PROGZW

APF ADD DSNAME(COLIN.ZOWE.SZWEAUTH) VOLUME(CBTU07)
APF ADD DSNAME(COLIN.ZOWE.CUST.ZWESAPL) VOLUME(CBTU07)

You can the set prog=ZW to activate it.

Once this works, you will need to put the PROGZW in the IPL startup list.

Create the security definitions needed by Zowe.

See Zowe:Create the security definitions .

Create the Started Task JCL procedures

The Zowe command 

zwe init stc -c in.yaml

copies 3 members (ZWESISCH, ZWESISTC, ZWESLSTC) from COLIN.ZOWE.SZWESAMP into the user specified PROCLIB, and does a small amount of configuration. For example 

...
//********************************************************************/ 
//ZWELNCH  EXEC PGM=ZWELNCH,REGION=&RGN,TIME=NOLIMIT, 
// PARM='ENVAR(_CEE_ENVFILE=DD:STDENV),POSIX(ON)/&HAINST.' 
//STEPLIB  DD  DSNAME={zowe.setup.dataset.authLoadlib},                  
//STEPLIB  DD   DSNAME=COLIN.ZOWE.SZWEAUTH, 
//             DISP=SHR 
//SYSIN    DD  DUMMY
...
//STDENV   DD  * 
_CEE_ENVFILE_CONTINUATION= 
_CEE_RUNOPTS=HEAPPOOLS(OFF),HEAPPOOLS64(OFF) 
_EDC_UMASK_DFLT=0002 
CONFIG={ZWE_CLI_PARAMETER_CONFIG}                        
CONFIG=/u/colin/zowec/in.yaml 
/* 
...

The bold lines are substituted with the configuration values.

For the ZWESLSTC member, check the correct yaml file is being used.

Once you have checked the JCL, and made any site specific changes, you need to copy them to the SYS1.PROCLIB concatenation.

You should use a started task procedure name of 7 characters or less. Zowe creates BPXAS jobs with the zowe procedure name as a prefix. If the name is 7 characters or less, these jobs are like ZWESTC1, ZWESTC2, ZWESTC3 etc.

If you use a procedure name of 8 characters, then the BPXAS jobs all have the same name, so when you issue an operator command such as F ZWESLSTC,APPL=DISPLAY, all the jobs will be passed the request, and you will get IEE342I MODIFY REJECTED-TASK BUSY from each of the BPXAS tasks.

Creating certificates

You can use existing certificate, key stores and trust stores. You will need to specify the names of them in the run time configuration.

Create the SCHED member definitions

Two modules have to be defined as KEY(4) NOSWAP. Either add them to an existing SCHEDxx member, or create a SCHEDZW member,

The source is in .SZWESAMP(ZWESISCH)

PPT PGMNAME(ZWESIS01) KEY(4) NOSWAP      
PPT PGMNAME(ZWESAUX)  KEY(4) NOSWAP

Zowe:Set-up messages on z/OS

I experienced the messages below when using Zowe. I’ve included additional text to further explain the messages

BPXP005I A FORK OR SPAWN ERROR WAS ENCOUNTERED. RETURN CODE 00000070 REASON CODE 0B250012

Unix return code  00000070 maps to EAGAIN (the resource is temporarily unavailable)

Unix reason code 0B250012 maps to JRMaxChild (the maximum number of processes for this user ID has been exceeded)

Increase the appropriate value MAXPROCSYS or MAXPROCUSER. This can be done dynamically using the SETOMVS command.

Use

  • D OMVS,O to display maxprocuser
  • setomvs MAXPROCUSER=40

BPXI039I SYSTEM LIMIT SHRLIBRGNSIZE HAS REACHED 90% OF ITS CURRENT CAPACITY OF 83886080

See Tuning the z/OS shared library region.

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -262894

JVMSHRC337E Platform error message: semctl : EDC5111I Permission denied. (errno2=0x070E0303)
JVMSHRC028E Permission Denied
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine

-262894 is SEMCTL EACCESS – Permission denied.

The owner of the shared file cache was not the userid trying to use it.

chown ZWESVUSR:SYS1 *

Changed the files, such as

ZWESVUSR SYS1     314572800 Feb  9 08:43 C290M17F1A64S_zoweGW_G43L00                   
ZWESVUSR SYS1          32 Feb 10 06:35 C290M17F1A64_semaphore_zoweGW_G43L00            
ZWESVUSR SYS1          40 Feb 10 06:35 C290M17F1A64_memory_zoweGW_G43L00

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360

JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.

-197360 is SEMGET ENOENT – No such file, directory, or IPC member exists.

I got these trying to restore a sharedclasses cache, when I did not have access to the file.

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot

I gave the userid access and it worked

chmod 777 /u/tmp/zowec/javasharedresources/*

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC336E Port layer error code = -459502


JVMSHRC337E Platform error message: shmctl : EDC5111I Permission denied.
JVMSHRC028E Permission Denied
JVMSHRC626I The stats of the shared cache cannot be obtained since a valid shared cache does not exist.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed

-459502 is SHMCTL EACCESS -Permission is denied.

The userid issuing the command does not have access to the resource.

The documentation says the shared class cache is created with ONLY USER read/write access by default unless the groupAccess command-line suboption is used, in which case the access is read/write for user and groups.

Note: Users with super user authority gid=0(SYS1) can issue the command with no additional authority.

To find the group list the directories containing the cache, for example if /var/zosmf/data/logs/.classCache/ was specified use ls -ltr /var/zosmf/data/logs/.classCache/javasharedresources.

For me it had owner IZUSVR group IZUADMIN.

I used the RACF command connect COLIN group(IZUADMIN) to connect the userid to the group. Even then the command failed, because groupAccess had not been defined on the -Xshareclasses… parameter. I had to delete the cache so it was recreated next time theJVM started. Then the java -Xshareclasses:cacheDir=/var/zosmf/data/logs/.classCache,name=liberty-IZUSVR,verbose,printStats worked.

JVMSHRC023E   Cache does not exist

I had

-Xshareclasses:cacheDir=/javasc,name=izusvr1cache,printStats

I had to remove the printStats.

JVMSHRC364E SH_OSCachesysv::acquireWriteLock() call to j9shsem_wait on semid … has failed with error -328433.

-328433 is SEMOP – The parameter is incorrect.

You can use the ipcs Unix commands to display the semaphore ids.

JVMSHRC005I No shared class caches available

I was using

/usr/lpp/java/J11.0_64/bin/java -Xshareclasses:cacheDir=/global/zosmf/data/logs/.classCache/,verbose,listAllCaches

to display information about shared cache usage, and kept getting the JVMSHRC005I No shared class caches available message. I experienced two reasons for this.

  1. The information in the file, was for last week’s IPL, and the the information in today’s memory was invalid.
  2. I was using the wrong level of Java. Once I used the right level of Java it worked!

restoreFromSnapshot

IBMUSER:/u/ibmuser: >cd /u/tmp/zowec
IBMUSER:/u/tmp/zowec: >/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot
JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360
JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.
JVMSHRC727E An error has occurred in creating the new non-persistent shared cache
JVMSHRC808I Compressed references shared cache “zoweGW” is destroyed. Use option -Xnocompressedrefs if you want to destroy a non-com pressed references cache.
JVMSHRC699E Failed to restore the non-persistent shared cache “zoweGW” from the snapshot

This may be connected to the the following

The following files were in the directory 

-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_semaphore_zoweGW_G43L00
-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_memory_zoweGW_G43L00

For example the above files were had owner: ZWESVUSR group: ZWEADMIN.

The userid was in group ZWEADMIN, and so does not get R/W access to the files.

Errno2

  • 0x071D0303: JRIpcDenied: Access was denied because the caller does not have the correct permission.
  • 0x053b006c: JRFileNotThere: The requested file does not exist
  • 0x0594003d: JRDirNotFound: A directory in the pathname was not found

Zowe:Starting Zowe on z/OS

This post will guide you starting a minimal Zowe instance and extending it.

You need to have configured the z/OS system ( started tasks, security, APF, SCHEDxx ) and configured the Zowe instance file, zowe.yaml.

In the zowe.yaml file find “components:” in column 1, and change the enabled value of the components to “false”.

  • gateway
  • discovery
  • caching-services
  • app-server
  • explorer-jes
  • explorer-mvs

but keep zss enabled.

Start the common services started task

Start the common services task. The default is ZWESISTC, but you may have changed the name of it.

s ZWESISTC

This displays output like

IEF403I ZWESISTC - STARTED - TIME=08.19.08                                                           
ZWES0001I ZSS Cross-Memory Server starting, version is 3.1.0+20250108                               
IEF761I ZWESISTC ZWESISTC PARMLIB ZWESIS DD IS ALREADY ALLOCATED AND WILL BE USED BY THIS TASK.       
IEE252I MEMBER ZWESIP00 FOUND IN COLIN.ZOWE.CUST.PARMLIB                                            
ZWES0105I Core server initialization started                                                     
ZWES0109I Core server ready                                                           
ZWES0200I Modify commands: 
...

Resolve any problems.

Start the main task.

Once the cross memory server has started successfully, start the main task

The default started task name is ZWESLSTC, but you may have changed it.

s ZWESLSTC

It generates messages like

$HASP373 ZWESLSTC  STARTED                                              
IEF403I ZWESLSTC - STARTED - TIME=10.02.38                              
+ZWEL0021I Zowe Launcher starting                                      
+ZWEL0018I Zowe instance prepared successfully                         
+ZWEL0006I starting components                                         
+ZWEL0001I component gateway started                                   
+ZWEL0001I component zss started                                       
+ZWES1013I ZSS Server has started. Version '3.1.0+20250108' 64-bit

You stop it using using 

 P ZWESLSTC                                  
 +ZWEL0008I stopping components             
 +ZWEL0002I component gateway stopped

It it fails to start, check the job output.

There is a log of activities SYSPRINT, and problems may be reported in SYS0001, for example

ZWEL0318E – Couldn’t scan file ‘/u/colin/zowec/zowe31.yaml’: mapping values are not allowed in this context at line 146, column 17.

Connecting to the gateway

Once you Zowe, with zss and the gateway, starts successfully, you can try to connect to it.

Before you can do this your web server, curl or other tool needs access to the Certificate Authority certificate.

I exported it using

//COLINEXP JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//STEPLIB  DD  DISP=SHR,DSN=SYS1.MIGLIB 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CERTAUTH   EXPORT(LABEL('DOCZOSCA'))   - 
       DSN('COLIN.CERT.DOC.CA.PEM') - 
       FORMAT(CERTB64) - 
       PASSWORD('PASSWORD') 
//

It creates a data set like 

-----BEGIN CERTIFICATE----- 
MIIDYDCCAkigAwIBAgIBADANBgkqhkiG9w0BAQsFADAwMQ4wDAYDVQQKEwVDT0xJ 
...
wL6XUA== 
-----END CERTIFICATE-----

I created a file on Linux/Uss called cacert.pem, and pasted the contents into it.

I used curl (from z/OS) to access the gateway

trace="--trace trace.txt1" 
ca="--cacert /u/colin/zowec/cacert.pem" 
curl  -v $trace $ca  https://127.0.0.1:7557/plugins

This gave data like

{"pluginDefinitions":[{"identifier":"org.zowe.explorer-jes","apiVersion":"2.0.0"....

if you get this then you successfully connected to the gateway.

The trace.txt1 file has the TLS handshake and the data content. For example it contained

== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey 
== Info: ALPN: server did not agree on a protocol. Uses default. 
== Info: Server certificate: 
== Info:  subject: O=MYORG; OU=COLIN; CN=127.0.0.1 
== Info:  start date: May  2 05:00:00 2025 GMT 
== Info:  expire date: Jul  3 04:59:59 2029 GMT 
== Info:  subjectAltName: host "127.0.0.1" matched cert's IP address! 
== Info:  issuer: O=COLIN; OU=CA; CN=DOCZOSCA 
== Info:  SSL certificate verify ok. 
== Info:   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption 
== Info:   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption 
== Info: Connected to 127.0.0.1 (127.0.0.1) port 7557

When you start Zowe it spawns threads into BPXAS address spaces, which show up as jobs with the same name (ZWESLSTC) but you cannot look at their output in the spool.

Problem determination

  • Check for messages on the system log.
  • Check within ZWESLSTC started task output for error messages.
  • You can enable debug trace in the zowe.yaml file with the launchScript: logLevel set to debug. Reset it to info once the problem is resolved.
  • You can set debug: true for the component with problems.

If the zss fails to start, check in the logs directory of the instance directory for the zss… log file, such as

*/u/tmp/zowec/logs/zssServer-2025-02-03-08-15.log*

If the system starts, but you cannot connect from a web browser,

change the yaml file and uncomment the https: trace statement

  zss: 
    enabled: true 
    port: 7557 
    crossMemoryServerName: ZWESIS_STD 
    agent: 
      jwt: 
        fallback: true 
      64bit: true 
      https: 
        trace: true

When you start Zowe it will produce a trace file like

logs/zssServer-2025-02-03-17-36.log.tlstrace

which you can format with

gsktrace logs/zssServer-2025-02-03-17-36.log.tlstrace >gsk.txt

I got

No supported CertificateVerify signature algorithm for EC key

Which means then the specified server key is not acceptable. For example Elliptic Curves with length 521 are not supported in all client environments, and key size 256 should be used.

Start the API Server

The API server needs to authenticate the user.

You can use

  • 1. SAF interface
  • 2. Another interface

To use the SAF interface you need to start the zss server