Category: client

Wow, how to logon securely is so complex….

I’ve been looking into login on to Zowe and z/OSMF, and have realised how complex it is to set up secure logon.

An obvious fact about real life

If someone has access to your machine, for example a hacker, then they have access to your files, and data in virtual storage.

I cover

A lot of the time it is not difficult, but you need to handle the edge cases. When there are multiple dimensions to the problem, the edge case becomes the corner case, and this is where it gets really hard.

I use the expression is not secure. Being secure is relative. If your machine is air gapped with no external access, the machine should be secure. If people can access your files, perhaps on a shared machine, or a hacker can accessed your machine, I would not consider this secure.

The simplest case of logging on manually from a client machine to the server

Of course you use a TLS session to ensure the session traffic is encrypted.

You can type the values into the userid and password fields. It works, simple. If I disconnect and reconnect, I need to re-enter the userid and password

If I am using more than one back end server, I’ll have to enter the userid and password for each system, while the session is active.

Once the userid and password have been used, the fields in memory should be overwritten (but I doubt if this is always the case), to minimise the time window when a hacker extract the values from memory.

The simple case of logging on from a script from the client to the server.

You will not be there to enter the password so it needs to be stored.

The script may prompt you for the userid and password. You enter the information once, and it keeps them in virtual storage, and does not write them to disk. At the end of the session these values need to be overwritten in case a hacker is wandering round your system.

The application may write the information to disk – and we immediately hit a major problem. If a hacker has access to your system, then they can access (and copy away from your machine) the file where the password is stored. On Linux I displayed the contents of the “secure store” with a few lines of Python. I believe it is the same for Windows and Apple machines. (If your userid needs access to the secure store for something, a hacker thread running with your userid can access the store).

We have quickly seen that the use of a password stored on the machine is not secure.

Use of a certificate

As part of the TLS handshake there is a private key kept on the local machine, and a public key is sent to the partner.

Trusting a certificate

As part of setting up the public key, I send the public key to the server. The server’s Certificate Authority does a checksum of the public key, then encrypts the checksum with the server’s private key. The public certificate, encrypted checksum and the CA’s public key are sent back to the originator.

When the public certificate is used as part of the TLS handshake, the server does the same checksum calculation as before on public certificate, and saves it momentarily. It takes the encrypted checksum from the payload, decrypts it, and compares the two checksum values. If they match, then the client’s certificate can be trusted and has not been changed. Therefore the certificate is trusted (for a given level of trust) to represent the end user.

My program has access to my private key – so authenticates as me

As part of the handshake data, the client encrypts some data using its private key, and sends this to the server. The server uses the public key it received, and decrypts this value. If the decrypted data matches what it is expecting, then this is proof that the client has the private key and is who they say they are.

The server can then use the mapping of “name in the public key” to a userid, to determine which userid the requester should run under.

Once this is set up, the client can connect to the server without specifying a userid and password.

There is a proposal to reduce the validity time of personal certificates from over a year to 47 (or less) days. This reduces the time window when a certificate can be used.

This is great … but

If your private key is stored on your disk, a hacker can steal a copy of the file and impersonate you. So once again this is not very secure.

Use a private key – external to your computer

You can get a dongle, such as a Yubikey which keep the private key secured on a detachable USB dongle. To use it, a request is passed to the dongle saying “please encrypt this data”. It encrypts the data, returns it, and it can be sent to the server. You cannot extract the private key from the dongle.

To use this for authentication you physically need the dongle. If someone has control of your machine, they can use this dongle when it is plugged in. If they just have a copy of your files they cannot use the dongle, and so this is secure.

If you remove the dongle when you are not using it, we have a secure solution, until you plug it back in! Most people may forget and leave the dongle in all day.

I’ve been to sites, where the private key is stored on their badge. When they want to authenticate they put their badge on a badge reader and authenticate. This is awkward, so they immediately take their badge off the reader after authentication.

Setting up these external key stores is not trivial.

Use of Multi Factor Authentication(MFA)

One of the main approaches to MFA is

  • something you have
  • something you know

MFA is often used in applications such as online banking.For some transactions you will have a code sent to your email address. Something you know is your application password, something you have is the code sent to you. To use the service, an application password and access to your email is required.

MFA can also use one time passwords – so if someone is monitoring your network traffic they will not be able to reuse the one-time-password.

You can get the one-time-password generated from your badge, a dongle attached to your machine, or an application, such as an authenticator application on your phone. Once these apps have been configured to the back-end, you can press a button and get a one time code.

To logon you may need your normal logon password (which you change monthly) and the one time code from the MFA device. You might just need the one-time-code – depends on how the environment has been set up.

This is great for you logon to a backed and stay connected. If you have script, or are using multiple back-end servers. This becomes impractical. You need a new one-time-code each time you logon. This can be automated if the MFA device is attached to your machine, but not if you are using a mobile phone application.

One small problem is when the user’s password has expired. They are prompted to change it, and now need another One Time Password – and they may have to wait for a period (seconds) before a new one is generated.

JSON Web tokens

See Are JSON Web Tokens secure? – Yes if used properly.

A JSON Web Token(JWT) provides a time limited key, and so avoids the problems when changing a password. Some of the concepts are similar to a certificate logon, but with a limited validity, from minutes to hours.

With RACF these are known as Identity token. (Having a different name, means RACF development can extend the support to cover additional tokens types.)

A JWT has three parts

  • Information about the JWT. This JWT has used an RSA certificate, and algorithm SHA256.
  • Identity information. The userid is…, it was issued by ( z/OSMF, or SAF), at this time…. and is valid until….
  • The above parts are check summed, and the check sum is encrypted using a private key.

The above 3 parts are base 64 encoded, and joined together with a ‘.’ between them.

Once a JWT token is created, it is valid until it expires. You need to consider if you want a JWT to be valid for a few minutes, or all day.

You create (or have the system create for you) the header and the payload. You calculate the checksum, and encrypt the checksum with the private key.

The parts of the JWT are assembled and sent to the partner.

The partner uses its copy of the public key, and checks that the checksum matches what it expects and, if it matches, can use the information in the payload part. This validation could be a call to a RACF service, a https request to a server, or Java.

Once the JWT has been validated, the server can use the information in the payload.

You do not want to use the userid in the JWT directly, because the userid COLIN on one system has no authority, but has super user authority on another system.

With RACF you can map a userid string to a RACF userid using profiles defined with the RACMAP command. See the r_usermap service.

While the JWT is valid it could be used by a hacker.

Overall

It looks like there is not one good solution to cover all cases.

Someone said “assume your machine will be hacked. Minimise what damage they can do”. This may be as simple as turning your machine off overnight, so the hackers cannot control it

The solution for scripts may be different to people logging on.

You may want to have a userid just for scripts, which has only enough authority and permissions to work, and no additional permissions.

You need to consider the options and risks, and not just sleepwalk into having an insecure system.

Zowe cli help command is not helpful!

The zowe cli help option does not easily tell you how to get all of the help. In order to get the syntax of the command – you have to know the full command with the and then add the --help option! (This is working as designed!)

There is some online help here in a tree view or a “flat view of all of the commands“.

Whoops profile options not found


Step 1

The command zowe --help gives output including

USAGE
   zowe <group>

   Where <group> is one of the following:

 GROUPS
   auth                          Connect to Zowe API ML authentication service    
   config                        Manage JSON project and global configuration
   zos-console | console         Issue z/OS console commands and collect responses
 
...

Step 2

Now you know there is a console command….

The command zowe --help console gives output including

 USAGE
 
   zowe zos-console <group>

   Where <group> is one of the following:

 GROUPS

   collect Collect z/OS console command responses
   issue   Issue z/OS console commands

Step 3

Now you know there is a console issue command…

The command zowe --help console issue finally gives lots of output including

  • OPTIONS
    • --console-name | --cn | -c
    • --include-details | --id | -i
    • --key-only | --ko | -k (boolean)
    • --return-first | --rf | -r (boolean)
    • --solicited-keyword | --sk | -s (string)
  • ZOSMF CONNECTION OPTIONS
    • --host | -H (string) The z/OSMF server host name.
    • --port | -P (number) The z/OSMF server port. Default value: 443
    • --user | -u (string) Mainframe (z/OSMF) user name, which can be the same as your TSO login. Your TSO logon userid
    • --password | --pass | --pw (string) Mainframe (z/OSMF) password, which can be the same as your TSO password. Your TSO userid’s password
    • --reject-unauthorized | --ru (boolean) Reject self-signed certificates. Default value: true
    • --base-path | --bp (string) The base path for your API mediation layer instance. Specify this option to prepend the base path to all z/OSMF resources when making REST requests. Do not specify this option if you are not using an API mediation layer.
    • --protocol (string)
    • --cert-file (local file path) The file path to a certificate file to use for authentication
    • --cert-key-file (local file path) The file path to a certificate key file to use for authentication
    • --completion-timeout | --cto (number) The amount in time, in seconds, a REST operation should wait to complete before timing out
    • --establish-connection-timeout | --ecto (number) The amount of time, in seconds, a REST operation should wait while connecting to the server before timing out.
  • PROFILE OPTIONS
    • --zosmf-profile | --zosmf-p (string) The name of a (zosmf) profile to load for this command execution.
    • --base-profile | --base-p (string) The name of a (base) profile to load for this command execution.
  • BASE CONNECTION OPTIONS
    • --token-type | --tt (string) The type of token to get and use for the API. Omit this option to use the default token type, which is provided by ‘zowe auth login’.
    • --token-value | --tv (string) The value of the token to pass to the API.
  • MQ options
    • --mq-profile | --mq-p (string) The name of a (MQ) profile to load for this command execution.

Now you know what the options are you can search for them. This pointed me to the console command page.

Whoops profile options not found

I fell over trying to specify a nested profile.

For example 

...
 "profiles": {
        "qa_lpar": { // Base profile connection properties are used unless overriden
            "type": "base",
            "properties": {
            }
        },
        "profiles": {
            "mq": {... 
            },

This is referred to as qa_lpar.mq .

What would I have done?

Personally I would have have a help page which listed all of the common options then list commands for example

  • Common options
    • --host etc

Specific commands

Zowe: How to save connection parameters: zowe CLI client configuration files

A Zowe client needs information to be able to invoke a service on Zowe on z/OS. Typically these are

  • host address
  • the host port
  • the service
  • the client public key .pem file
  • the client private key .pem file
  • whether to not allow self signed certificates sent from the host
  • the file containing the CA from the host servers.

These options can be given on the zowe cli command, or can be stored in profile files.

You can have multiple profiles, for example, tso, ssh, z/OSMF, production z/OSMF and test z/OSMF.

The above information may be different for different Zowe server instances- such as development, test and production, and for different applications within a Zowe instance.

Some configuration files are stored in the directory pointed to by the environment variable ZOWE_CLI_HOME, on Linux this defaults to ~/.zowe. These files are available where ever your working directory is. You can also have a configuration file in your working directory so is “application specific”.

Zowe supports a hierarchy of configuration files.

  • zowe.config.json within the current directory. If you change directory you will not use it.
  • zowe.config.user.json , the global user file in the ZOWE_CLI_HOME environment variable ( or it might default to ~/.zowe ). This is used for any user specific options (if any)
  • zowe.config.json the global file in the ZOWE_CLI_HOME environment variable ( or it might default to ~/.zowe ). This is used for project wide options.

The Zowe CLI command will start at the top of the hierarchy and go down until it finds the attributes. If it cannot find an attribute it may have a default.

Within the configuration file are different sections, for example

  • a base section with common parameters,
  • ftp, with specific parameters such as the FTP port number,
  • zOSMF, with with specific parameters such as the z/OSMF port number

You can edit the files manually, or use the Zowe config command to make changes.

Sigh

To specify a Certificate Authority(CA) certificate to a zowe cli command, you have to use an environment variable, for example

export NODE_EXTRA_CA_CERTS=/home/colinpaice/ssl/ssl2/doczosca.pem

You need to allow for this when writing your scripts, because different backend servers may have different CA certificates.

See here.

Getting started

Once you are familiar with Zowe and have a better idea of your Zowe configuration, you can develop the various configuration files; what information is user specific, what information is project specific, and what information is Zowe instance specific. Some files are stored in the direction in the ZOWE_CLI_HOME environment variable which defaults to ~/.zowe (on Linux).

If you may have more than one Zowe instance, you may decide NOT to use ~/.zowe, but have directories like ~/.zoweproduction and ~/.zowetest. This means you have to explicitly specify the location.

Get started by creating zowe.config.json in your working directory. See below, or use cut and paste from here, step 3. If you use an IDE such as slickedit or vscode, these have syntax assist, and can highlight errors as you type.

This is a good blog post on initial configuration. But does not cover use of certificates.

The Zowe config command

You can use the zowe config command to create and update the configuration files. The command has options including

This page has a good description of the options.

  • –dry-run : display the changes, but do not apply the changes
  • –edit and –editor name : this opens the file in the specified editor.

Create a zowe.config.json in the current directory

The command

zowe config init

will create a file zowe.config.json in the current directory.

I created a minimum (hand crafted) file to illustrate how the zowe config commands work.

{
    "$schema": "./zowe.schema.json",
    "profiles": {
        "zosmfcp": {
            "type": "zosmf",
            "properties": {
                "port": 443
            },
            "secure": []
        },
    
        "project_base": {
            "type": "base",
            "properties": {
                "host": "10.1.1.2",
                "rejectUnauthorized": true
            },
            "secure": [
                "user",
                "password"
            ]
        }
    },
    "defaults": {
        "zosmf": "zosmfcp",
        "base": "project_base"
    },
    "autoStore": true
}

Key elements of this file

  • “defaults”:
    • zosmf : For a z/OSMF request,if --base-profile is not specified, use the referenced profile – zosmfcp.
    • common parameters are defined in the base: profile -> project_base
  • “profiles”: There are two profiles defined
    • zosmfcp with the zosmf specific parameters
    • project_base with the common parameters.

When a zowe command is used, the parameter –base-profile can specify which profile to use.

Commands to display and set profile information

My working directory is /home/colinpaice/ssl/ssl2/ .

zowe config list –locations –root

This lists the only the file names used by Zowe CLI.

/home/colinpaice/ssl/ssl2/zowe.config.json
/home/colinpaice/.zowe/zowe.config.json

zowe config list –locations

This shows the file names and their contents

/home/colinpaice/ssl/ssl2/zowe.config.json: 
  $schema:   ./zowe.schema.json
  profiles: 
    zosmf: 
      type:       zosmf
      properties: 
        port: 10443
      secure: 
        (empty array)
    project_base: 
     ...
/home/colinpaice/.zowe/zowe.config.json: 
  $schema:   ./zowe.schema.json
  profiles: 
    zosmf: 
      type:       zosmf
      properties: 
        port: 10443
      secure: 
        (empty array)
    tso: 
      type:       tso
....
   global_base: 
      type:       base
      properties: 
        host:               10.1.1.2
        rejectUnauthorized: true
        user:               (secure value)
        password:           (secure value)
      secure: 
        - user
        - password
  defaults: 
    zosmf: zosmf
    tso:   tso
    ssh:   ssh
    base:  global_base
  autoStore: true

Change a parameter in the file in the current directory

zowe config set “profiles.zosmf.properties.port “30443”

Change/Add a profile (and content) to the the file in the current directory

zowe config set “profiles.zosmf2.properties.port” “40443”

If the profile zosmf2 did not exist before, it will create it, and add the property port

{
    "$schema": "./zowe.schema.json",
    "profiles": {
         ...
        "zosmf2": {
            "properties": {
                "port": 40443
            }
        }
    },
    ...
}

Set a parameter in the file in the user configuration file

zowe config set “profiles.zosmf2.properties.port” “40443” –user-config

set a parameter in the file in the global configuration file

zowe config set “profiles.zosmf2.properties.port” “50443” –global-config

Example of using the set commands

zowe config set “profiles.zosmf2.properties.port” “50443” –global-config
zowe config set “profiles.zosmf2.properties.port” “4444” –user-config
zowe config set “profiles.zosmf2.properties.port” “1111”

  • file zowe.config.user.json has zosmf2.properties.port:4444
  • file zowe.config.json: has zosmf2.properties.port: 1111
  • file ~/.zowe/zowe.config.json: has zosmf2.properties.port: 50443

Using the list commands, and displaying only the above fields I have

zowe config list gives

profiles: 
  ...
  zosmf2: 
    properties: 
      port: 4444

zowe config list –locations gives

/home/colinpaice/ssl/ssl2/zowe.config.user.json: 
  profiles: 
    zosmf2: 
      properties: 
        port: 4444
  defaults: 
/home/colinpaice/ssl/ssl2/zowe.config.json:
  profiles:
    ...
    zosmf2: 
      properties: 
        port: 1111
    ... 
/home/colinpaice/.zowe/zowe.config.json: 
  $schema:   ./zowe.schema.json
  profiles: 
   ...
   zosmf2: 
      properties: 
        port: 50443

So we can see the profile /home/colinpaice/ssl/ssl2/zowe.config.user.json: provided the port value 444.

If I use –base-profile zosmf2, the port used will be 4444.

If I add

zowe config set “profiles.zosmf2.properties.ca” “MYCA”

This create property zosmf2:properties:ca:”MYCA” in file zowe.config.json.

zowe config list gives

profiles:
  zosmf2: 
    properties: 
      port: 4444
      ca:   MYCA

so we can see what is being used.

If I use zowe config list –locations I can see the definition is in /home/colinpaice/ssl/ssl2/zowe.config.json

display profiles

zowe config profiles gave

zosmf
tso
ssh
global_base
zosmf2

What are the valid options?

If you used the zowe config init command, then this generated a schema file.

For zosmf this schema has

  • “host”: “The z/OSMF server host name.”
  • “port”: “default”: 443
  • “user”: “Mainframe (z/OSMF) user name, which can be the same as your TSO login.”
  • “password”: “Mainframe (z/OSMF) password, which can be the same as your TSO password.”
  • “rejectUnauthorized”:”Reject self-signed certificates.”, “default”: true
  • “certFile”: “The file path to a certificate file to use for authentication” “certKeyFile”: “The file path to a certificate key file to use for authentication”
  • “basePath”: “The base path for your API mediation layer instance. Specify this option to prepend the base path to all z/OSMF resources when making REST requests. Do not specify this option if you are not using an API mediation layer.”
  • “protocol”: “The protocol used (HTTP or HTTPS)” “default”: “https”, “http”, “https”
  • “encoding”: “The encoding for download and upload of z/OS data set and USS files. The default encoding if not specified is IBM-1047.”
  • “responseTimeout”: “The maximum amount of time in seconds the z/OSMF Files TSO servlet should run before returning a response. Any request exceeding this amount of time will be terminated and return an error. Allowed values: 5 – 600”

The CA certificate to use is specified by an environment variable, and is not a parameter in the profiles.


Zowe: What should I used on my client to talk to z/OS?

There are different ways of accessing the z/OS server.

  • You can use a web server. You will need to configure the certificates.
  • You can write your own program, for example in C or Java.
  • cUrl is a very common tool. Handy for small tasks.
  • HTTPie is an easy tool to generate one of commands. The output comes back nicely formatted, with colour. HTTPie is written in Python.
  • Python has good support for TLS and working with back send-servers. You can capture the data in a dictionary to make it easy to process.
  • You can use zowe Command Level Interface (CLI). This uses the best bits of all of the above.

Which is best?

The answer is the old phrase “horses for courses”.

If you want to do a one-of then the above tools are all very similar.

If you want to use multiple back ends, for example production and test, then you need to have code which selects the host, port, certificates, and applications within a z/OSMF or Zowe instance, then you will need to have code which handles this. Zowe CLI has this code built in.

I could write some basic code in about 100 lines of Python to do this – but why bother?

A Zowe developer’s view

I asked a developer what is good about zowe CLI.

The most common use case I see with Zowe CLI is for use in automation/Jenkins. It saves you the trouble of having to write code to handle the REST requests. It also supports the use of plug-ins for other products or services running on z/OS, like Endeavor, ChangeMan, Db2, CICS, etc. to do more than just download/upload files or submit jobs (e.g., you could also interact with a version control system, issue Db2 queries, deploy changes to a CICS region, and so on). 

Regarding output/data types, Zowe CLI commands can also provide json output (by specifying the `–response-format-json` or `–rfj` flag on commands), which is pretty easy to work with in bash scripts using something jq on Linux to parse the output and extract fields that you need.

For example

With –response-format-json

zowe zos-files list data-set ‘COLIN.Z*’ … –response-format-json

Generated

{
  "success": true,
  "exitCode": 0,
  "message": "",
  "stdout": "COLIN.ZCONNECT.BETA.ZFS\nCOLIN.ZCONNECT.BETA.ZFS.DATA\nCOLIN.ZFS\n...
  "stderr": "",
  "data": {
    "success": true,
    "commandResponse": null,
    "apiResponse": {
      "items": [
        {
          "dsname": "COLIN.ZCONNECT.BETA.ZFS"
        },
        {
          "dsname": "COLIN.ZCONNECT.BETA.ZFS.DATA"
        },
     ...
    "returnedRows": 44,
      "JSONversion": 1
    }
  }

Without –response-format-json

COLIN.ZCONNECT.BETA.ZFS
COLIN.ZCONNECT.BETA.ZFS.DATA
COLIN.ZFS
COLIN.ZFS.DATA
COLIN.ZFS.DSFS
COLIN.ZFS.DSFS.DATA
...

The first output is easier to post-process.

Some articles on Zowe CLI