HOME sweet HOME – understanding TCP/IP home statements

I was just(!) trying to get my Liberty web server running on z/OS to be able to be moved to a different LPAR, and get it working.  Moving it was easy, but the server’s certificate needs the IP address of the TCP/IP stack – with RACF you can only have one “Subject Alternative Name”.  A SAN of IP:10.1.2.4 works fine when it comes from TCP/IP stack 10.1.2.4 – but not from TCP/IP stack 10.1.2.5.   The web browser checks, and complains if they do not match.

To get this to work I read the z/OS TCP/IP documentation. There is lots of it, but it seems to be written for people who are experts in it.   There is a saying “Question: how do you eat an elephant?  Answer: A bit at a time”.   This post takes a small bit area – and expands it in terms I understand.  It may not be accurate – but the concepts should be right.

What is a TCP/IP stack?

This is another name of a TCP/IP instance, a started address space.

What is HOME?

Each connection coming into to a TCP/IP instance has an IP address. On TCPIP1 I have a connection (a virtual bit of wire) defined for IP address 10.1.1.2

On Linux if I use the command ip route get 10.1.1.2 it says

10.1.1.2 dev tap0 src 10.1.1.1 uid 1000

So 10.1.1.2  is going via device tap0 (which is a tunnel device, TAP = Tunnelling Application Protocol?).   The Linux machine has IP address 10.1.1.1.  Through some configuration magic this ends up in my TCP/IP instance as

DEVICE PORTA MPCIPA
LINK ETH1 IPAQENET PORTA
HOME 10.1.1.2 ETH1

Where 10.1.1.2 is the address for a link called ETH1 on the TCPIP instance on my LPAR. The magic is a bit like the Negro spirtual song Dem Bones which has “Thigh bone connected to the hip bone,  Hip bone connected to the back bone, Back bone connected to the shoulder bone Now hear the word of the Lord”.   ETH1 is defined as  being on PORTA, and PORTA is a device which maps to a tunnelled device using protocol MPCIPA.   …  maps to a VTAM TRL definition, Now hear the word of the Lord.

I can use the TSO netstat command for TCP address space called TCPIP1  netstat home tcp tcpip1

Home address list:
Address       Link        Flg
-------       ----        ---
10.1.1.2      ETH1        P
192.168.0.61  ETH2
10.1.1.5      EZASAMEMVS
127.0.0.1     LOOPBACK

For external connections the IP address, 10.1.1.2, must match with the definition on Linux.  This may be pointed to from outside Linux.  The other “home” connections are described below.

I have multiple instances working together.

I have three TCP/IP instances on my LPAR.  You might have an instance to talk to your internal network,(the intranet), and an instance talking to the internet, facing out from your enterprise.

You can also have TCP/IP instances with different security profiles, and provide total isolation.

You can set up connections between your enterprise and my enterprise,  these definitions will need a DEVICE, and a LINK etc as above (or an INTERFACE definition).

Setting up multiple instances within a Sysplex, or within an LPAR.

In your TCP/IP definitions you can set up

IPCONFIG DYNAMICXCF 10.1.1.6 255.255.255.0 2

and another instance with 10.1.1.5 etc.  The DYNAMICXCF says this is within the Sysplex(LPAR).  The software is smart enough to generate the device and link statements automatically.  In the netstat home command above, it gave 10.1.1.5 EZASAMEMVS which is eza_SAME_ MVS, and it has found a second TCP instance in the same LPAR.  See here for dynamic XCF, and here for the IPCONFIG statement.

I think you can use any unused IP address range; so you could use 2.2.2.5 and 2.2.2.6 instead of 10.1.1.5 and 10.1.1.6.  I believe these address are only used within the Sysplex.  So as long as these addresses are consistent and not being used else where, the values are not critical.

For a TCP/IP instance TCPIP3 on my LPAR with no external connections netstat home tc tcip3 gave me

Home address list:
Address   Link       Flg
-------   ----       ---
10.1.1.7  EZASAMEMVS P
127.0.0.1 LOOPBACK

This has the ever present LOOPBACK, and a virtual connection 10.1.1.7 to a TCP/IP instance in the same LPAR because of the ezaSAMEmvs definition

A practical path to installing Liberty and z/OS Connect servers – 10 use the MQ sample

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Use the MQ sample

You need to have installed the service, and protected it.

You need to configure MQ to include the MQ support and tell JMS where the libraries are

<server> 
<!-- Enable features --> 
<featureManager> 
    <feature>zosconnect:mqService-1.0</feature> 
</featureManager> 
                                                                                                         
<wmqJmsClient nativeLibraryPath="/Z24A/usr/lpp/mqm/V9R1M1/java/lib"/> 

<variable name="wmqJmsClient.rar.location"
   value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/jca/wmq.jmsra.rar"/> 
</server> 

You could configure a variable for the mq directory so you use it once, and use

<variable name="wmq"  value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/"
<wmqJmsClient nativeLibraryPath="${wmq}”/>
<variable name="wmqJmsClient.rar.location"
   value="${wmq}wmq.jmsra.rar}">

You could also pass the mq location as a variable in STDENV, and so pass it in through JCL.

Configure the jms to define the queue manager and queues

<server> 
 <jmsConnectionFactory jndiName="jms/cf1" 
     connectionManagerRef="ConMgr1"> 
    <properties.wmqJms transportType="BINDINGS" 
         queueManager="CSQ9"/> 
 </jmsConnectionFactory> 
                                                                                                      
 <connectionManager id="ConMgr1" maxPoolSize="5"/> 
                                                                                                      
 <!-- A queue definition where request messages 
      for the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockRequestQueue"> 
    <properties.wmqJms 
       baseQueueName="STOCK_REQUEST" 
       targetClient="MQ"/> 
 </jmsQueue> 
                                                                                                      
 <!-- A queue definition where response messages from 
      the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockResponseQueue"> 
    <properties.wmqJms baseQueueName="STOCK_RESPONSE" targetClient="MQ"/> 
 </jmsQueue> 
</server>

and include these in the server.xml file.

You need to compile and run the back end service.  See here.  Take care if using cut and paste as there a long lines which wrap, and cause compilation errors.

Because the MQ path name is long I used

export HLQ="/usr/lpp/mqm/V9R1M1/java/lib"

java -cp $HLQ/com.ibm.mq.allclient.jar:. -Djava.library.path=$HLQ TwoWayBackend CSQ9 STOCK_REQUEST STOCK_RESPONSE

I set up a job to run this in back ground, so I could free up my TSO terminal.

Use the API

Once installed you should be able to use the API. For example

curl –insecure -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem https://10.1.3.10:9443/stockmanager/stock/items/999999

If the back end application was working I got

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

If the back end application was not working I got back an empty response.

The back-end application runs until Ctrl+c is used.  On my USS, the ESCape key is cent symbol ¢ (Unicode 00a2) which I do not have on my default keyboard.    See  x3270 – where’s the money key? for guidance on how to set it.

 

Use the Service

To use the API I used a web browser with

https:10.1.3.10/9443/stockmanager/stock/items/999999

and got back

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

or curl with

 curl --insecure -X post -H Content-Type: application/json
  -H "Content-Type:application/json" -i --cacert cacert.pem
  --cert adcdd.pem:password --key adcdd.key.pem 
  --data {"STOCKQRYOperation": {"sqreq" : { "item_id": 2033}}}
  https://10.1.3.10:9443/zosConnect/services/stockQuery?action=invoke

A practical path to installing Liberty and z/OS Connect servers – 11 WLM classifying a service

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Classify the services with WLM to give them the right priority

You can configure an API or service, for example https://10.1.3.10:9443/stockmanager/stock/items/999999, so it gets an appropriate service class in WLM, for example high priority.

You can use RMF to report on service classes to see the response time profile, and if the service class is meeting its performance goals.

See here for a good article.

This page gives the example

<wlmClassification>
  <httpClassification transactionClass="CLASS001" 
      host="127.0.0.1" 
      port="9080" 
      method="GET"
      resource="/testResource" />
</wlmClassification>

You can classify the traffic depending on the IP address and port of the server, as well as the resource name.

A practical path to installing Liberty and z/OS Connect servers – 3 Initial customisation

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Copy the PDS from the install system to where they will run.

These files can be sent using FTP or XMIT.

Copy the product ZFS to where it will be used.

You can use DFDSS to dump the ZFS and move it to the SYSPLEX where it will be used.

  1. Mount the file system read/write for example /usr/lpp/IBM/zosconnect/v3r0beta
  2. Perform Setting up the product extensions directory.  You need to do this on each LPAR, as it set up files in /var/zosconnect
  3. Mount the file system READ
  4. Change the BPXPRMxx parmlib member so the file sytsem is mounted READ it at IPL
  5. You may want to create an alias /usr/zosc to the product executables.
    • ln -s /usr/lpp/IBM/zosconnect/v3r0beta/bin /usr/zosc

Create the angel started task if needed.

If you will be using an existing Angel you will need to set up RACF permissions.

Follow the instructions the z/OS Connect  documentation

  • If you need a named Angel, specify the value in the NAME=value.  You can specify NAME when you start the task.
  • You do not need to classify this in WLM as it uses little CPU once it has been started.

Create the file system for the server

  1. See Creating the z/OS Connect EE shared directory. You will need to know the path for server’s home directory. The path /var/zosconnect can only be used on the LPAR.   The files cannot be shared.
  2. If you need to allocate a file system.
    1. Allocate a ZFS. If the initial allocation of the ZFS is too small, then the ZFS will grow on demand, or using the zfsadm grow command to expand it.
    2. Mount the file system over the directory.
    3. Change the BPXPRMxx parmlib member so the file system is mounted RW at IPL.

Create the Z/OS Connect started task

See Creating a z/OS Connect Enterprise Edition server.

Using /usr/zosc alias defined above

export  WLP_USER_DIR=/var/zosconnect
/usr/zosc/zosconnect create default --template=zosconnect:default

If you need to start again, use the following to remove the server.

rm -r  /var/zosconnect/servers/default
  1. Copy BAQSTRT to the SYS1.PROCLIB concatenation, to the name you are going to use, BAQSTRT, BAQSTRT1 etc
  2. When you edit the member, use caps off to use lower case text in the member.
  3.  If you will be using MQ, add the MQ libraries to STEPLIB
    //STEPLIB   DD DSN=CSQ913.SCSQANLE,DISP=SHR 
    //          DD DSN=CSQ913.SCSQAUTH,DISP=SHR
  4. You may find it easier to remove the _CEE_RUNOPTS= and add
    //CEEOPTS DD DISP=SHR,DSN=...(CEEOPTS)

    to the JCL. This makes it easier to add CEE options, and specify storage heap and pool sizes.

  5. In my CEEOPTS I use RPTSTG(ON), to report the storage pool usage. This provides useful information which may be useful when investigating performance problems.
  6. Despite what the Liberty documentation says you do need
      • JVM_OPTIONS=<Optional JVM parameters>
    • for example
      • JVM_OPTIONS=-Xoptionsfile=/var/zosconnect/servers/default/jvm.options
  7. You may want a JVM specific to a server, as you may want to set some service specific options such as TLS trace.
  8. You may want to specify a default server name in the JCL parms. You can specify a different one when z/OS Connect is started.
  9. Specify WLP_USER_DIR for your server’s directory. You could make this as a parameter to the started task JCL if you want to provide isolation for the servers, and give each server its own file system.

Create the JVM options file

Go to the server’s directory.

cd /var/zosconnect/servers/default

use oedit jvm.options to create an EBCDIC file.

add

-Dcom.ibm.ws.zos.core.angelRequired=true 
#-Dcom.ibm.ws.zos.core.angelName=MYZANGL

The -Dcom.ibm.ws.zos.core.angelRequired=true option stops the server from starting if the angel process is not running. You get message:

CWWKB0116I: This server is not registered with an angel process even though it is configured to require registration with an angel process. This server is attempting to stop.

The option -Dcom.ibm.ws.zos.core.angelName=MYZANGL allows you to specify a non default angel process name. The # in front of the statement makes it into a comment.

add the following, if required, with your time zone.

-Duser.timezone=Europe/London

Override TCPIP

If you are using a non standard TCPIP, edit server.env and add

_BPXK_SETIBMOPT_TRANSPORT=TCPIP2

Tailor the server.xml file

It was easy to get my configuration options mixed up, and hard to compare two server configurations. I found it much easier to manage my configuration by putting the components into their own xml file. This also makes having shared components much easier to implement- as you just include the name of the shared file.

The configuration files have to be in ASCII, and you have to take care ensure the files are in the correct code page.

There are at least three ways of creating the .xml files (I use the third option through ISPF 3.17)

  1. using omvs commands to copy and edit existing ASCII files
  2. use omvs commands to create ASCII files
  3. use ispf option 3.17

1 Using omvs commands to copy an existing ascii file,  then edit it it

For example

cp server.xml httpEndpoint.xml

edit the httpEndpoint.xml to keep the <server> ,httpEndpoint information, </server> and delete the rest.  The file should look like

<server description="new server"> 
    <httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" /> 
</server>

then edit the server.xml and replace

<httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" />

with

<include location="${server.config.dir}/httpEndpoint.xml"/> 

Note: if you use the edit command source ? It should say ASCII already set.

2 Use OMVS to created and edit files

By default, if you edit a file in OMVS it becomes an EBCDIC file.

If you create a file without copying it, you need to use

touch httpEndpoint.xml 
chtag -t -c ISO8859-1 httpEndpoint.xml     
oedit httpEndpoint.xml

Make the same changes as above.

3 Use ispf option 3.17

If you use ISPF 3.17 (z/OS UNIX Directory List) you have more flexibility with editing files.

Edit server.xml

use the line commands mmmm and the primary command create to create a new member, give the new name when prompted, for example httpEndpoint.xml

Replace the lines you copied out with

	<include location="${server.config.dir}/httpEndpoint.xml"/>

and edit the file you just created to put <server> … </server> around the content.

Add more logging to the JCL’s STDOUT.

The default logging to the jobs STDOUT is only audit records. I found it much easier to include other levels of information so I did not have to keep looking in the message.log file. Note: the STDOUT content does not have time stamps so you do not know when the output was produced- so it is easier to use – but has less information!

I created logging.xml with the following content.

<server> 
 <logging 
 consoleLogLevel="INFO" 
 /> 

 <!-- this is a comment where I store trace information
   traceFormat="BASIC" traceSpecification="*=info:
       com.ibm.ws.security.*=fine:zos.native.03=fine" 
  end comment   -->  
</server>

I added

<include location="${server.config.dir}/logging.xml"/> 

to the server.xml file.

The consolLogLevel=”INFO” writes additional information to the STDOUT file, than the default AUDIT value which just outputs AUDIT records.

Define the http address and ports

Edit the httpEndpoint definitions

    <httpEndpoint id="defaultHttpEndpoint" 
                  accessLoggingRef="ALR" 
                  host="*" 
                  httpPort="9081" 
                  httpsPort="9443" /> 
    <httpAccessLogging 
       enabled="true" 
       id="ALR" 
       logFormat="%{c}a! %h %u %{t}W %q %r %s %b" 
   /> 

Check

  • host matches the TCPIP value to use. With my VIPA I specify the VIPA address host=”10.1.3.10”
  • httpPort – if you already have a Liberty instance running port 9081 will be in use by another server
  • httpsPort – if you already have a Liberty instance running port 9443 will be in use by another server

You will need the port values when you use the server.

The httpAccessLogging statement pointed to by accessLoggingRef, will create a log file with every attempt to use the server.

Example output

10.1.1.1! 10.1.1.1 ADCDC {12/Sep/2020:16:07:17 +0000} - 
GET /zosConnect/apis/stockmanager HTTP/1.1 200 296

Where 10.1.1.1 is the source IP address and ADCDC is the userid used.

By default this file is http_access.log in the …/logs directory.

See here for the format of the logFormat statement.

After tailoring

I did more packaging the content into include .xml files.

After my “repackaging” my server.xml file looked like

<?xml version="1.0" encoding="UTF-8"?> 
<server description="new server"> 
    <include location="${server.config.dir}/applicationMonitor"/> 
    <include location="${server.config.dir}/logging.xml"/> 
    <include location="${server.config.dir}/httpEndpoint.xml"/> 
    <include location="${server.config.dir}/cors.xml"/> 
    <include location="${server.config.dir}/config.xml"/> 
    <include location="${server.config.dir}/zosconect.xml"/>                                                                              
</server>

You can use ispf option 3.17 to list the files in the servers’s directory.

Once you have configured the server you can use the admin centre to manage the files.  This will require

<remoteFileAccess>  
    <writeDir>${server.config.dir}</writedir>
</remoteFileAccess>

Finally ! You can start the server.

A practical path to installing Liberty and z/OS Connect servers – 4 start the server

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Starting the server

Using the definitions above I started the server, using the default configuration directory, and used jobname=default.

s baqstart,parms=’default’,jobname=default

On my z/OS system, running on zPDT on Linux, it takes about 5 minutes to start.

During start up, check that the parameter have been accepted – with the correct case.

If you get configuration errors, fix them and use the

f ...zcon,refresh

command to get the server to pickup the changes. Note. Not all changes are picked up by the refresh command, and you have to stop and restart the server to pick up changes.

There is output in the STDOUT and possibly in STDERR, also in the /var/zosconnect/default/logs/messages.log

STDOUT will have messages like

[AUDIT] CWWKG0028A: Processing included configuration resource: 
/var/zosconnect/servers/default/httpEndpoint.xml

which tells you which configuration files you are using.

If you get any syntax errors you should fix them. The command f …,zcon,refresh  may cause it to reread the definitions.

The second +  start of server start  after  IPL may be faster than the first if Java has cached some of the executables.

Check for messages

Authorized service group … is available.

Which tells you that the Angel service is enabled.

Web application available (default_host): http://10.1.3.10:9080/

check this is what you expect. If the port is not valid check the httpEndoint.xml. You do not get an entry for the TLS port, because TLS is not enabled.

If you use a web browser to logon to the URL http:10.1.3.10:9081 (note:not hppts) you will just get a blank screen. If you look in logs/http_access.log you will see a code 302. In this case, 302 means you were trying to use an http session to a session that requires https.

If you use other URLs you may get a message saying you need to use https.

If you use z/OS explorer and use the http:…9081 above you will get “302, Found”. This means you were trying to use an http session to a session that requires https.

You have successfully created a z/OS connect server, and used it – even though you could not do much.

To stop the server use the z/OS P … command.

Most of the facilities in z/OS Connect require TLS, so you cannot do much until you have configured that.

A practical path to installing Liberty and z/OS Connect servers – 5 enable logon security

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Enable logon security

To enable security checking you need  to use the zosconnect_zosConnectManager tag.

See here for information, and here for the tag  syntax.

<zosconnect_zosConnectManager 
     requireAuth="true"  
     requireSecure="true" 
     setUTF8ResponseEncoding="true"/>

add to the zosconnect.xml or server.xml file.

You cannot do much with this enabled as you need TLS

Add in the SAF repository to use z/OS security services.

Within the server directory create a file called saf.xml. (I always used ISPF 3.17, and use cc…cc create.. to create a new file.

<server> 
                                                                                
  <featureManager> 
     <feature>transportSecurity-1.0</feature>   
     <feature>zosSecurity-1.0</feature> 
     <feature>appSecurity-2.0</feature> 
  </featureManager> 
  <authentication id="Basic" cacheEnabled="false" /> 
  <safAuthorization id="saf-authorization"  racRouteLog="ASIS" />                                                                                 

  <safRegistry id="saf" realm="myrealm"/> 
  <safCredentials profilePrefix="ZZZZDFLT" 
     suppressAuthFailureMessages="false" 
     unauthenticatedUser="WSGUEST"/> 
</server>

Where you need to specify the profile prefix (for example ZZZBFLT), and define the security profile.

I do not think the realm is used for SAF processing.

If you are planning on using a new profilePrefix  (YYYYDFLT) you will need to create several profiles.

RDEFINE APPL YYYYDFLT  UACC(NONE) 
PERMIT YYYYDFLT  CLASS(APPL) ACCESS(READ) ID(group1,group2)
SETROPTS raclist(APPL) refresh

RDEFINE server BBG.SECPFX.YYYYDFLT uacc(NONE)
permit  BBG.SECPFX.YYYYDFLT class(server) access(read) id(start1)
setropts raclist(server) refresh

rdefine ejbrole YYYYDFLT.zos.connect.access.roles.zosConnectAccess 
   uacc(none)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
   class(ejbrole) 
   id(Adcdc) access(READ)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
    class(ejbrole) 
    id(group1,group2) access(READ)
setropts raclist(ejbrole) refresh

These profiles are only accessed once someone tries to use the server – not at startup, so try logging on using the https port.

Every one using the server needs read access to YYYYDFLT CLASS(APPL).

As an interim step towards getting your server up and secure, enable userid and password authentication.

<webAppSecurity overrideHttpAuthMethod="BASIC"/> 
<webAppSecurity allowFailOverToBasicAuth="true"/> 

overrideHttpAuthMethod=”BASIC” says used userid and password – and not client certificates.

allowFailOverToBasicAuth=”true” says if the client certificate logon fails – fail over to basic, userid and password.

Once you have configured the server, you can remove the overrideHttpAuthMethod tag, so client certificates are used.  Once these are working remove the allowFailOverToBasicAuth, to remove the userid and password option.  With these both removed, people will have to use client certificates to authenticate.

A practical path to installing Liberty and z/OS Connect servers – 6 Enabling TLS

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Configuring TLS

  1. You can configure the server to creates a keystore file on its first use. This creates a self signed certificate. This is good enough to provide encryption of the traffic. Certificates sent from the client are ignored as the trust store does not have the Certificate Authority certificate to validate them.
  2. You can use your site’s keystore and trust store. The server can use them to process certificate sent from the client for authentication.

Decide how you want to authenticate

Most of the functions require an https connection. This will require a keystore.

You can decide if

  1. The server uses the client’s certificate for authentication,
    1. if that does not work then use userid and password
    2. if that does not work, then fail the request; there is no fall back to userid and password.
  2. The server does not use the clients certificate.
    1. You can configure that userid and password will used for authentication
    2. There is no authentication

Have the server create a keystore.

You can get Liberty to create a keystore for you. This creates a self signed certificate and is used to encrypt the traffic between client and server. This is a good start, while you validate the set up, but is not a good long term solution.

Create keystore.xml with

<server>
<keyStore id="defaultKeyStore" password="${keystore_password}" /> 

<ssl clientAuthentication="false" 
    clientAuthenticationSupported="false" 
    keyStoreRef="defaultKeyStore" 
    id="defaultSSLSettings" 
    sslProtocol="TLSv1.2" 
/> 
</server>

Add to the bottom of the server.xml file

 <include location="${server.config.dir}/keystore.xml"/>

If you have keyStore id=”defaultKeyStore”, (it must be defaultKeyStore) and do not have a keystore defined, the the server will create the keystore in the default location (${server.output.dir}/resources/security/key.p12) with the password taken from the server.env file.  See here.

Restart the server.

I got the messages

CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started 
and is now listening for requests on host 10.1.3.10  
(IPv4: 10.1.3.10) port 9443.

Showing TLS was active, and listening on the 9443 port.

If the keystore was created, you will get messages like

[AUDIT   ] CWPKI0803A: SSL certificate created in 87.578 seconds. 
SSL key file: /var/zosconnect/servers/d3/resources/security/key.p12 
[INFO    ] Successfully loaded default keystore: 
/var/zosconnect/servers/d3/resources/security/key.p12 of type: PKCS12

The certificate has a problem (a bug). It has been generated with CN:localhost, O:ibm: ou:d3 where d3 is the server name. The Subject Alternative Name (SAN) is DNS:localhost. It should have a SAN of the server’s IP address (10.3.1.10 in my case).

Clients check the SAN and compare it with the server’s IP address.

  1. Chrome complain. “Your connection is not private NET:ERROR_CERT_AUTHORITY_INVALID”, and the option to accept it
  2. Firefox gives “Warning: Potential Security Risk Ahead”, and the option to accept it.
  3. Z/OS explorer gives a Server certificate alert pop up, saying “Host:10.1.3.10 does not match certificate:localhost” and gives two buttons Decline or Accept.
  4. With curl I got SSL_ERROR_SYSCALL.

You can accept it, and use it until you have your own keystores set up. You can also reset this decision.

Using a RACF keyring as the keystore.

You can use a file based keystore or a RACF keying.  Below are the definitions for my RACF keyrings. The started task userid is START1. The keystore (containing the private key for the server is keyring START1/KEY. The server should use key ZZZZ.

The trust store, containing the Certificate Authority certificates and any self signed certificates from clients, is START/TRUST.

The <ssl.. /> points to the different keystores, so it makes sense to keep all these definitions in one file.  You may already have a file of these definitions which you can use from another Liberty server.

<server>

<sslDefault sslRef="defaultSSLSettings"/> 
<ssl clientAuthentication="true" 
    clientAuthenticationSupported="true" 
    id="defaultSSLSettings" keyStoreRef="racfKeyStore"  
    serverKeyAlias="ZZZZ" 
    sslProtocol="TLSv1.2" 
    trustStoreRef="racfTrustStore"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfKeyStore" 
     location="safkeyring://START1/KEY" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfTrustStore" 
     location="safkeyring://START1/TRUST" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/>                                                                                                                  
</server>

This sets clientAuthentication=”true” and clientAuthenticationSupported=”true”

Specify if you want to use a client certificate for authentication

If you specify clientAuthenticationSupported=”true”… the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.

The default keystore will not be able to validate any certificates sent from the client. When connecting to Chrome with certificates set up, I got an FFDC and messages

  • [INFO ] FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is: java.security.cert.CertPathValidatorException:
  • [ERROR ] CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired.

If you specify clientAuthentication=”false” (the default) the server does not request that a client send a certificate during the handshake.

If you specify <webAppSecurity allowFailOverToBasicAuth=”true” />  the client certificate connection is not used or it fails,

  1. if  you specify<webAppSecurity allowFailOverToBasicAuth=”true” /> the user will be prompted for userid and password
  2. If you specify <webAppSecurity allowFailOverToBasicAuth= false > or not specified, the connection will fail.

If a userid and password can be used, the first time a browser uses the server it will be prompted for userid and password. As part of the handshake, the LTPA2 cookie is sent from the server. This has the userid and password encrypted within it. If you close down the browser and restart it (not just restart it from within the browser) you will be prompted again for userid and password. You can also be prompted for userid and password once the LPTA cookie has expired.

If you are using z/OS explorer and get a code 401, unauthorised, you may be using a certificate credential ( format userid@CertificateAuthority(CommonName)) rather than a userid and password with format of just the userid eg COLIN. Use “Set Credentials” to change credentials.

You can see what userid is being used for the requests, from the …/logs/http_access.log file.

To make it even more complex you can have different keystores for different connections or ports.  See here. But I would not try that just yet.

Map client certificates to a SAF userid

If you are using certificate authentication you will need to map the certificate to a userid using the RACDCERT MAP command.

Testing it

If the server starts successfully you can use a web browser with URL

  http:/10.1.3.10:9443/zosConnect/api-docs

and it should display json data.

If you get “Context Root Not Found” or code 404 you should wait and retry, as the https processing code is active, but the code to process the requests is not yet active.

Review the contents of …/servers/…/logs/http_access.log to see the request being issued and the http completion code.

If you have problems connecting clients over TLS add -Djavax.net.debug=ssl:handshake to the jvm.options file and restart the server.

If you connect to the z/OS Explorer, and logon to the z/OS Connect EE Server, you should have a folder for APIs and Services – which may have no elements.

A practical path to installing Liberty and z/OS Connect servers – 7 adding apis and services

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Adding APIs and Services

For services you have to include a feature. For example with mq you need

 <feature>zosconnect:mqService-1.0</feature>

These are listed here.   You need the CICS service to be able to use CICS applications etc.

You also need to copy the files application (*.aar) files into the apis directory, and the service(*.sar) files into the services directory.

There are several ways of doing this.

You can do this with the uss cp command. The command below copies an MQ one – you can use others

Find what applications and services are available

find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.aar
find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.sar   

copy the files to the servers directory

cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/apis/*
  /var/zosc*/ser*/default/re*/z*/ap*                                                                                   
cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/ser*/* 
  /var/zosc*/ser*/default/re*/z*/se*                                                                                        

Use a REST request or use the zOS explorer.

Once you have done this either restart the server or use the operator command

f ...zcon,refresh

and the application and services should be available

You should get in the message.log on the STDOUT (if you have configured it to print information messages – see earlier posts).

BAQR7000I: z/OS Connect EE API archive file stockmanager installed successfully.
BAQR7043I: z/OS Connect EE service archive stockQuery installed successfully.

Using a web browser, or curl with the URL https://10.1.3.10:9443/zosConnect/services gave me

{"zosConnectServices":
  [
    {"ServiceName":"stockQuery",
     "ServiceDescription":"A stock query service based on IBM MQ.",
     "ServiceProvider":"IBM MQ for z/OS",
     "ServiceURL": "https://10.1.3.10:9443/zosConnect/services/stockQuery"
      }
  ]
}

URL https://10.1.3.10:9443/zosConnect/apis gave me

{"apis":    
  [
    {"name":"stockmanager","version":"1.0.0","description":"",
     "adminUrl":"https://10.1.3.10:9443/zosConnect/apis/stockmanager"
     }
   ]
}

You can invoke the service with https://10.1.3.10:9443/zosConnect/services/stockQuery and get details of the stockQuery service.

Once it all works…

I played with the curl interface to deploy a service.

  • I copied the stockQuery.sar file down to linux in binary
  • I could use jar -tvf stockQuery.sar to display the contents
  • I used curl –insecure -v –header Content-Type:application/zip -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem -X post –data-binary @/home/zPDT/stock2.sar https://10.1.3.10:9443/zosConnect/services
    • Note:  If I used –data-binary “@/home/zPDT/stock2.sarin quotes I got
      • HTTP/1.1 415 Unsupported Media Type
      • “errorMessage” : “BAQR0418W: An unsupported media type of application/x-www-form-urlencoded was specified.”}
    • because the string was sent up – not the file.  I used the –verbose option to see the number of bytes sent.
    • I also got these messages if the file was not found.  It sends the string instead!

A practical path to installing Liberty and z/OS Connect servers – 8 protecting APIs and services

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protect the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Protect the service and APIs

z/OS connect provides interceptors to allow the product function to be extended. These are like exits in other program products.

z/OS Connect provides (at least)  2 interceptors

  1. For authorisation checks, to see if a userid is allowed to perform an operation.
  2. Creating SMF records.

You can also write your own interceptors, for example for data validation, or for collecting statistics.

You can configure APIs and services to have a list of interceptors. One service can have authorisation and SMF records, another service can just have authorisation

You create a list like

<!-- Interceptor list configuration -->
<!-- this refers to the configuration elements following -->
<zosconnect_zosConnectInterceptors 
   id="interceptorList1" 
   interceptorRef="auditInterceptor,zosConnectAuthorizationInterceptor"
/>

<!-- Audit interceptor configuration -->
<zosconnect_auditInterceptor 
   id="auditInterceptor" 
   sequence="1"    
   apiProviderSmfVersion="2"
/>
<!-- Authorisation checking --> 
<zosconnect_authorizationInterceptor 
    id="zosConnectAuthorizationInterceptor"
/> 

To protect the server, and control the global roles, have you need to use the following where you provide lists of group names such as SYS1.

 <zosconnect_zosConnectManager 
     globalInterceptorsRef="interceptorList1" 
     globalAdminGroup="SYS1,SYSADMIN" 
     globalInvokeGroup="SYS1" 
     globalOperationsGroup="SYS1" 
     globalReaderGroup="SYS1" 
       /> 
<!-- "interceptorList1" above points to …  -->
 <zosconnect_zosConnectInterceptors 
     id="interceptorList1" 
     interceptorRef="IR1,..."/>

<!--  zosConnectAuthorizationInterceptor is defined    -->
 <zosconnect_authorizationInterceptor 
     id="IR1"/>

This shows the global security definitions. The globalInterceptorsRef=”interceptorList1″ points to the <zosconnect_zosConnectInterceptors .. which in turn points to the <zosconnect_authorizationInterceptor . There is a program or interceptor zosConnectAuthorizationInterceptor which does the actual checking of userid and roles.

With this set of definitions when I try to query the service using an unauthorised userid, I got

{"errorMessage":"BAQR0435W: The zosConnectAuthorization interceptor 
  encountered an error while processing a request. ",
"errorDetails":"BAQR0409W: User ADCDC is not authorized to 
  perform the request."}

I changed the definitions to globalReaderGroup=”TEST” , refreshed the configuration, and the request worked.

You can make API security more specific

 <zosconnect_zosConnectAPIs> 
   <zosConnectAPI name="stockmanager" 
     adminGroup="SYS1" 
     invokeGroup="TEST" 
     operationsGroup="TEST" 
    readerGroup="SYS1" 
    /> 
 </zosconnect_zosConnectAPIs>

and make the service security more specific.

<zosconnect_services> 
   <service name="stockQuery" 
     serviceDescription="stockQueryServiceDescriptionColin" 
     id="stockQueryService" 
     adminGroup="SYS1,TEST2" 
     invokeGroup="TES2" 
     operationsGroup="SYS1" 
    readerGroup="SYS1,TEST2" 
    /> 
</zosconnect_services> 

If you use the swagger to try it – and get the json data with

response body no content
response code 0
response header { “error”: no response from server}

This is what Swagger UI displays when a request fails due to a security issue such as an untrusted self-signed cert, invalid cert, or bad username:password.

A practical path to installing Liberty and z/OS Connect servers – 9 collecting monitoring data

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Monitoring data

You can collect SMF data and/or Http audit data to get reports on the performance and usage of your system.

There are some queries – you can issue via a REST query – but you do not get much data back.

http_access.log

If you have configured <httpEndpoint   ..  accessLoggingRef=…>  you can collect audit information for traffic through that end point.  If you have more than one httpEndpoint, for example with a different port, you can collect different information, or log it to a different file.

The information you can log (see here for a full description) includes

  • the client IP address
  • the userid
  • date time
  • the service requested
  • the response code
  • bytes sent and received
  • the response time ( in seconds, milliseconds, or microseconds)

You can include delimiters (for example quotes around a string, or !.. !)  in the output to simplify post processing.

If you have high throughout, this solution may not scale, and SMF may be a better solution.

Collecting SMF records

You can collect SMF 120 records from the Liberty base, and SMF 123 records from z/OS connect.

To collect SMF 120 records you need to add

<featureManager> 
    <feature>zosRequestLogging-1.0</feature> 
</featureManager>

to your configuration.

SMF 123 records are produced by another interceptor (exit). You need to define it, and add it to the list of global, API or service list of interceptors.

Configure the auditInterceptor

<zosconnect_auditInterceptor id="auditInterceptor" 
   sequence="1" 
   apiProviderSmfVersion="2"/>

and add it to the list of the intereptors

<zosconnect_zosConnectInterceptors 
    id="interceptorList1" 
    interceptorRef="zosConnectAuthorizationInterceptor,auditInterceptor"
/>

For both record types, the server started task needs access to the BPX.SMF class.

PERMIT BPX.SMF CLASS(FACILITY) ACCESS(READ) ID(USERID)
setropts raclist(facility) refresh

If the server does not have this permission it will get an FFDC with

Stack Dump = java.io.IOException: Failed to write SMF record, __smf_record errno/errno2 return code 139

Processing SMF 120. You can download SMF Browser for WebSphere Application Server for z/Osfrom

This is a java “formatter” here which provides just a dump of the records, and so is not very usable.

I wrote a formatter for this to summarise key information ( and ignore irrelevant stuff).   I’ll put this up on github when Ive got it documented.

Some of the interesting data is

  • Request start and stop time for example 2020/09/26 16:35:42.977709, from which you can calculate request duration
  • CPU for the request
  • The userid
  • The URI /zosConnect/services/stockQuery
  • TCPIP Origin and port 10.1.1.1 (33546) into the server port (9443)
  • Sysplex, LPAR, Server name, Server job number, level,

I took the data and accumulated it, so I could see which requests used all of the CPU, and report it by hour, and userid.

Processing SMF 123.

z/OS connect provides a sample C program, and JCL to compile it.   See here.

The SMF 123 records are written when the z/OS Connect server shuts down, or when the SMF buffer is full, so there is a risk that data from today, is not produced until tomorrow because there was no activity overnight.

I typically got about 20 services/APIs per SMF record.

Combing the records

I could not see how to correlate the SMF 123 and the SMF 120 records.   This would be useful to get the CPU used by each API or service.

Rest request

This page describes how to get REST statistics.  For example

https://10.1.3.10:9443/zosConnect/operations/getStatistics

This returned

{"zosConnectStatistics":
  [
   {"stockQuery":
     {
       "ServiceProvider":"IBM MQ for z\/OS",
       "InvokeRequestCount":21,
       "TimeOfRegistrationWithZosConnect":
       "2020-10-01 14:52:26:049 BST",
       "ServiceStatistics":{}
    }
   }
  ]
}

With nothing in the ServiceStatistics{}.

You can ask for a specific service https://10.1.3.10:9443/zosConnect/operations/getStatistics?service=stockQuery.  You get the same data back as above.

I could not find how to get information on APIs.

You can get real time statistics data see here.

I had

<zosconnect_zosConnectManager
globalInterceptorsRef=”interceptorList1″
globalAdminGroup=”TEST”
globalInvokeGroup=”SYS1″
globalOperationsGroup=”TEST”
globalReaderGroup=”TEST”
/>
<zosconnect_authorizationInterceptor id=”zosConnectAuthorizationInterceptor”/>
<zosconnect_auditInterceptor id=”zoscauditInterceptor” sequence=”1″ apiProviderSmfVersion=”2″/>
<auditInterceptor id=”auditInterceptor” sequence=”1″/>
<zosconnect_zosConnectInterceptors
id=“interceptorList1”
interceptorRef=”zosConnectAuthorizationInterceptor,auditInterceptor,zoscauditInterceptor “/>