Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Classify the services with WLM to give them the right priority

You can configure an API or service, for example https://10.1.3.10:9443/stockmanager/stock/items/999999, so it gets an appropriate service class in WLM, for example high priority.

You can use RMF to report on service classes to see the response time profile, and if the service class is meeting its performance goals.

See here for a good article.

This page gives the example

<wlmClassification>
  <httpClassification transactionClass="CLASS001" 
      host="127.0.0.1" 
      port="9080" 
      method="GET"
      resource="/testResource" />
</wlmClassification>

You can classify the traffic depending on the IP address and port of the server, as well as the resource name.

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Copy the PDS from the install system to where they will run.

These files can be sent using FTP or XMIT.

Copy the product ZFS to where it will be used.

You can use DFDSS to dump the ZFS and move it to the SYSPLEX where it will be used.

1. Mount the file system read/write for example /usr/lpp/IBM/zosconnect/v3r0beta
2. Perform Setting up the product extensions directory.  You need to do this on each LPAR, as it set up files in /var/zosconnect
3. Mount the file system READ
4. Change the BPXPRMxx parmlib member so the file sytsem is mounted READ it at IPL
5. You may want to create an alias /usr/zosc to the product executables.
   • ln -s /usr/lpp/IBM/zosconnect/v3r0beta/bin /usr/zosc

Create the angel started task if needed.

If you will be using an existing Angel you will need to set up RACF permissions.

Follow the instructions the z/OS Connect  documentation

• If you need a named Angel, specify the value in the NAME=value.  You can specify NAME when you start the task.
• You do not need to classify this in WLM as it uses little CPU once it has been started.

Create the file system for the server

1. See Creating the z/OS Connect EE shared directory. You will need to know the path for server’s home directory. The path /var/zosconnect can only be used on the LPAR.   The files cannot be shared.
2. If you need to allocate a file system.
   1. Allocate a ZFS. If the initial allocation of the ZFS is too small, then the ZFS will grow on demand, or using the zfsadm grow command to expand it.
   2. Mount the file system over the directory.
   3. Change the BPXPRMxx parmlib member so the file system is mounted RW at IPL.

Create the Z/OS Connect started task

See Creating a z/OS Connect Enterprise Edition server.

Using /usr/zosc alias defined above

export  WLP_USER_DIR=/var/zosconnect
/usr/zosc/zosconnect create default --template=zosconnect:default

If you need to start again, use the following to remove the server.

rm -r  /var/zosconnect/servers/default

1. Copy BAQSTRT to the SYS1.PROCLIB concatenation, to the name you are going to use, BAQSTRT, BAQSTRT1 etc
2. When you edit the member, use caps off to use lower case text in the member.
3.  If you will be using MQ, add the MQ libraries to STEPLIB

   //STEPLIB   DD DSN=CSQ913.SCSQANLE,DISP=SHR 
   //          DD DSN=CSQ913.SCSQAUTH,DISP=SHR

4. You may find it easier to remove the _CEE_RUNOPTS= and add

   //CEEOPTS DD DISP=SHR,DSN=...(CEEOPTS)

   to the JCL. This makes it easier to add CEE options, and specify storage heap and pool sizes.

5. In my CEEOPTS I use RPTSTG(ON), to report the storage pool usage. This provides useful information which may be useful when investigating performance problems.
6. Despite what the Liberty documentation says you do need

   • • JVM_OPTIONS=<Optional JVM parameters>

   • for example

     • JVM_OPTIONS=-Xoptionsfile=/var/zosconnect/servers/default/jvm.options

7. You may want a JVM specific to a server, as you may want to set some service specific options such as TLS trace.
8. You may want to specify a default server name in the JCL parms. You can specify a different one when z/OS Connect is started.
9. Specify WLP_USER_DIR for your server’s directory. You could make this as a parameter to the started task JCL if you want to provide isolation for the servers, and give each server its own file system.

Create the JVM options file

Go to the server’s directory.

cd /var/zosconnect/servers/default

use oedit jvm.options to create an EBCDIC file.

add

-Dcom.ibm.ws.zos.core.angelRequired=true 
#-Dcom.ibm.ws.zos.core.angelName=MYZANGL

The -Dcom.ibm.ws.zos.core.angelRequired=true option stops the server from starting if the angel process is not running. You get message:

CWWKB0116I: This server is not registered with an angel process even though it is configured to require registration with an angel process. This server is attempting to stop.

The option -Dcom.ibm.ws.zos.core.angelName=MYZANGL allows you to specify a non default angel process name. The # in front of the statement makes it into a comment.

add the following, if required, with your time zone.

-Duser.timezone=Europe/London

Override TCPIP

If you are using a non standard TCPIP, edit server.env and add

_BPXK_SETIBMOPT_TRANSPORT=TCPIP2

Tailor the server.xml file

It was easy to get my configuration options mixed up, and hard to compare two server configurations. I found it much easier to manage my configuration by putting the components into their own xml file. This also makes having shared components much easier to implement- as you just include the name of the shared file.

The configuration files have to be in ASCII, and you have to take care ensure the files are in the correct code page.

There are at least three ways of creating the .xml files (I use the third option through ISPF 3.17)

1. using omvs commands to copy and edit existing ASCII files
2. use omvs commands to create ASCII files
3. use ispf option 3.17

1 Using omvs commands to copy an existing ascii file,  then edit it it

For example

cp server.xml httpEndpoint.xml

edit the httpEndpoint.xml to keep the <server> ,httpEndpoint information, </server> and delete the rest.  The file should look like

<server description="new server"> 
    <httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" /> 
</server>

then edit the server.xml and replace

<httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" />

with

<include location="${server.config.dir}/httpEndpoint.xml"/> 

Note: if you use the edit command source ? It should say ASCII already set.

2 Use OMVS to created and edit files

By default, if you edit a file in OMVS it becomes an EBCDIC file.

If you create a file without copying it, you need to use

touch httpEndpoint.xml 
chtag -t -c ISO8859-1 httpEndpoint.xml     
oedit httpEndpoint.xml

Make the same changes as above.

3 Use ispf option 3.17

If you use ISPF 3.17 (z/OS UNIX Directory List) you have more flexibility with editing files.

Edit server.xml

use the line commands mm…mm and the primary command create to create a new member, give the new name when prompted, for example httpEndpoint.xml

Replace the lines you copied out with

	<include location="${server.config.dir}/httpEndpoint.xml"/>

and edit the file you just created to put <server> … </server> around the content.

Add more logging to the JCL’s STDOUT.

The default logging to the jobs STDOUT is only audit records. I found it much easier to include other levels of information so I did not have to keep looking in the message.log file. Note: the STDOUT content does not have time stamps so you do not know when the output was produced- so it is easier to use – but has less information!

I created logging.xml with the following content.

<server> 
 <logging 
 consoleLogLevel="INFO" 
 /> 

 <!-- this is a comment where I store trace information
   traceFormat="BASIC" traceSpecification="*=info:
       com.ibm.ws.security.*=fine:zos.native.03=fine" 
  end comment   -->  
</server>

I added

<include location="${server.config.dir}/logging.xml"/> 

to the server.xml file.

The consolLogLevel=”INFO” writes additional information to the STDOUT file, than the default AUDIT value which just outputs AUDIT records.

Define the http address and ports

Edit the httpEndpoint definitions

    <httpEndpoint id="defaultHttpEndpoint" 
                  accessLoggingRef="ALR" 
                  host="*" 
                  httpPort="9081" 
                  httpsPort="9443" /> 
    <httpAccessLogging 
       enabled="true" 
       id="ALR" 
       logFormat="%{c}a! %h %u %{t}W %q %r %s %b" 
   /> 

Check

• host matches the TCPIP value to use. With my VIPA I specify the VIPA address host=”10.1.3.10”
• httpPort – if you already have a Liberty instance running port 9081 will be in use by another server
• httpsPort – if you already have a Liberty instance running port 9443 will be in use by another server

You will need the port values when you use the server.

The httpAccessLogging statement pointed to by accessLoggingRef, will create a log file with every attempt to use the server.

Example output

10.1.1.1! 10.1.1.1 ADCDC {12/Sep/2020:16:07:17 +0000} - 
GET /zosConnect/apis/stockmanager HTTP/1.1 200 296

Where 10.1.1.1 is the source IP address and ADCDC is the userid used.

By default this file is http_access.log in the …/logs directory.

See here for the format of the logFormat statement.

After tailoring

I did more packaging the content into include .xml files.

After my “repackaging” my server.xml file looked like

<?xml version="1.0" encoding="UTF-8"?> 
<server description="new server"> 
    <include location="${server.config.dir}/applicationMonitor"/> 
    <include location="${server.config.dir}/logging.xml"/> 
    <include location="${server.config.dir}/httpEndpoint.xml"/> 
    <include location="${server.config.dir}/cors.xml"/> 
    <include location="${server.config.dir}/config.xml"/> 
    <include location="${server.config.dir}/zosconect.xml"/>                                                                              
</server>

You can use ispf option 3.17 to list the files in the servers’s directory.

Once you have configured the server you can use the admin centre to manage the files.  This will require

<remoteFileAccess>  
    <writeDir>${server.config.dir}</writedir>
</remoteFileAccess>

Finally ! You can start the server.

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Starting the server

Using the definitions above I started the server, using the default configuration directory, and used jobname=default.

s baqstart,parms=’default’,jobname=default

On my z/OS system, running on zPDT on Linux, it takes about 5 minutes to start.

During start up, check that the parameter have been accepted – with the correct case.

If you get configuration errors, fix them and use the

f ...zcon,refresh

command to get the server to pickup the changes. Note. Not all changes are picked up by the refresh command, and you have to stop and restart the server to pick up changes.

There is output in the STDOUT and possibly in STDERR, also in the /var/zosconnect/default/logs/messages.log

STDOUT will have messages like

[AUDIT] CWWKG0028A: Processing included configuration resource: 
/var/zosconnect/servers/default/httpEndpoint.xml

which tells you which configuration files you are using.

If you get any syntax errors you should fix them. The command f …,zcon,refresh  may cause it to reread the definitions.

The second +  start of server start  after  IPL may be faster than the first if Java has cached some of the executables.

Check for messages

Authorized service group … is available.

Which tells you that the Angel service is enabled.

Web application available (default_host): http://10.1.3.10:9080/

check this is what you expect. If the port is not valid check the httpEndoint.xml. You do not get an entry for the TLS port, because TLS is not enabled.

If you use a web browser to logon to the URL http:10.1.3.10:9081 (note:not https) you will just get a blank screen. If you look in logs/http_access.log you will see a code 302. In this case, 302 means you were trying to use an http session to a session that requires https.

If you use other URLs you may get a message saying you need to use https.

If you use z/OS explorer and use the http:…9081 above you will get “302, Found”. This means you were trying to use an http session to a session that requires https.

You have successfully created a z/OS connect server, and used it – even though you could not do much.

To stop the server use the z/OS P … command.

Most of the facilities in z/OS Connect require TLS, so you cannot do much until you have configured that.

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Enable logon security

To enable security checking you need  to use the zosconnect_zosConnectManager tag.

See here for information, and here for the tag  syntax.

<zosconnect_zosConnectManager 
     requireAuth="true"  
     requireSecure="true" 
     setUTF8ResponseEncoding="true"/>

add to the zosconnect.xml or server.xml file.

You cannot do much with this enabled as you need TLS

Add in the SAF repository to use z/OS security services.

Within the server directory create a file called saf.xml. (I always used ISPF 3.17, and use cc…cc create.. to create a new file.

<server> 
                                                                                
  <featureManager> 
     <feature>transportSecurity-1.0</feature>   
     <feature>zosSecurity-1.0</feature> 
     <feature>appSecurity-2.0</feature> 
  </featureManager> 
  <authentication id="Basic" cacheEnabled="false" /> 
  <safAuthorization id="saf-authorization"  racRouteLog="ASIS" />                                                                                 

  <safRegistry id="saf" realm="myrealm"/> 
  <safCredentials profilePrefix="ZZZZDFLT" 
     suppressAuthFailureMessages="false" 
     unauthenticatedUser="WSGUEST"/> 
</server>

Where you need to specify the profile prefix (for example ZZZBFLT), and define the security profile.

I do not think the realm is used for SAF processing.

If you are planning on using a new profilePrefix  (YYYYDFLT) you will need to create several profiles.

RDEFINE APPL YYYYDFLT  UACC(NONE) 
PERMIT YYYYDFLT  CLASS(APPL) ACCESS(READ) ID(group1,group2)
SETROPTS raclist(APPL) refresh

RDEFINE server BBG.SECPFX.YYYYDFLT uacc(NONE)
permit  BBG.SECPFX.YYYYDFLT class(server) access(read) id(start1)
setropts raclist(server) refresh

rdefine ejbrole YYYYDFLT.zos.connect.access.roles.zosConnectAccess 
   uacc(none)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
   class(ejbrole) 
   id(Adcdc) access(READ)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
    class(ejbrole) 
    id(group1,group2) access(READ)
setropts raclist(ejbrole) refresh

These profiles are only accessed once someone tries to use the server – not at startup, so try logging on using the https port.

Every one using the server needs read access to YYYYDFLT CLASS(APPL).

As an interim step towards getting your server up and secure, enable userid and password authentication.

<webAppSecurity overrideHttpAuthMethod="BASIC"/> 
<webAppSecurity allowFailOverToBasicAuth="true"/> 

overrideHttpAuthMethod=”BASIC” says used userid and password – and not client certificates.

allowFailOverToBasicAuth=”true” says if the client certificate logon fails – fail over to basic, userid and password.

Once you have configured the server, you can remove the overrideHttpAuthMethod tag, so client certificates are used.  Once these are working remove the allowFailOverToBasicAuth, to remove the userid and password option.  With these both removed, people will have to use client certificates to authenticate.

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Adding APIs and Services

For services you have to include a feature. For example with mq you need

 <feature>zosconnect:mqService-1.0</feature>

These are listed here.   You need the CICS service to be able to use CICS applications etc.

You also need to copy the files application (*.aar) files into the apis directory, and the service(*.sar) files into the services directory.

There are several ways of doing this.

You can do this with the uss cp command. The command below copies an MQ one – you can use others

Find what applications and services are available

find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.aar
find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.sar   

copy the files to the servers directory

cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/apis/*
  /var/zosc*/ser*/default/re*/z*/ap*                                                                                   
cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/ser*/* 
  /var/zosc*/ser*/default/re*/z*/se*                                                                                        

Use a REST request or use the zOS explorer.

Once you have done this either restart the server or use the operator command

f ...zcon,refresh

and the application and services should be available

You should get in the message.log on the STDOUT (if you have configured it to print information messages – see earlier posts).

BAQR7000I: z/OS Connect EE API archive file stockmanager installed successfully.
BAQR7043I: z/OS Connect EE service archive stockQuery installed successfully.

Using a web browser, or curl with the URL https://10.1.3.10:9443/zosConnect/services gave me

{"zosConnectServices":
  [
    {"ServiceName":"stockQuery",
     "ServiceDescription":"A stock query service based on IBM MQ.",
     "ServiceProvider":"IBM MQ for z/OS",
     "ServiceURL": "https://10.1.3.10:9443/zosConnect/services/stockQuery"
      }
  ]
}

URL https://10.1.3.10:9443/zosConnect/apis gave me

{"apis":    
  [
    {"name":"stockmanager","version":"1.0.0","description":"",
     "adminUrl":"https://10.1.3.10:9443/zosConnect/apis/stockmanager"
     }
   ]
}

You can invoke the service with https://10.1.3.10:9443/zosConnect/services/stockQuery and get details of the stockQuery service.

Once it all works…

I played with the curl interface to deploy a service.

• I copied the stockQuery.sar file down to linux in binary
• I could use jar -tvf stockQuery.sar to display the contents
• I used curl –insecure -v –header Content-Type:application/zip -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem -X post –data-binary @/home/zPDT/stock2.sar https://10.1.3.10:9443/zosConnect/services
  • Note:  If I used –data-binary “@/home/zPDT/stock2.sar” in quotes I got
    • HTTP/1.1 415 Unsupported Media Type
    • “errorMessage” : “BAQR0418W: An unsupported media type of application/x-www-form-urlencoded was specified.”}
  • because the string was sent up – not the file.  I used the –verbose option to see the number of bytes sent.
  • I also got these messages if the file was not found.  It sends the string instead!

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protect the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Protect the service and APIs

z/OS connect provides interceptors to allow the product function to be extended. These are like exits in other program products.

z/OS Connect provides (at least)  2 interceptors

1. For authorisation checks, to see if a userid is allowed to perform an operation.
2. Creating SMF records.

You can also write your own interceptors, for example for data validation, or for collecting statistics.

You can configure APIs and services to have a list of interceptors. One service can have authorisation and SMF records, another service can just have authorisation

You create a list like

<!-- Interceptor list configuration -->
<!-- this refers to the configuration elements following -->
<zosconnect_zosConnectInterceptors 
   id="interceptorList1" 
   interceptorRef="auditInterceptor,zosConnectAuthorizationInterceptor"
/>

<!-- Audit interceptor configuration -->
<zosconnect_auditInterceptor 
   id="auditInterceptor" 
   sequence="1"    
   apiProviderSmfVersion="2"
/>
<!-- Authorisation checking --> 
<zosconnect_authorizationInterceptor 
    id="zosConnectAuthorizationInterceptor"
/> 

To protect the server, and control the global roles, have you need to use the following where you provide lists of group names such as SYS1.

 <zosconnect_zosConnectManager 
     globalInterceptorsRef="interceptorList1" 
     globalAdminGroup="SYS1,SYSADMIN" 
     globalInvokeGroup="SYS1" 
     globalOperationsGroup="SYS1" 
     globalReaderGroup="SYS1" 
       /> 
<!-- "interceptorList1" above points to …  -->
 <zosconnect_zosConnectInterceptors 
     id="interceptorList1" 
     interceptorRef="IR1,..."/>

<!--  zosConnectAuthorizationInterceptor is defined    -->
 <zosconnect_authorizationInterceptor 
     id="IR1"/>

This shows the global security definitions. The globalInterceptorsRef=”interceptorList1″ points to the <zosconnect_zosConnectInterceptors .. which in turn points to the <zosconnect_authorizationInterceptor . There is a program or interceptor zosConnectAuthorizationInterceptor which does the actual checking of userid and roles.

With this set of definitions when I try to query the service using an unauthorised userid, I got

{"errorMessage":"BAQR0435W: The zosConnectAuthorization interceptor 
  encountered an error while processing a request. ",
"errorDetails":"BAQR0409W: User ADCDC is not authorized to 
  perform the request."}

I changed the definitions to globalReaderGroup=”TEST” , refreshed the configuration, and the request worked.

You can make API security more specific

 <zosconnect_zosConnectAPIs> 
   <zosConnectAPI name="stockmanager" 
     adminGroup="SYS1" 
     invokeGroup="TEST" 
     operationsGroup="TEST" 
    readerGroup="SYS1" 
    /> 
 </zosconnect_zosConnectAPIs>

and make the service security more specific.

<zosconnect_services> 
   <service name="stockQuery" 
     serviceDescription="stockQueryServiceDescriptionColin" 
     id="stockQueryService" 
     adminGroup="SYS1,TEST2" 
     invokeGroup="TES2" 
     operationsGroup="SYS1" 
    readerGroup="SYS1,TEST2" 
    /> 
</zosconnect_services> 

If you use the swagger to try it – and get the json data with

response body no content
response code 0
response header { “error”: no response from server}

This is what Swagger UI displays when a request fails due to a security issue such as an untrusted self-signed cert, invalid cert, or bad username:password.

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Monitoring data

You can collect SMF data and/or Http audit data to get reports on the performance and usage of your system.

There are some queries – you can issue via a REST query – but you do not get much data back.

http_access.log

If you have configured <httpEndpoint   ..  accessLoggingRef=…>  you can collect audit information for traffic through that end point.  If you have more than one httpEndpoint, for example with a different port, you can collect different information, or log it to a different file.

The information you can log (see here for a full description) includes

• the client IP address
• the userid
• date time
• the service requested
• the response code
• bytes sent and received
• the response time ( in seconds, milliseconds, or microseconds)

You can include delimiters (for example quotes around a string, or !.. !)  in the output to simplify post processing.

If you have high throughout, this solution may not scale, and SMF may be a better solution.

Collecting SMF records

You can collect SMF 120 records from the Liberty base, and SMF 123 records from z/OS connect.

To collect SMF 120 records you need to add

<featureManager> 
    <feature>zosRequestLogging-1.0</feature> 
</featureManager>

to your configuration.

SMF 123 records are produced by another interceptor (exit). You need to define it, and add it to the list of global, API or service list of interceptors.

Configure the auditInterceptor

<zosconnect_auditInterceptor id="auditInterceptor" 
   sequence="1" 
   apiProviderSmfVersion="2"/>

and add it to the list of the intereptors

<zosconnect_zosConnectInterceptors 
    id="interceptorList1" 
    interceptorRef="zosConnectAuthorizationInterceptor,auditInterceptor"
/>

For both record types, the server started task needs access to the BPX.SMF class.

PERMIT BPX.SMF CLASS(FACILITY) ACCESS(READ) ID(USERID)
setropts raclist(facility) refresh

If the server does not have this permission it will get an FFDC with

Stack Dump = java.io.IOException: Failed to write SMF record, __smf_record errno/errno2 return code 139

Processing SMF 120. You can download SMF Browser for WebSphere Application Server for z/Osfrom

This is a java “formatter” here which provides just a dump of the records, and so is not very usable.

I wrote a formatter for this to summarise key information ( and ignore irrelevant stuff).   I’ll put this up on github when Ive got it documented.

Some of the interesting data is

• Request start and stop time for example 2020/09/26 16:35:42.977709, from which you can calculate request duration
• CPU for the request
• The userid
• The URI /zosConnect/services/stockQuery
• TCPIP Origin and port 10.1.1.1 (33546) into the server port (9443)
• Sysplex, LPAR, Server name, Server job number, level,

I took the data and accumulated it, so I could see which requests used all of the CPU, and report it by hour, and userid.

Processing SMF 123.

z/OS connect provides a sample C program, and JCL to compile it.   See here.

The SMF 123 records are written when the z/OS Connect server shuts down, or when the SMF buffer is full, so there is a risk that data from today, is not produced until tomorrow because there was no activity overnight.

I typically got about 20 services/APIs per SMF record.

Combing the records

I could not see how to correlate the SMF 123 and the SMF 120 records.   This would be useful to get the CPU used by each API or service.

Rest request

This page describes how to get REST statistics.  For example

https://10.1.3.10:9443/zosConnect/operations/getStatistics

This returned

{"zosConnectStatistics":
  [
   {"stockQuery":
     {
       "ServiceProvider":"IBM MQ for z\/OS",
       "InvokeRequestCount":21,
       "TimeOfRegistrationWithZosConnect":
       "2020-10-01 14:52:26:049 BST",
       "ServiceStatistics":{}
    }
   }
  ]
}

With nothing in the ServiceStatistics{}.

You can ask for a specific service https://10.1.3.10:9443/zosConnect/operations/getStatistics?service=stockQuery.  You get the same data back as above.

I could not find how to get information on APIs.

You can get real time statistics data see here.

I had

<zosconnect_zosConnectManager
globalInterceptorsRef=”interceptorList1″
globalAdminGroup=”TEST”
globalInvokeGroup=”SYS1″
globalOperationsGroup=”TEST”
globalReaderGroup=”TEST”
/>
<zosconnect_authorizationInterceptor id=”zosConnectAuthorizationInterceptor”/>
<zosconnect_auditInterceptor id=”zoscauditInterceptor” sequence=”1″ apiProviderSmfVersion=”2″/>
<auditInterceptor id=”auditInterceptor” sequence=”1″/>
<zosconnect_zosConnectInterceptors
id=“interceptorList1”
interceptorRef=”zosConnectAuthorizationInterceptor,auditInterceptor,zoscauditInterceptor “/>

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Planning

Summary checklist

1. Allocate HTTPS and HTTP ports
2. Decide how many Started Task procedures you need – and what to call them.
3. Decide where to install the product
4. Where to put the server’s home directory – and how much space to allocate
5. What Angel task will be used – do you need to create a task or use an existing one
6. Security
   1. Can you share the profile prefix or do you need to allocate a new one
   2. Do you need to set up a new ejbrole profiles
   3. Decide what groups can access the ejbrole profile
   4. Decide what groups can access the global roles
   5. Decide what groups can have API and Service specific roles
7. What SMF data do you want to collect
8. Do you want to use WLM to classify the priority that URLs get?

TCPIP Port

Most of the work with Liberty is done with an HTTPS port. However most sites allocate an HTTP and an HTTPS port.  The default ports, http:9081 and https:9443, may already be in use by another Liberty instance.

You can see if a port is in use by using the command

tso netstat allconn tcp tcpip ( port 9081

If the port is in use, it will report the job name.

Customising the JCL

There will be updates to the SYS1.PROCLIB concatenation, and some security definitions to be done. If you have the authority, you can make these changes yourself. If not, you will need to do some planning, and request the changes.

Where does the executable code go?

Products are usually installed in /usr/lpp file path.

If you intend to have only one version of the product installed at a time, you can create a directory /usr/lpp/IBM/zosconnect/v3r0 and mount the product file system over this directory.

If you only plan to use more than one version in parallel, you can create /usr/lpp/IBM/zosconnect/v3r0beta and mount the beta file system over it.

I found it convenient to define an alias /usr/zosc to /usr/lpp/IBM/zosconnect/v3r0beta/bin. By changing the alias I could easily switch between versions, and had less typing!

How many JCL procedures do I need to create?

There are two ways of defining multiple servers.

1. You have one JCL procedure and pass the server name as a parameter.

S BAQSTART,Parms=’server1’
S BAQSTART,Parms=’server2’

Note: If you use the z/OS command STOP BAQZSTRT then both servers will stop.

If you use the same JCL procedure for different servers you can use

S BAQSTART,Parms=’server1’,jobname=ZERVER1
S BAQSTART,Parms=’server2’jobaname=ZERVER2

and use the stop command P ZERVER1 to stop just the first one.

You can use WLM to classify ZERVER1 and ZERVER2 and give them different service classes.

2. You can use a different JCL procedure for each server.

S BAQSTRT1,parms=”server”
S BAQSTRT2,parms=”server”

You can also issue S BAQSTRT1,parms=”server”,jobname=ZERVER1

I can see no major advantage either way.  Having one started task JCL per server means more JCL to support but you can upgrade the servers one at a time.

You could also set up the procedure so you use

S BAQSTRT1,parms=”server”,WLP="/u/zosc"

Server file system.

Each server has a “home” directory. This contains

1. server configuration files – the servers only reads these files.
2. a log directory where the server writes log files, trace files, and and FDC failure events.

You may want each server to have its own file system, so if it produces a lot of output and fills up the file system, it does not impact other servers using the same file system.

You might start with one file system shared by many servers, and move to dedicated file systems before going into production.

The default file system in the zOS Connect documentation is /var/zosconnect ; this cannot be shared across LPARs. You might want to create and use /u/zosc as a shared file system, and use /u/zosc/server1 etc. The Liberty shared directory would be /u/zosc/shared.

Before you decide where you put your server’s files you need to think about what your environment could be in a years time.

If you want to have more than one server using a shared configuration, you can include files into the server.xml file. Shared files could be keystore definitions, or security definitions, and these need to be on a shared file system.

Some file systems are specific to an LPAR and not shared, (/var/ /etc/tmp, /dev), other file systems can be shared across the SYSPLEX.

Include common configuration into the server.xml file

When you include configuration files (in server.xml)  the syntax is like

<include location="/u/zosc/servers/stockManager/mq.xml"/>
<include location=”${shared.config.dir}/security.xml”/> 
<include location="${server.config.dir}/saf.xml"/> 
<include location="${COLIN}/servers/d2/jms.xml"/>
<variable name="colin2" value="/ZZZ/zosconnect/"/>
<include location="${colin2}/servers/d3/jms.xml"/>  

Where you can

• give the explicitly file path name,
• use a Liberty property ${server.config.dir} which says in the servers directory,
• use the Liberty property ${shared.config.dir} which points to a shared directory within the server’s environment.
• Use an environment variable COLIN defined as
  • //STDENV DD *
  • COLIN=COLINJCL
• Create and use, your own property – colin2
• or combinations of these.

If you get the location wrong, it is easy to change, and to move the configuration files to a new directory.

As you move changed from test through to production you may want to use the same server.xml and included files.  If so, you could set an environment variable in the JCL whose value depends on the LPAR.

How much disk space is needed?

The configuration files do not need much disk space. If you use the trace capability then the trace files can be large, and have many of them , but you can control the number and size of the logs and traces. FDC’s are also stored in the file system, and these can also be large, and you may get a lot of them. ZFS can automatically expand the file system – and your automation can respond to the ZFS message on the console to notify you that your file system is filling up.

If the JVM abends, you can get SDUMPS taken. On my machine they were taken with the HLQ of the started task (START1).

Angel task needed

You need an ANGEL task to support authorised services. You can have only one unnamed Angel per LPAR. You need to decide if your server can use this, or if your server needs its own, named Angel.

You should use the Angel at the latest service level. If servers share an Angel, and the Angel is running back level, you will get a message informing you.

You configure the Liberty instance to point to a named Angel.

Planning for security.

Liberty requires a RACF APPL profile prefix set up. The default profile prefix is BBQZDFLT. This name is used as a prefix to the RACF profile which allows users to access Liberty. For example in the EJBROLE class

BBQZDFLT.zos.connect.access.roles.zosConnectAccess

To provide isolation, and security you may want to use a different profile prefix for different groups of servers. For example you may want to isolate MQWEB from z/OS Connect, and from WebSphere Application Servers.

In summary, there are three level of security

1. A userid needs access to EJBPROF profile (above) to get access to the z/OS connect instance.
2. There is Global access, with four predefined roles. You specify a list of groups and Liberty checks to see if the userid is a member of the groups. This is not a SAF check. This checking is done in an interceptor (exit) which you specify.
3. You can specify security at the API or service level. This checking is done in an interceptor (exit) which you specify.

You will need to set up an EJBPROF profile and permit groups to connect to the server.

Once a user has access to the server, there is another layer of security with categories:

• globalAdminGroup – Identifies the users that are able to use administrative functions on all APIs, services, service endpoints and API requesters.
• globalOperationsGroup -Identifies the users that are able to perform operations such as starting, stopping or obtaining the status of all APIs, services, service endpoints and API requesters.
• globalInvokeGroup – Identifies the users that are able to invoke all APIs, services, service endpoints and API requesters.
• globalReaderGroup –Identifies the users that are able to get lists of, or information about, all APIs, services, service endpoints and API requesters, including Swagger documentation.

You can refine the security for the APIs, Services, and Service endpoints, using tags like

<zosconnect_services…

• adminGroup
• operationsGroup
• invokeGroup
• readerGroup

To be able to operate a service or API, you need to be in both globalOperationsGroup, and in the operationsGroup lists of groups.

If you have different applications within a server, you need to be careful how you set up the security profile. If someone is authorised through the global* profile  to operate service A, and you add service B, then by default the person will be allowed to operate service B. You need to define the zosconnect_services for service B, and specify the operationsGroup to restrict access to service B.

Because of this, you need to consider if you need separate default prefix for the servers to give application isolation from a security perspective.

During this planning stage you need to plan the default prefix you will be using, the groups of users for the different roles, and if you want to use both global and API/services level authorisation checks.

If you change the configuration and change the groups in the configuration, you an activate the change using the

f ….zcon,refresh

operator command.

Unauthenticated user.

When Liberty uses SAF to authenticate, it requires an Unauthenticated User which is usually “WSGUEST”. This userid can be used for all Liberty instances.

Liberty does most of its work using a https connection. If you specify some particular options, the server can set up a default keystore. This is fine while you are setting up – but not for the long term, as it does not validate certificates sent from clients.

You will need to set up a keystore to provide the server with a private certificate. You will need a trust store which contains the Certificate Authority and any client self signed certificates.   The keystores and truststores can be shared by all servers.

You can have different keystores depending on the IP address or port. See https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/rwlp_ssl_outbound_filter.html. I suggest you do not do this until you have basic TLS working.

SMF

Liberty can produce SMF 120 records. There are no good tools freely available to provide reports on usage.

Z/OS connect can produce SMF data record type 123. You will need to collect it. Some samples are provided to print out the data. There are no good tools to provide reports on usage.

Classifying request using WLM.

You can classify request to give priorities to particular services.  See here. You do not need to decide on the classification until the server is operational, and the services are available.  Essentially you configure services as a transaction class, then use WLM to classify the transaction class within the server.

<httpClassification transactionClass="TCIC" method="GET" 
resource="/catalogManager/items"/>

 

The instructions I have seen for installing products based on Liberty that seem to be written as if there would only be one server; one server on the LPAR, and one server in the whole SYSPLEX. In reality you are likely to have the “same” server running on multiple LPARS sharing configuration to provide availability, and have more than one server running on an LPAR, for example MQWEB, WAS, z/OSMF, and z/OS connect. The series of blog post below are to help you implement multiple servers, across a sysplex.

Some of the areas not adequately addressed by the IBM product documentation include

1. Sharing of definitions
2. Sharing of keystore and trust stores
3. Providing isolation, to prevent someone who has access to MQWEB from accessing Z/OS Connect.
4. How many Angel tasks do I need – can one be shared?
5. Some areas such as TLS can be hard to get working.

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

1. Overview
2. planning to help you decide what you need to create, and what options you have to choose
3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
4. starting the server
5. enable logon security and add SAF definitions
6. add keystores for TLS, and client authentication
7. adding an API and service application
8. protecting the API and service applications
9. collecting monitoring data including SMF
10. use the MQ sample
11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

I wrote the blog post  How many servers do I need? Every one know this – or no one knows this. when I was first thinking about planning my servers.