Understanding Z/OS Connect SMF 123 subtype 2 record

Introduction to the z/OS Connect SMF records

z/OS Connect can provide two types of SMF record

  1. SMF 120 subtype 11, provided by the base Liberty support. This gives information on the URL used to access Liberty, and the CPU used to perform requests. This is enabled at the Server level – so you can have records for all request, or no requests. There is one SMF record for each web server request.
  2. SMF 123 provides information about API and the service used, and the “pass through” services. It provides elapsed time of the request, and of the the “pass through” requests. It does not provide CPU usage figures. This can be configured to produce records depending on the http host and port used to access z/OS Connect. One SMF record can have data for multiple web server requests. The SMF records are produced when the SMF record is full – or the server is shut down.

The SMF 120-11 and SMF 123 records are produced independently, and there is no correlating field between them. They both have a URI field, and time stamps, so at low volumes it may be possible to correlate the SMF data.

I’ll document the fields which I think are interesting. If you think other fields are useful please let me know and I’ll update this document.

I have written an SMF formatter in C which prints out interesting data, and summarises it.

SMF 123 subtype 2 fields

  • You get the standard date and time the record was produced, and with the LPAR. You can use PGM=IFASMFDP with the following to filter which records are copied
DATE(2020282,2020282)
START(1000)
START(2359)
  • There is server version (3), system(SOW1), SYSPLEX(ADCDPLEX) and job id(STC04774) which are not very interesting
  • Server job name(SM3) is more interesting. I started the server with s baqstrt,parms=’MQTEST’,jobname=sm3
  • The config dir (/var/zosconnect/servers/MQTEST/) is boring – as is server level (open beta)
  • The HTTP code, for example 200 OK, 403 Forbidden. You may want to report requests with errors in a separate file
    1. So you know you have errors and can fix them
    2. Your statistics, such as average response time do not have dirty data in them.
  • An HTTP 1 character flag – this has always been 00 for me. I cannot find it documented.
  • The Client IP address. 10.1.1.1
  • You get userid information.
    • I used a certificate to authenticate. The DN from the certificate is not available. You only get the userid from the RACF mapping of DN to userid. This mapped userid was in the 64 byte field. The 8 byte userid field was empty for me. The lack of certificate DN, and having the userid in the wrong field feels like a couple of buglets. If you use LDAP, I think the long ID is stored in the long field, and the z/OS userid stored in the short field – so inconsistent.
  • You get the URL(URI) used /stockmanager/stock/items/999999. I treat this as a main key for processing and summarising data. You may need to process this value as the partnumber (999999) will be different for each query. You may want to have standards which say the first two elements (/stockmanager/stock) are useful for reporting. The remaining elements should be ignored when summarising records.
  • The start and stop times (2020/10/08 09:18:19.852360 and 2020/10/08 09:18:22.073946) are interesting. You can calculate the duration – which is the difference between them.
  • Request type API, Service, Admin. An Admin request is like using an URL like /zosConnect/services/stockQuery to get information about the stockQuery service.
  • The API name and version – stockmanager 1.0.0
  • The service name and version – stockQuery 1.0.0. You get the version information. If you do an online display of the service the version is not available.
  • Method GET/POST etc
  • The service provider. This is the code which does the real work of connection to CICS, MQ, or passing the request through. IBM MQ for z/OS, restclient-1.0
  • Request id starts at 1 and increments for the life of the server. If you restart the server it will restart from 1. I do not think this is very useful.
  • For “pass through” requests, z/OS Connect confusingly calls the back end service the Statement of Record or (SOR). (MQ is a transport, not a Statement of Record.) The “pass through” service definition is built from a parameter file and zconbt program. The reported data is
    • SOR ID the host and port 10.1.3.10:19443. These are from the <zosconnect_zosConnectServiceRestClientConnection host and port values.
    • SOR Reference restClientServiceY This is from the connectionRef=in the parameter file and the <zosconnect_zosConnectServiceRestClientConnection…> definition
    • SOR Resource zosConnect/apis/through. This is from the uri= in the parameter file.
    • Time of entry and time of return of the SOR service.
    • From the times calculate the difference to get the duration of the remote request.
  • It would be useful to have this “pass through time” for services calling MQ, CICS etc, so we could get a true picture of the time spent processing the requests.
  • The size of the data payload (0) , and the size of the response(94) excluding any headers.
  • A tracking token. The hex 64 byte string is passed to the some called servers. It is passed to some backends (CICS and “pass through”) to be able to correlate the request across systems. It is not passed to MQ. See X-Correlation-ID below for an example. This field is nulls for Admin request. When a request was “passed through” to another z/OS Connect server which processed the request, the tracking token was not reported in the SMF data of the second system. I dont know if the CICS SMF data records this token, but it is of little use for MQ, or for “pass through”.
  • You get 4 request header, and 4 response header fields. They were blank in my measurements, even though headers were passed to the pass through service. Looking a the http traffic, the request coming in had a header “Content-Type:application/json”. The request passed through to the back end included
    • User-Agent: IBM_zOS_Connect_REST_SP/open beta/20200803-0918
    • X-Correlation-ID: BAQ1wsHYAQAYwcTDxNfTQEDi8ObxQEBAQNikjpk+1klAAA==

What can you do with the data?

Do I need to use the SMF data?

From a performance perspective these records do not provide much information, as they are lacking information about CPU usage. From an audit perspective they have some useful information – but the records are missing information which would provide useful audit information. There is an overlap between the information in the SMF 123 records and the …/servers/…logs/http_access.log file which provides, date time, userid, URI, HTTP code.

What do I want to report on?

Decide what elements of the URI you want to report on. For example the URI /stockmanager/stock/items/999999 includes the stock part number, which may be different for each request. You might decide to summarise API usage on just the first two elements /stockmanager/stock/. You may have to treat each API individually to extract the key information.

I’ll use the term key for the interesting part of the URI – for example /stockmanager/stock.

What reports are interesting?

I think typical questions are:

  1. Which is the most popular API key?
  2. Is the usage of an API key increasing?
  3. How many API key requests were unsuccessful? This can show set-up problems, or penetration attempts.
  4. What is the response time profile of the requests? Are you meeting the business response time criteria?
  5. Which sites are sending in most of the requests. You cannot charge back on CPU used, as you do not know the CPU usage. You could do charge back at a fixed cost per API request, with each API having a different rate.
  6. Which userids are sending in most of the requests. You may want to provide more granular certificate to userid mapping to give you more detailed information

Understanding z/OS Connect SMF 120 subtype 11 data

z/OS Connect can provide two types of SMF record

  1. SMF 120 subtype 11, provided by the base Liberty support. This gives information on the URL used to access Liberty, and the CPU used to perform requests. This is enabled at the Server level – so you can have records for all request, or no requests. There is one SMF record for each web server request. Would I use this to report CPU used ? No – see the bottom of this blog.
  2. SMF 123 provides information about API and the service used, and the “pass through” services. It provides elapsed time of the request, and of the the “pass through” requests. It does not provide CPU usage figures. This can be configured to produce records depending on the http host and port used to access z/OS Connect. One SMF record can have data for multiple web server requests. The SMF records are produced when the SMF record is full – or the server is shut down.

The SMF 120-11 and SMF 123 records are produced independently, and there is no correlating field between them. They both have a URI field, and time stamps, so at low volumes it may be possible to correlate the SMF data.

I’ll document the fields which I think are interesting. If you think other fields are useful please let me know and I’ll update this document.

I have written an SMF formatter in C which prints out interesting data, and summarises it.

SMF 120-11

  • You get the standard date and time the record was produced, and with the LPAR. You can use PGM=IFASMFDP with the following to filter which records are copied
DATE(2020282,2020282)
START(1000)
START(2359)
  • There is server version (3), system(SOW1), and job id(STC04774) which are not very interesting
  • Server job name(SM3) is more interesting. I started the server with s baqstrt,parms=’MQTEST’,jobname=sm3
  • The config dir (/var/zosconnect/servers/MQTEST/) is boring – as is code level (20.0.0.6)
  • The start and stop times (2020/10/08 09:18:19.852360 and 2020/10/08 09:18:22.073946) are interesting as is the duration – which is the difference between them.
  • You get userid information.
    • I used a certificate to authenticate. The DN from the certificate is not available. You only get the userid from the RACF mapping of DN to userid. This mapped userid was in the 64 byte field. The 8 byte userid field was empty for me. The lack of certificate DN, and having the userid in the wrong field feels like a couple of buglets.
  • You get the URL used /stockmanager/stock/items/999999 I treat this as a main key for processing and summarising data. If you want to summarise the data, you may want so summarise it just on /stockmanager/stock/. The full URI contains the part number – and so I would expect a large number of parts.
  • You can configure your requests to WLM. For example
<wlmClassification>
<httpClassification transactionClass="TCI1" method="GET" 
    resource="/zosConnect/services/stockQuery"/>
</wlmClassification>

This produced in the SMF record

WLMTRan :TCI1
WLM Classify type :URI :/zosConnect/services/stockQuery
WLM Classify type :Target Host :10.1.3.10
WLM Classify type :Target Port :19443

This means that the URL, the host, and the port were passed to WLM to classify.

If you get the WLM classification you also get CPU figures when the enclave request ended (was deleted).

  • You get the ports associated with the request.
    • Which port was used on the server – Target Port :9443
    • Where did the request come from? Origin :10.1.1.1 and port :36786
  • The number of bytes in the response Response bytes :791
  • CPU figures for the CPU used on the TCB. See discussion below on the usefulness of this number. You get the CPU figures before the request, and after the request – so you have to calculate the difference yourself! The values come from the timeused facility. You can calculate the delta and get
    • CPU Used Total : 0.967417
    • CPU Used on CP : 0.026327
    • and calculate these to to get CPU Delta. on Z**P : 0.941090 This is the CPU offloaded to ZIIP or ZAAP.
  • If you had the URI classified with WLM, you get Enclave data, see below for a discussion on what the numbers mean.
    • Enclave CPU time : 0.148803
    • Enclave CPU service : 0.000000
    • Enclave ZIIP time : 0.148803
    • Enclave ZIIP Service : 0.000000

What do the CPU numbers mean?

Typically a transaction flow is as follows

  1. A listening thread listens on the HTTP(s) host and port.
  2. When a request arrives, it passes the request to a worker thread, and goes back to listening
    1. The worker thread may do some work and send the response back
    2. The worker thread may need to call another thread to do some work. For example to issue an MQ request,
      1. the MQ code looks for a thread in a pool for a matching queue manager and userid. If it find one it uses it the thread and issues the MQ request.
      2. If it does not find a matching thing thread it may allocate a new thread, and issue an MQCONN to connect to MQ. These are both expensive operations, which is why having a pool of threads with queue manager and userid is a good way of saving CPU
      3. The work is done
      4. The thread is put back into the MQ pool, and the application returns to the worker thread
      5. The worker thread sends the response back to the originator

A thread can ask the operating system, how much CPU time it(the thread) has used. What usually happens is

  1. the thread requests how much CPU it has used
  2. the thread does some work
  3. the thread requests how much CPU it has used,
  4. the thread calculates the difference between the two CPU values and reports this delta.

I the SMF 120 record records the CPU from just the worker thread – and no other thread.

Enclaves

When there are more than one thread involved it gets more complex, as you could have a CICS transaction issuing an MQ request, then a DB2 request, and then an IMS request. You can set up z/OS WorkLoad Manager(WLM) to say “these CICS transactions in this CICS region are high priority”.

With some subsystems you can pass a WLM token into a request. The thread being invoked call tell WLM that the thread is now working on behalf of this token. The thread does some work, and tells WLM that it has finished doing the work. WLM can manage the priority of the threads to achieve the best throughput, for example making the thread high or low priority. WLM can manage a thread doing work in multiple LPARs across a sysplex!

WLM records the CPU used by the thread while performing the work, accumulates and reports this.

This use of multiple threads for a business transaction across one or more address spaces is known as an enclave.

What happens with enclaves?

  1. A request arrives at the listener thread.
  2. The Liberty looks up the URI in the <wlmClassification httpClassification…. It compare the server’s host, server’s port, the URI resource /stockmanager… method ( GET) and finds the best match for the transactionClass.
    1. If there is a transactionClass,
      1. the server calls WLM with the Subsystem type of CB, the specified collectionName, and the transactionClass.
      2. WLM looks for these parameters and if WLM has a matching definition then WLM will manage the priority of the work,
      3. WLM returns a WLM token.
      4. This WLM token is passed to threads which are set up for enclaves.
    2. If there is no transaction class specified in Liberty, or WLM does not have the subsystem, collectionname, transactionClass then there is no token or a null WLM token
    3. The work continue as before.
    4. If another thread is used then pass the WLM token. If the code is set up for WLM token then report “work started”, when it has finished report “work ended”

What happens if the request is not known to WLM.

The worker thread calculates the CPU used for just its work, and reports this. The CPU used by any other thread is not report. The figures reported are the CPUTotal timeused values. You have to calculate the difference yourself

What happens if the request is known to WLM.

You get the timeused CPU for the worker thread – as with the case where the request is not known to WLM.

From RMF (or other products) you get out reports for an interval with

  1. The number of requests in the interval
  2. The rate of requests in the interval
  3. The amount of time on a CP engine in seconds
  4. The amount of time on a ZIIP engine is seconds
  5. The amount of time on a ZAAP in seconds.
  6. Over the interval, what percentage of time was CP on CP engines, zAAP on zAAP engines, zAAP on CP engines, zIIP on zIIP engines.

From the SMF 120 records you get

Enclave CPU time
Enclave ZAAP time
Enclave ZIIP time

Example Enclave figures.

For 100 API requests, the figures as reported by SMF 120-11, and I averaged the values.

  1. Average CPU(1) 0.023
  2. Average CPU(2) 0.0008
  3. Enclave CPU 0.029
  4. Enclave ZAAP 0
  5. Enclave ZIIP 0.028

The figures reported by RMF per request

  1. CPU 0.031
  2. ZIIP 0.039
  3. ZAAP 0.000
  4. Total 0.070 seconds of CPU per transaction

These figures tie up – the Enclave CPU, ZIIP, and ZAAP are similar.

The CPU used by the server address space was

  1. CPU 30.1 seconds
  2. ZIIP 28.7 seconds
  3. ZAPP 0 seconds.
  4. Total 58.8.

Each request took 0.070, and there were 100 requests – so reported 7 second of CPU.

The difference(51) seconds is not reported in the transaction costs. It looks like the “timeused” value is less than 1% of the CPU value, and the enclave figures are under 2% of the grand total.

Looking at the trace in a dump, I can see many hot TCBs using much more CPU that is reported by WLM and RMF. I expect that many TCBs used in a request, but they do not have the enclave support in them. Overall – pretty useless for charge back and understanding the cost per transaction.

What’s the difference between MQ Web, and z/OS Connect MQ support?

With MQ Web

  1. You can issue commands to administer MQ  for example display, define, delete MQ objects.
  2. You can put and get messages to and from a queue.  The message is what you specify – typically a character string.

With Z/OS Connect MQ support

  1. You can put and get messages to and from a queue, and do transformations on the message.  For example mapping a COBOL structure to JSON.  
  2. You can do field validation.
  3. You can covert HTTP code “200” to “great it worked”.

What is common?

They both use z/OS WebSphere Liberty to provide the basic web server.

A practical path to installing Liberty and z/OS Connect servers – 10 use the MQ sample

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Use the MQ sample

You need to have installed the service, and protected it.

You need to configure MQ to include the MQ support and tell JMS where the libraries are

<server> 
<!-- Enable features --> 
<featureManager> 
    <feature>zosconnect:mqService-1.0</feature> 
</featureManager> 
                                                                                                         
<wmqJmsClient nativeLibraryPath="/Z24A/usr/lpp/mqm/V9R1M1/java/lib"/> 

<variable name="wmqJmsClient.rar.location"
   value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/jca/wmq.jmsra.rar"/> 
</server> 

You could configure a variable for the mq directory so you use it once, and use

<variable name="wmq"  value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/"
<wmqJmsClient nativeLibraryPath="${wmq}”/>
<variable name="wmqJmsClient.rar.location"
   value="${wmq}wmq.jmsra.rar}">

You could also pass the mq location as a variable in STDENV, and so pass it in through JCL.

Configure the jms to define the queue manager and queues

<server> 
 <jmsConnectionFactory jndiName="jms/cf1" 
     connectionManagerRef="ConMgr1"> 
    <properties.wmqJms transportType="BINDINGS" 
         queueManager="CSQ9"/> 
 </jmsConnectionFactory> 
                                                                                                      
 <connectionManager id="ConMgr1" maxPoolSize="5"/> 
                                                                                                      
 <!-- A queue definition where request messages 
      for the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockRequestQueue"> 
    <properties.wmqJms 
       baseQueueName="STOCK_REQUEST" 
       targetClient="MQ"/> 
 </jmsQueue> 
                                                                                                      
 <!-- A queue definition where response messages from 
      the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockResponseQueue"> 
    <properties.wmqJms baseQueueName="STOCK_RESPONSE" targetClient="MQ"/> 
 </jmsQueue> 
</server>

and include these in the server.xml file.

You need to compile and run the back end service.  See here.  Take care if using cut and paste as there a long lines which wrap, and cause compilation errors.

Because the MQ path name is long I used

export HLQ="/usr/lpp/mqm/V9R1M1/java/lib"

java -cp $HLQ/com.ibm.mq.allclient.jar:. -Djava.library.path=$HLQ TwoWayBackend CSQ9 STOCK_REQUEST STOCK_RESPONSE

I set up a job to run this in back ground, so I could free up my TSO terminal.

Use the API

Once installed you should be able to use the API. For example

curl –insecure -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem https://10.1.3.10:9443/stockmanager/stock/items/999999

If the back end application was working I got

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

If the back end application was not working I got back an empty response.

The back-end application runs until Ctrl+c is used.  On my USS, the ESCape key is cent symbol ¢ (Unicode 00a2) which I do not have on my default keyboard.    See  x3270 – where’s the money key? for guidance on how to set it.

 

Use the Service

To use the API I used a web browser with

https:10.1.3.10/9443/stockmanager/stock/items/999999

and got back

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

or curl with

 curl --insecure -X post -H Content-Type: application/json
  -H "Content-Type:application/json" -i --cacert cacert.pem
  --cert adcdd.pem:password --key adcdd.key.pem 
  --data {"STOCKQRYOperation": {"sqreq" : { "item_id": 2033}}}
  https://10.1.3.10:9443/zosConnect/services/stockQuery?action=invoke

A practical path to installing Liberty and z/OS Connect servers – 11 WLM classifying a service

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Classify the services with WLM to give them the right priority

You can configure an API or service, for example https://10.1.3.10:9443/stockmanager/stock/items/999999, so it gets an appropriate service class in WLM, for example high priority.

You can use RMF to report on service classes to see the response time profile, and if the service class is meeting its performance goals.

See here for a good article.

This page gives the example

<wlmClassification>
  <httpClassification transactionClass="CLASS001" 
      host="127.0.0.1" 
      port="9080" 
      method="GET"
      resource="/testResource" />
</wlmClassification>

You can classify the traffic depending on the IP address and port of the server, as well as the resource name.

A practical path to installing Liberty and z/OS Connect servers – 3 Initial customisation

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Copy the PDS from the install system to where they will run.

These files can be sent using FTP or XMIT.

Copy the product ZFS to where it will be used.

You can use DFDSS to dump the ZFS and move it to the SYSPLEX where it will be used.

  1. Mount the file system read/write for example /usr/lpp/IBM/zosconnect/v3r0beta
  2. Perform Setting up the product extensions directory.  You need to do this on each LPAR, as it set up files in /var/zosconnect
  3. Mount the file system READ
  4. Change the BPXPRMxx parmlib member so the file sytsem is mounted READ it at IPL
  5. You may want to create an alias /usr/zosc to the product executables.
    • ln -s /usr/lpp/IBM/zosconnect/v3r0beta/bin /usr/zosc

Create the angel started task if needed.

If you will be using an existing Angel you will need to set up RACF permissions.

Follow the instructions the z/OS Connect  documentation

  • If you need a named Angel, specify the value in the NAME=value.  You can specify NAME when you start the task.
  • You do not need to classify this in WLM as it uses little CPU once it has been started.

Create the file system for the server

  1. See Creating the z/OS Connect EE shared directory. You will need to know the path for server’s home directory. The path /var/zosconnect can only be used on the LPAR.   The files cannot be shared.
  2. If you need to allocate a file system.
    1. Allocate a ZFS. If the initial allocation of the ZFS is too small, then the ZFS will grow on demand, or using the zfsadm grow command to expand it.
    2. Mount the file system over the directory.
    3. Change the BPXPRMxx parmlib member so the file system is mounted RW at IPL.

Create the Z/OS Connect started task

See Creating a z/OS Connect Enterprise Edition server.

Using /usr/zosc alias defined above

export  WLP_USER_DIR=/var/zosconnect
/usr/zosc/zosconnect create default --template=zosconnect:default

If you need to start again, use the following to remove the server.

rm -r  /var/zosconnect/servers/default
  1. Copy BAQSTRT to the SYS1.PROCLIB concatenation, to the name you are going to use, BAQSTRT, BAQSTRT1 etc
  2. When you edit the member, use caps off to use lower case text in the member.
  3.  If you will be using MQ, add the MQ libraries to STEPLIB
    //STEPLIB   DD DSN=CSQ913.SCSQANLE,DISP=SHR 
    //          DD DSN=CSQ913.SCSQAUTH,DISP=SHR
  4. You may find it easier to remove the _CEE_RUNOPTS= and add
    //CEEOPTS DD DISP=SHR,DSN=...(CEEOPTS)

    to the JCL. This makes it easier to add CEE options, and specify storage heap and pool sizes.

  5. In my CEEOPTS I use RPTSTG(ON), to report the storage pool usage. This provides useful information which may be useful when investigating performance problems.
  6. Despite what the Liberty documentation says you do need
      • JVM_OPTIONS=<Optional JVM parameters>
    • for example
      • JVM_OPTIONS=-Xoptionsfile=/var/zosconnect/servers/default/jvm.options
  7. You may want a JVM specific to a server, as you may want to set some service specific options such as TLS trace.
  8. You may want to specify a default server name in the JCL parms. You can specify a different one when z/OS Connect is started.
  9. Specify WLP_USER_DIR for your server’s directory. You could make this as a parameter to the started task JCL if you want to provide isolation for the servers, and give each server its own file system.

Create the JVM options file

Go to the server’s directory.

cd /var/zosconnect/servers/default

use oedit jvm.options to create an EBCDIC file.

add

-Dcom.ibm.ws.zos.core.angelRequired=true 
#-Dcom.ibm.ws.zos.core.angelName=MYZANGL

The -Dcom.ibm.ws.zos.core.angelRequired=true option stops the server from starting if the angel process is not running. You get message:

CWWKB0116I: This server is not registered with an angel process even though it is configured to require registration with an angel process. This server is attempting to stop.

The option -Dcom.ibm.ws.zos.core.angelName=MYZANGL allows you to specify a non default angel process name. The # in front of the statement makes it into a comment.

add the following, if required, with your time zone.

-Duser.timezone=Europe/London

Override TCPIP

If you are using a non standard TCPIP, edit server.env and add

_BPXK_SETIBMOPT_TRANSPORT=TCPIP2

Tailor the server.xml file

It was easy to get my configuration options mixed up, and hard to compare two server configurations. I found it much easier to manage my configuration by putting the components into their own xml file. This also makes having shared components much easier to implement- as you just include the name of the shared file.

The configuration files have to be in ASCII, and you have to take care ensure the files are in the correct code page.

There are at least three ways of creating the .xml files (I use the third option through ISPF 3.17)

  1. using omvs commands to copy and edit existing ASCII files
  2. use omvs commands to create ASCII files
  3. use ispf option 3.17

1 Using omvs commands to copy an existing ascii file,  then edit it it

For example

cp server.xml httpEndpoint.xml

edit the httpEndpoint.xml to keep the <server> ,httpEndpoint information, </server> and delete the rest.  The file should look like

<server description="new server"> 
    <httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" /> 
</server>

then edit the server.xml and replace

<httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" />

with

<include location="${server.config.dir}/httpEndpoint.xml"/> 

Note: if you use the edit command source ? It should say ASCII already set.

2 Use OMVS to created and edit files

By default, if you edit a file in OMVS it becomes an EBCDIC file.

If you create a file without copying it, you need to use

touch httpEndpoint.xml 
chtag -t -c ISO8859-1 httpEndpoint.xml     
oedit httpEndpoint.xml

Make the same changes as above.

3 Use ispf option 3.17

If you use ISPF 3.17 (z/OS UNIX Directory List) you have more flexibility with editing files.

Edit server.xml

use the line commands mmmm and the primary command create to create a new member, give the new name when prompted, for example httpEndpoint.xml

Replace the lines you copied out with

	<include location="${server.config.dir}/httpEndpoint.xml"/>

and edit the file you just created to put <server> … </server> around the content.

Add more logging to the JCL’s STDOUT.

The default logging to the jobs STDOUT is only audit records. I found it much easier to include other levels of information so I did not have to keep looking in the message.log file. Note: the STDOUT content does not have time stamps so you do not know when the output was produced- so it is easier to use – but has less information!

I created logging.xml with the following content.

<server> 
 <logging 
 consoleLogLevel="INFO" 
 /> 

 <!-- this is a comment where I store trace information
   traceFormat="BASIC" traceSpecification="*=info:
       com.ibm.ws.security.*=fine:zos.native.03=fine" 
  end comment   -->  
</server>

I added

<include location="${server.config.dir}/logging.xml"/> 

to the server.xml file.

The consolLogLevel=”INFO” writes additional information to the STDOUT file, than the default AUDIT value which just outputs AUDIT records.

Define the http address and ports

Edit the httpEndpoint definitions

    <httpEndpoint id="defaultHttpEndpoint" 
                  accessLoggingRef="ALR" 
                  host="*" 
                  httpPort="9081" 
                  httpsPort="9443" /> 
    <httpAccessLogging 
       enabled="true" 
       id="ALR" 
       logFormat="%{c}a! %h %u %{t}W %q %r %s %b" 
   /> 

Check

  • host matches the TCPIP value to use. With my VIPA I specify the VIPA address host=”10.1.3.10”
  • httpPort – if you already have a Liberty instance running port 9081 will be in use by another server
  • httpsPort – if you already have a Liberty instance running port 9443 will be in use by another server

You will need the port values when you use the server.

The httpAccessLogging statement pointed to by accessLoggingRef, will create a log file with every attempt to use the server.

Example output

10.1.1.1! 10.1.1.1 ADCDC {12/Sep/2020:16:07:17 +0000} - 
GET /zosConnect/apis/stockmanager HTTP/1.1 200 296

Where 10.1.1.1 is the source IP address and ADCDC is the userid used.

By default this file is http_access.log in the …/logs directory.

See here for the format of the logFormat statement.

After tailoring

I did more packaging the content into include .xml files.

After my “repackaging” my server.xml file looked like

<?xml version="1.0" encoding="UTF-8"?> 
<server description="new server"> 
    <include location="${server.config.dir}/applicationMonitor"/> 
    <include location="${server.config.dir}/logging.xml"/> 
    <include location="${server.config.dir}/httpEndpoint.xml"/> 
    <include location="${server.config.dir}/cors.xml"/> 
    <include location="${server.config.dir}/config.xml"/> 
    <include location="${server.config.dir}/zosconect.xml"/>                                                                              
</server>

You can use ispf option 3.17 to list the files in the servers’s directory.

Once you have configured the server you can use the admin centre to manage the files.  This will require

<remoteFileAccess>  
    <writeDir>${server.config.dir}</writedir>
</remoteFileAccess>

Finally ! You can start the server.

A practical path to installing Liberty and z/OS Connect servers – 4 start the server

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Starting the server

Using the definitions above I started the server, using the default configuration directory, and used jobname=default.

s baqstart,parms=’default’,jobname=default

On my z/OS system, running on zPDT on Linux, it takes about 5 minutes to start.

During start up, check that the parameter have been accepted – with the correct case.

If you get configuration errors, fix them and use the

f ...zcon,refresh

command to get the server to pickup the changes. Note. Not all changes are picked up by the refresh command, and you have to stop and restart the server to pick up changes.

There is output in the STDOUT and possibly in STDERR, also in the /var/zosconnect/default/logs/messages.log

STDOUT will have messages like

[AUDIT] CWWKG0028A: Processing included configuration resource: 
/var/zosconnect/servers/default/httpEndpoint.xml

which tells you which configuration files you are using.

If you get any syntax errors you should fix them. The command f …,zcon,refresh  may cause it to reread the definitions.

The second +  start of server start  after  IPL may be faster than the first if Java has cached some of the executables.

Check for messages

Authorized service group … is available.

Which tells you that the Angel service is enabled.

Web application available (default_host): http://10.1.3.10:9080/

check this is what you expect. If the port is not valid check the httpEndoint.xml. You do not get an entry for the TLS port, because TLS is not enabled.

If you use a web browser to logon to the URL http:10.1.3.10:9081 (note:not hppts) you will just get a blank screen. If you look in logs/http_access.log you will see a code 302. In this case, 302 means you were trying to use an http session to a session that requires https.

If you use other URLs you may get a message saying you need to use https.

If you use z/OS explorer and use the http:…9081 above you will get “302, Found”. This means you were trying to use an http session to a session that requires https.

You have successfully created a z/OS connect server, and used it – even though you could not do much.

To stop the server use the z/OS P … command.

Most of the facilities in z/OS Connect require TLS, so you cannot do much until you have configured that.

A practical path to installing Liberty and z/OS Connect servers – 5 enable logon security

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Enable logon security

To enable security checking you need  to use the zosconnect_zosConnectManager tag.

See here for information, and here for the tag  syntax.

<zosconnect_zosConnectManager 
     requireAuth="true"  
     requireSecure="true" 
     setUTF8ResponseEncoding="true"/>

add to the zosconnect.xml or server.xml file.

You cannot do much with this enabled as you need TLS

Add in the SAF repository to use z/OS security services.

Within the server directory create a file called saf.xml. (I always used ISPF 3.17, and use cc…cc create.. to create a new file.

<server> 
                                                                                
  <featureManager> 
     <feature>transportSecurity-1.0</feature>   
     <feature>zosSecurity-1.0</feature> 
     <feature>appSecurity-2.0</feature> 
  </featureManager> 
  <authentication id="Basic" cacheEnabled="false" /> 
  <safAuthorization id="saf-authorization"  racRouteLog="ASIS" />                                                                                 

  <safRegistry id="saf" realm="myrealm"/> 
  <safCredentials profilePrefix="ZZZZDFLT" 
     suppressAuthFailureMessages="false" 
     unauthenticatedUser="WSGUEST"/> 
</server>

Where you need to specify the profile prefix (for example ZZZBFLT), and define the security profile.

I do not think the realm is used for SAF processing.

If you are planning on using a new profilePrefix  (YYYYDFLT) you will need to create several profiles.

RDEFINE APPL YYYYDFLT  UACC(NONE) 
PERMIT YYYYDFLT  CLASS(APPL) ACCESS(READ) ID(group1,group2)
SETROPTS raclist(APPL) refresh

RDEFINE server BBG.SECPFX.YYYYDFLT uacc(NONE)
permit  BBG.SECPFX.YYYYDFLT class(server) access(read) id(start1)
setropts raclist(server) refresh

rdefine ejbrole YYYYDFLT.zos.connect.access.roles.zosConnectAccess 
   uacc(none)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
   class(ejbrole) 
   id(Adcdc) access(READ)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
    class(ejbrole) 
    id(group1,group2) access(READ)
setropts raclist(ejbrole) refresh

These profiles are only accessed once someone tries to use the server – not at startup, so try logging on using the https port.

Every one using the server needs read access to YYYYDFLT CLASS(APPL).

As an interim step towards getting your server up and secure, enable userid and password authentication.

<webAppSecurity overrideHttpAuthMethod="BASIC"/> 
<webAppSecurity allowFailOverToBasicAuth="true"/> 

overrideHttpAuthMethod=”BASIC” says used userid and password – and not client certificates.

allowFailOverToBasicAuth=”true” says if the client certificate logon fails – fail over to basic, userid and password.

Once you have configured the server, you can remove the overrideHttpAuthMethod tag, so client certificates are used.  Once these are working remove the allowFailOverToBasicAuth, to remove the userid and password option.  With these both removed, people will have to use client certificates to authenticate.

A practical path to installing Liberty and z/OS Connect servers – 6 Enabling TLS

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Configuring TLS

  1. You can configure the server to creates a keystore file on its first use. This creates a self signed certificate. This is good enough to provide encryption of the traffic. Certificates sent from the client are ignored as the trust store does not have the Certificate Authority certificate to validate them.
  2. You can use your site’s keystore and trust store. The server can use them to process certificate sent from the client for authentication.

Decide how you want to authenticate

Most of the functions require an https connection. This will require a keystore.

You can decide if

  1. The server uses the client’s certificate for authentication,
    1. if that does not work then use userid and password
    2. if that does not work, then fail the request; there is no fall back to userid and password.
  2. The server does not use the clients certificate.
    1. You can configure that userid and password will used for authentication
    2. There is no authentication

Have the server create a keystore.

You can get Liberty to create a keystore for you. This creates a self signed certificate and is used to encrypt the traffic between client and server. This is a good start, while you validate the set up, but is not a good long term solution.

Create keystore.xml with

<server>
<keyStore id="defaultKeyStore" password="${keystore_password}" /> 

<ssl clientAuthentication="false" 
    clientAuthenticationSupported="false" 
    keyStoreRef="defaultKeyStore" 
    id="defaultSSLSettings" 
    sslProtocol="TLSv1.2" 
/> 
</server>

Add to the bottom of the server.xml file

 <include location="${server.config.dir}/keystore.xml"/>

If you have keyStore id=”defaultKeyStore”, (it must be defaultKeyStore) and do not have a keystore defined, the the server will create the keystore in the default location (${server.output.dir}/resources/security/key.p12) with the password taken from the server.env file.  See here.

Restart the server.

I got the messages

CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started 
and is now listening for requests on host 10.1.3.10  
(IPv4: 10.1.3.10) port 9443.

Showing TLS was active, and listening on the 9443 port.

If the keystore was created, you will get messages like

[AUDIT   ] CWPKI0803A: SSL certificate created in 87.578 seconds. 
SSL key file: /var/zosconnect/servers/d3/resources/security/key.p12 
[INFO    ] Successfully loaded default keystore: 
/var/zosconnect/servers/d3/resources/security/key.p12 of type: PKCS12

The certificate has a problem (a bug). It has been generated with CN:localhost, O:ibm: ou:d3 where d3 is the server name. The Subject Alternative Name (SAN) is DNS:localhost. It should have a SAN of the server’s IP address (10.3.1.10 in my case).

Clients check the SAN and compare it with the server’s IP address.

  1. Chrome complain. “Your connection is not private NET:ERROR_CERT_AUTHORITY_INVALID”, and the option to accept it
  2. Firefox gives “Warning: Potential Security Risk Ahead”, and the option to accept it.
  3. Z/OS explorer gives a Server certificate alert pop up, saying “Host:10.1.3.10 does not match certificate:localhost” and gives two buttons Decline or Accept.
  4. With curl I got SSL_ERROR_SYSCALL.

You can accept it, and use it until you have your own keystores set up. You can also reset this decision.

Using a RACF keyring as the keystore.

You can use a file based keystore or a RACF keying.  Below are the definitions for my RACF keyrings. The started task userid is START1. The keystore (containing the private key for the server is keyring START1/KEY. The server should use key ZZZZ.

The trust store, containing the Certificate Authority certificates and any self signed certificates from clients, is START/TRUST.

The <ssl.. /> points to the different keystores, so it makes sense to keep all these definitions in one file.  You may already have a file of these definitions which you can use from another Liberty server.

<server>

<sslDefault sslRef="defaultSSLSettings"/> 
<ssl clientAuthentication="true" 
    clientAuthenticationSupported="true" 
    id="defaultSSLSettings" keyStoreRef="racfKeyStore"  
    serverKeyAlias="ZZZZ" 
    sslProtocol="TLSv1.2" 
    trustStoreRef="racfTrustStore"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfKeyStore" 
     location="safkeyring://START1/KEY" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfTrustStore" 
     location="safkeyring://START1/TRUST" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/>                                                                                                                  
</server>

This sets clientAuthentication=”true” and clientAuthenticationSupported=”true”

Specify if you want to use a client certificate for authentication

If you specify clientAuthenticationSupported=”true”… the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.

The default keystore will not be able to validate any certificates sent from the client. When connecting to Chrome with certificates set up, I got an FFDC and messages

  • [INFO ] FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is: java.security.cert.CertPathValidatorException:
  • [ERROR ] CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired.

If you specify clientAuthentication=”false” (the default) the server does not request that a client send a certificate during the handshake.

If you specify <webAppSecurity allowFailOverToBasicAuth=”true” />  the client certificate connection is not used or it fails,

  1. if  you specify<webAppSecurity allowFailOverToBasicAuth=”true” /> the user will be prompted for userid and password
  2. If you specify <webAppSecurity allowFailOverToBasicAuth= false > or not specified, the connection will fail.

If a userid and password can be used, the first time a browser uses the server it will be prompted for userid and password. As part of the handshake, the LTPA2 cookie is sent from the server. This has the userid and password encrypted within it. If you close down the browser and restart it (not just restart it from within the browser) you will be prompted again for userid and password. You can also be prompted for userid and password once the LPTA cookie has expired.

If you are using z/OS explorer and get a code 401, unauthorised, you may be using a certificate credential ( format userid@CertificateAuthority(CommonName)) rather than a userid and password with format of just the userid eg COLIN. Use “Set Credentials” to change credentials.

You can see what userid is being used for the requests, from the …/logs/http_access.log file.

To make it even more complex you can have different keystores for different connections or ports.  See here. But I would not try that just yet.

Map client certificates to a SAF userid

If you are using certificate authentication you will need to map the certificate to a userid using the RACDCERT MAP command.

Testing it

If the server starts successfully you can use a web browser with URL

  http:/10.1.3.10:9443/zosConnect/api-docs

and it should display json data.

If you get “Context Root Not Found” or code 404 you should wait and retry, as the https processing code is active, but the code to process the requests is not yet active.

Review the contents of …/servers/…/logs/http_access.log to see the request being issued and the http completion code.

If you have problems connecting clients over TLS add -Djavax.net.debug=ssl:handshake to the jvm.options file and restart the server.

If you connect to the z/OS Explorer, and logon to the z/OS Connect EE Server, you should have a folder for APIs and Services – which may have no elements.

A practical path to installing Liberty and z/OS Connect servers – 7 adding apis and services

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Adding APIs and Services

For services you have to include a feature. For example with mq you need

 <feature>zosconnect:mqService-1.0</feature>

These are listed here.   You need the CICS service to be able to use CICS applications etc.

You also need to copy the files application (*.aar) files into the apis directory, and the service(*.sar) files into the services directory.

There are several ways of doing this.

You can do this with the uss cp command. The command below copies an MQ one – you can use others

Find what applications and services are available

find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.aar
find  /usr/lpp/IBM/zosconnect/v3r0beta/runtime/* -name *.sar   

copy the files to the servers directory

cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/apis/*
  /var/zosc*/ser*/default/re*/z*/ap*                                                                                   
cp /usr/lpp/IBM/zosconnect/v3r0beta/runtime/templates/servers
  /sampleMqStockManager/resources/zosconnect/ser*/* 
  /var/zosc*/ser*/default/re*/z*/se*                                                                                        

Use a REST request or use the zOS explorer.

Once you have done this either restart the server or use the operator command

f ...zcon,refresh

and the application and services should be available

You should get in the message.log on the STDOUT (if you have configured it to print information messages – see earlier posts).

BAQR7000I: z/OS Connect EE API archive file stockmanager installed successfully.
BAQR7043I: z/OS Connect EE service archive stockQuery installed successfully.

Using a web browser, or curl with the URL https://10.1.3.10:9443/zosConnect/services gave me

{"zosConnectServices":
  [
    {"ServiceName":"stockQuery",
     "ServiceDescription":"A stock query service based on IBM MQ.",
     "ServiceProvider":"IBM MQ for z/OS",
     "ServiceURL": "https://10.1.3.10:9443/zosConnect/services/stockQuery"
      }
  ]
}

URL https://10.1.3.10:9443/zosConnect/apis gave me

{"apis":    
  [
    {"name":"stockmanager","version":"1.0.0","description":"",
     "adminUrl":"https://10.1.3.10:9443/zosConnect/apis/stockmanager"
     }
   ]
}

You can invoke the service with https://10.1.3.10:9443/zosConnect/services/stockQuery and get details of the stockQuery service.

Once it all works…

I played with the curl interface to deploy a service.

  • I copied the stockQuery.sar file down to linux in binary
  • I could use jar -tvf stockQuery.sar to display the contents
  • I used curl –insecure -v –header Content-Type:application/zip -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem -X post –data-binary @/home/zPDT/stock2.sar https://10.1.3.10:9443/zosConnect/services
    • Note:  If I used –data-binary “@/home/zPDT/stock2.sarin quotes I got
      • HTTP/1.1 415 Unsupported Media Type
      • “errorMessage” : “BAQR0418W: An unsupported media type of application/x-www-form-urlencoded was specified.”}
    • because the string was sent up – not the file.  I used the –verbose option to see the number of bytes sent.
    • I also got these messages if the file was not found.  It sends the string instead!