Using enclaves in a Java program – capturing elapsed and CPU time used by a Java transaction.

Ive blogged about using enclaves from a C program.  There is an interface from Java which uses this C interface.

Is is relatively easy to use enclave services from a java program, as there are java classes for most of the functions, available from JZOS toolkit.  For example the WorkloadManager class is defined here.

Below is a program I used to get the Work Load Manager(WLM) services working.

import java.util.concurrent.TimeUnit;
import com.ibm.jzos.wlm.ServerClassification;
import com.ibm.jzos.wlm.WorkUnit;
import com.ibm.jzos.wlm.WorkloadManager;
public class main
{
// run it with /usr/lpp/java/J8.0_64/bin/java main
public static void main(String[] args) throws Exception
{
WorkloadManager wlmToken = new WorkloadManager("JES", "SM3");
ServerClassification serverC = wlmToken.createServerClassification();
serverC.setTransactionName("TCI3");
for ( int j = 0;j<1000;j++)
{
WorkUnit wU = new WorkUnit(serverC, "MAINCP");
wU.join();
float f;
for (int i = 0;i<1000000;i++)
{
f=ii2;
TimeUnit.MICROSECONDS.sleep(20*1000); // 200 milliseconds
}
wU.leave();
wU.delete(); // end the workload
}
wlmToken.disconnect();
}
}

The WLM statements are explained below.

WorkloadManager wlmToken = new WorkloadManager(“JES”, “SM3”);

This connects to the Work Load Manager and returns a connection token.    This needs to be done once per JVM.  You can use any relevant subsystem type, I used JES, and a SubsystemInstance (SI) of SM3. As a test, I created a new  subsystem category in WLM called DOG, and used that.  I defined ServerInstance SI with a value of SM3 within DOG and it worked.

z/OS uses uses subsystems such as JES for jobs submitted into JES2, and STC for Started task.

ServerClassification serverC = m.createServerClassification();

If your application is going to classify the transaction to determine the WLM service class and reporting  class you need this.  You create it, then add the classification criteria to it, see the following section.

Internally this passes the connection token wlmToken to the createServerClassification function.

serverC.setTransactionName(“TCI3”);

This passes information to WLM to determine the best service class and reporting class.  Within Subsystem CAT, Subsystem Instance SM1, I had a sub rule TransactionName (TN) with a value TCI3.  I defined the service class and a reporting class.

WorkUnit wU = new WorkUnit(serverC, “MAINCP”);

This creates the Independent (business transaction) enclave.  I have not see the value MAINCP reported in any reports.   This invokes the C run time function CreateWorkUnit(). The CreateWorkUnit function requires a STCK value of when the work unit started.  The Java code does this for you and passes the STCK through.

wU.join();

This connect the current task to the enclave, and any CPU it uses will be recorded against the enclave. 

wU.leave();

Disconnect the current task from the enclave.  After this call any CPU used by the thread will be recorded against the address space.

wU.delete();

The Independent enclave(Business transaction) has finished. WLM records the elapsed time and resources used for the business transaction.

m.disconnect();

The program disconnects from WLM.

Reporting class output.

I used RMF to print the SMF 72 records for this program.   The Reporting class for this program had

-TRANSACTIONS--  TRANS-TIME HHH.MM.SS.FFFFFF 
AVG        0.29  ACTUAL                36320 
MPL        0.29  EXECUTION             35291 
ENDED       998  QUEUED                 1028 
END/S      8.31  R/S AFFIN                 0 
#SWAPS        0  INELIGIBLE                0 
EXCTD         0  CONVERSION                0 
                 STD DEV               18368 
                                             
----SERVICE----   SERVICE TIME  ---APPL %--- 
IOC           0   CPU   12.543  CP      0.01 
CPU       10747   SRB    0.000  IIPCP   0.01 
MSO           0   RCT    0.000  IIP    10.44 
SRB           0   IIT    0.000  AAPCP   0.00 
TOT       10747   HST    0.000  AAP      N/A 

From this we can see that for the interval

  1. 998 transactions ended.  (Another report interval had 2 transactions ending)
  2. the response time was an average of 36.3 milliseconds
  3. a total of 12.543 seconds of CPU was used.
  4. it spent 10.44 % of the time on a ZIIP.
  5. 0.01 % of the time it was executing ZIIP eligible work on a CP as there was no available ZIIP.

Additional functions.

The functions below

  • ContinueWorkUnit – for dependent enclave
  • JoinWorkUnit – as before
  • LeaveWorkUnit – as before
  • DeleteWorkUnit – as before

can be used to record CPU against the dependent (Address space) enclave.  There is no WLM classify for a dependent enclave.

Java threads and WLM

A common application pattern is to use connection pooling.  For example the connect/disconnect to a database or MQ is expensive.  If you have a pool of threads, which connect, and start connected, an application can request a thread and get a thread which has already been connected to the resource manager.

It should be a simple matter of changing the interface from

connectionPool.getConnection()

to

connectionPool.getConnection(WorkUnit wU)
{ connection = connectionPool.getConnection()
 connection.join(wU)
}

and add a connection.leave(wU) to the releaseConnection.

Using enclaves in a C program -capturing elapsed and CPU time used by a transaction.

On z/OS enclaves allow you to set the priority of business transactions within your program, and to record the CPU used by the threads involved in the transaction – even if they are in a different address space.

Many of the WLM functions are assembler macros which require supervisor state. There are C run time functions which do some of the WLM functions. There are also Java methods which invoke the C run time functions.

Not all of the WLM functions are available in the C run time environment. You can set up enclaves, but the application cannot query information about the enclave, such as total CPU used, or the WLM parameters.

Minimal C program

The minimal program is below, and the key WLM functions are explained afterwards.

#include <sys/__wlm.h>
int main(void) { 
  wlmetok_t  enclavetoken; 
  server_classify_t   classify; 
  long rc; 
  // 
  //  Connect to work manager 
  // 
  unsigned int  connectToken = ConnectWorkMgr("JES","SM3");
  classify = __server_classify_create( ); 
  // pass the connection token to the classify
  //  This is needed but not documented 
  rc = __server_classify(classify, 
                 _SERVER_CLASSIFY_CONNTKN, 
                (char *) connectToken 
       ); 
  rc = __server_classify(classify, 
                         _SERVER_CLASSIFY_TRANSACTION_NAME, 
                         "TCI2" 
  for ( int loop=0;loop < 1000 ;loop++) 
  { 
    rc= CreateWorkUnit(&enclavetoken, 
                       classify, 
                       NULL, 
                       "COLINS"   ); 
    rc = JoinWorkUnit(& enclavetoken); 
 
    // do some work to burn some CPU 
    for ( int i = 0;i< 100000 ;i++) 
    { 
      double xx = i/0.5; 
      double yy = xx * xx; 
    } 
    rc = LeaveWorkUnit(&enclavetoken); 
    rc = DeleteWorkUnit(&enclavetoken);   
    rc = DisconnectServer(&connectToken);
}
 

What are they key functions?

unsigned int connectToken = ConnectWorkMgr(“JES”,”SM3″); 

This creates a connection to WLM, and uses the subsystem JES, and subsystem name of SM3.  Note: On my system it is JES, not JES2.   The WLM dialogs, option 6. Classification Rules list the subsystems available.  You can browse a subsystem type and see the available definitions.  I had

         -------Qualifier--------                 -------Class--------  
Action   Type Name     Start        Service     Report   
 ____  1 SI   SM3      ___          TCI1SC      THRU
 ____  2   TN  TCI3    ___          TCI1SC      TCI3
 ____  2   TN  TCI2    ___          TCI1SC      TCI2

server_classify_t  classify = __server_classify_create( );

CreateWorkUnit, the function used  to create an independent enclave (business transaction), needs to be able to classify the transaction to determine what service class (priority) to give the enclave.  This request sets up the classify control block.

rc = __server_classify(classify, _SERVER_CLASSIFY_CONNTKN, (char *)&connectToken );

The documentation does not tell you to pass the connection token.  If you omit this step the CreateWorkUnit fails with error code errno2=0x0330083B.

The __server_classify expects a char * as the value, so you have to use (char *) & connectionToken.

rc = __server_classify(classify, _SERVER_CLASSIFY_TRANSACTION_NAME, “TCI2” );

This tells WLM about the transaction we want to use.  TRANSACTION_NAME matches up with TN above in the WLM definitions.  This says the business transaction is called TCI2.  There are other criteria such as userid, plan or LU. See here for the list.   The list is incomplete, as it does not support classifiers like Client IP address which is available with the assembler macros.

rc= CreateWorkUnit(&enclavetoken, classify,  NULL, “COLINS” );

This uses the classification parameter defined above, to create the independent enclave, and return the enclave token. 

The documentation for CreateWorkUnit says you need to pass the arrival time,  Address of a doubleword (unsigned long long) field that contains the arrival time of the work request  in STCK format.  I created a small assembler function which just returned a STCK value to my C program.  However I passed NULL and it seemed to produce the correct values  – I think CreateWorkUnit does a STCK for you.

You pass in the name of the function(“COLIN”).  The only place I had seen this is if you use the QueryWorkUnitClassification() to extract the classify information.  For example QueryWorkUnitClassification gave a control block with non empty fields   _ecdtrxn[8]=TCI2 , _ecdsubt[4]=JES , _ecdfcn[8]=COLIN  , _ecdsubn[8]=SM3 . This function does not return the report class or service class.

rc = JoinWorkUnit(& enclavetoken);

This cause any work this TCB does to be recorded against the enclave.

rc =LeaveWorkUnit(&enclavetoken);

This stop work being be recorded against the enclave.  Subsequent work gets charged to the home address space.

rc= DeleteWorkUnit(& enclavetoken);

The business transaction has finished.  Information about the response time and CPU used are stored.

rc =  DisconnectServer(&connectToken);

This disconnects from WLM.

Using a subtask

I had a different thread n the program which did some work for the transaction. Using the enclave token, this work can be recorded against the transaction using

// The enclave token is passed with the request
rc = JoinWorkUnit(&enclaveToken); 
do some work...
rc = LeaveWorkUnit(&enclaveToken);  

This would be useful if you are using a connect pool to connect to MQ or DB2 subsystem. You have a pool of threads which have done the expensive connect with a particular userid, and the thread is used to execute the MQ or DB2 subsystem as that userid.

Other function available

Other functions available

Dependent (address space) enclave

The above discussion was for an business transaction, know in the publications as an Independent enclave.   An address space can have a Dependent enclave where the CPU is recorded as “Dependent Enclave” within the address space.  You use the function ContinueWorkUnit(&enclave) to return the enclave token.    You then use JoinWorkUnit and LeaveWorkUnit as before.  I can not see why you might want to use this.

Display the classification

You can use the QueryWorkUnitClassification to return a structure for the classification.

Reset the classification.

If you want to classify a different transaction, you can use server_classify_init() to reset the structure.

Set up a server

You can set up a server where your application puts work onto WLM queues, and other threads can get work.   This is an advanced topic which I have not looked into.

Make your enclave visible across the sysplex

You can use ExportWorkUnit  and ImportWorkUnit to have your enclave be visible in the sysplex.

Query what systems in the sysplex are running in goal mode.

You can use QueryMetrics() to obtain the systems in the sysplex that are in goal mode. This includes  available CPU capacity and resource constraint status. 

What is not available to the C interface

One reason why I was investigating enclaves was to understand the enclave data in the SMF 30 records.  There is an assembler macro IWMEQTME which returns the CPU, ZIIP and ZAPP times, used by the independent enclaves.  Unfortunately this requires supervisor state.    I wrote some assembler code to extract this and display the data.  Another complication is that the IWLM macros are AMODE 31 – so it did not work with my 64 bit C program.

Understanding z/OS Connect SMF 120 subtype 11 data

z/OS Connect can provide two types of SMF record

  1. SMF 120 subtype 11, provided by the base Liberty support. This gives information on the URL used to access Liberty, and the CPU used to perform requests. This is enabled at the Server level – so you can have records for all request, or no requests. There is one SMF record for each web server request. Would I use this to report CPU used ? No – see the bottom of this blog.
  2. SMF 123 provides information about API and the service used, and the “pass through” services. It provides elapsed time of the request, and of the the “pass through” requests. It does not provide CPU usage figures. This can be configured to produce records depending on the http host and port used to access z/OS Connect. One SMF record can have data for multiple web server requests. The SMF records are produced when the SMF record is full – or the server is shut down.

The SMF 120-11 and SMF 123 records are produced independently, and there is no correlating field between them. They both have a URI field, and time stamps, so at low volumes it may be possible to correlate the SMF data.

I’ll document the fields which I think are interesting. If you think other fields are useful please let me know and I’ll update this document.

I have written an SMF formatter in C which prints out interesting data, and summarises it.

SMF 120-11

  • You get the standard date and time the record was produced, and with the LPAR. You can use PGM=IFASMFDP with the following to filter which records are copied
DATE(2020282,2020282)
START(1000)
START(2359)
  • There is server version (3), system(SOW1), and job id(STC04774) which are not very interesting
  • Server job name(SM3) is more interesting. I started the server with s baqstrt,parms=’MQTEST’,jobname=sm3
  • The config dir (/var/zosconnect/servers/MQTEST/) is boring – as is code level (20.0.0.6)
  • The start and stop times (2020/10/08 09:18:19.852360 and 2020/10/08 09:18:22.073946) are interesting as is the duration – which is the difference between them.
  • You get userid information.
    • I used a certificate to authenticate. The DN from the certificate is not available. You only get the userid from the RACF mapping of DN to userid. This mapped userid was in the 64 byte field. The 8 byte userid field was empty for me. The lack of certificate DN, and having the userid in the wrong field feels like a couple of buglets.
  • You get the URL used /stockmanager/stock/items/999999 I treat this as a main key for processing and summarising data. If you want to summarise the data, you may want so summarise it just on /stockmanager/stock/. The full URI contains the part number – and so I would expect a large number of parts.
  • You can configure your requests to WLM. For example
<wlmClassification>
<httpClassification transactionClass="TCI1" method="GET" 
    resource="/zosConnect/services/stockQuery"/>
</wlmClassification>

This produced in the SMF record

WLMTRan :TCI1
WLM Classify type :URI :/zosConnect/services/stockQuery
WLM Classify type :Target Host :10.1.3.10
WLM Classify type :Target Port :19443

This means that the URL, the host, and the port were passed to WLM to classify.

If you get the WLM classification you also get CPU figures when the enclave request ended (was deleted).

  • You get the ports associated with the request.
    • Which port was used on the server – Target Port :9443
    • Where did the request come from? Origin :10.1.1.1 and port :36786
  • The number of bytes in the response Response bytes :791
  • CPU figures for the CPU used on the TCB. See discussion below on the usefulness of this number. You get the CPU figures before the request, and after the request – so you have to calculate the difference yourself! The values come from the timeused facility. You can calculate the delta and get
    • CPU Used Total : 0.967417
    • CPU Used on CP : 0.026327
    • and calculate these to to get CPU Delta. on Z**P : 0.941090 This is the CPU offloaded to ZIIP or ZAAP.
  • If you had the URI classified with WLM, you get Enclave data, see below for a discussion on what the numbers mean.
    • Enclave CPU time : 0.148803
    • Enclave CPU service : 0.000000
    • Enclave ZIIP time : 0.148803
    • Enclave ZIIP Service : 0.000000

What do the CPU numbers mean?

Typically a transaction flow is as follows

  1. A listening thread listens on the HTTP(s) host and port.
  2. When a request arrives, it passes the request to a worker thread, and goes back to listening
    1. The worker thread may do some work and send the response back
    2. The worker thread may need to call another thread to do some work. For example to issue an MQ request,
      1. the MQ code looks for a thread in a pool for a matching queue manager and userid. If it find one it uses it the thread and issues the MQ request.
      2. If it does not find a matching thing thread it may allocate a new thread, and issue an MQCONN to connect to MQ. These are both expensive operations, which is why having a pool of threads with queue manager and userid is a good way of saving CPU
      3. The work is done
      4. The thread is put back into the MQ pool, and the application returns to the worker thread
      5. The worker thread sends the response back to the originator

A thread can ask the operating system, how much CPU time it(the thread) has used. What usually happens is

  1. the thread requests how much CPU it has used
  2. the thread does some work
  3. the thread requests how much CPU it has used,
  4. the thread calculates the difference between the two CPU values and reports this delta.

I the SMF 120 record records the CPU from just the worker thread – and no other thread.

Enclaves

When there are more than one thread involved it gets more complex, as you could have a CICS transaction issuing an MQ request, then a DB2 request, and then an IMS request. You can set up z/OS WorkLoad Manager(WLM) to say “these CICS transactions in this CICS region are high priority”.

With some subsystems you can pass a WLM token into a request. The thread being invoked call tell WLM that the thread is now working on behalf of this token. The thread does some work, and tells WLM that it has finished doing the work. WLM can manage the priority of the threads to achieve the best throughput, for example making the thread high or low priority. WLM can manage a thread doing work in multiple LPARs across a sysplex!

WLM records the CPU used by the thread while performing the work, accumulates and reports this.

This use of multiple threads for a business transaction across one or more address spaces is known as an enclave.

What happens with enclaves?

  1. A request arrives at the listener thread.
  2. The Liberty looks up the URI in the <wlmClassification httpClassification…. It compare the server’s host, server’s port, the URI resource /stockmanager… method ( GET) and finds the best match for the transactionClass.
    1. If there is a transactionClass,
      1. the server calls WLM with the Subsystem type of CB, the specified collectionName, and the transactionClass.
      2. WLM looks for these parameters and if WLM has a matching definition then WLM will manage the priority of the work,
      3. WLM returns a WLM token.
      4. This WLM token is passed to threads which are set up for enclaves.
    2. If there is no transaction class specified in Liberty, or WLM does not have the subsystem, collectionname, transactionClass then there is no token or a null WLM token
    3. The work continue as before.
    4. If another thread is used then pass the WLM token. If the code is set up for WLM token then report “work started”, when it has finished report “work ended”

What happens if the request is not known to WLM.

The worker thread calculates the CPU used for just its work, and reports this. The CPU used by any other thread is not report. The figures reported are the CPUTotal timeused values. You have to calculate the difference yourself

What happens if the request is known to WLM.

You get the timeused CPU for the worker thread – as with the case where the request is not known to WLM.

From RMF (or other products) you get out reports for an interval with

  1. The number of requests in the interval
  2. The rate of requests in the interval
  3. The amount of time on a CP engine in seconds
  4. The amount of time on a ZIIP engine is seconds
  5. The amount of time on a ZAAP in seconds.
  6. Over the interval, what percentage of time was CP on CP engines, zAAP on zAAP engines, zAAP on CP engines, zIIP on zIIP engines.

From the SMF 120 records you get

Enclave CPU time
Enclave ZAAP time
Enclave ZIIP time

Example Enclave figures.

For 100 API requests, the figures as reported by SMF 120-11, and I averaged the values.

  1. Average CPU(1) 0.023
  2. Average CPU(2) 0.0008
  3. Enclave CPU 0.029
  4. Enclave ZAAP 0
  5. Enclave ZIIP 0.028

The figures reported by RMF per request

  1. CPU 0.031
  2. ZIIP 0.039
  3. ZAAP 0.000
  4. Total 0.070 seconds of CPU per transaction

These figures tie up – the Enclave CPU, ZIIP, and ZAAP are similar.

The CPU used by the server address space was

  1. CPU 30.1 seconds
  2. ZIIP 28.7 seconds
  3. ZAPP 0 seconds.
  4. Total 58.8.

Each request took 0.070, and there were 100 requests – so reported 7 second of CPU.

The difference(51) seconds is not reported in the transaction costs. It looks like the “timeused” value is less than 1% of the CPU value, and the enclave figures are under 2% of the grand total.

Looking at the trace in a dump, I can see many hot TCBs using much more CPU that is reported by WLM and RMF. I expect that many TCBs used in a request, but they do not have the enclave support in them. Overall – pretty useless for charge back and understanding the cost per transaction.

HOME sweet HOME – understanding TCP/IP home statements

I was just(!) trying to get my Liberty web server running on z/OS to be able to be moved to a different LPAR, and get it working.  Moving it was easy, but the server’s certificate needs the IP address of the TCP/IP stack – with RACF you can only have one “Subject Alternative Name”.  A SAN of IP:10.1.2.4 works fine when it comes from TCP/IP stack 10.1.2.4 – but not from TCP/IP stack 10.1.2.5.   The web browser checks, and complains if they do not match.

To get this to work I read the z/OS TCP/IP documentation. There is lots of it, but it seems to be written for people who are experts in it.   There is a saying “Question: how do you eat an elephant?  Answer: A bit at a time”.   This post takes a small bit area – and expands it in terms I understand.  It may not be accurate – but the concepts should be right.

What is a TCP/IP stack?

This is another name of a TCP/IP instance, a started address space.

What is HOME?

Each connection coming into to a TCP/IP instance has an IP address. On TCPIP1 I have a connection (a virtual bit of wire) defined for IP address 10.1.1.2

On Linux if I use the command ip route get 10.1.1.2 it says

10.1.1.2 dev tap0 src 10.1.1.1 uid 1000

So 10.1.1.2  is going via device tap0 (which is a tunnel device, TAP = Tunnelling Application Protocol?).   The Linux machine has IP address 10.1.1.1.  Through some configuration magic this ends up in my TCP/IP instance as

DEVICE PORTA MPCIPA
LINK ETH1 IPAQENET PORTA
HOME 10.1.1.2 ETH1

Where 10.1.1.2 is the address for a link called ETH1 on the TCPIP instance on my LPAR. The magic is a bit like the Negro spirtual song Dem Bones which has “Thigh bone connected to the hip bone,  Hip bone connected to the back bone, Back bone connected to the shoulder bone Now hear the word of the Lord”.   ETH1 is defined as  being on PORTA, and PORTA is a device which maps to a tunnelled device using protocol MPCIPA.   …  maps to a VTAM TRL definition, Now hear the word of the Lord.

I can use the TSO netstat command for TCP address space called TCPIP1  netstat home tcp tcpip1

Home address list:
Address       Link        Flg
-------       ----        ---
10.1.1.2      ETH1        P
192.168.0.61  ETH2
10.1.1.5      EZASAMEMVS
127.0.0.1     LOOPBACK

For external connections the IP address, 10.1.1.2, must match with the definition on Linux.  This may be pointed to from outside Linux.  The other “home” connections are described below.

I have multiple instances working together.

I have three TCP/IP instances on my LPAR.  You might have an instance to talk to your internal network,(the intranet), and an instance talking to the internet, facing out from your enterprise.

You can also have TCP/IP instances with different security profiles, and provide total isolation.

You can set up connections between your enterprise and my enterprise,  these definitions will need a DEVICE, and a LINK etc as above (or an INTERFACE definition).

Setting up multiple instances within a Sysplex, or within an LPAR.

In your TCP/IP definitions you can set up

IPCONFIG DYNAMICXCF 10.1.1.6 255.255.255.0 2

and another instance with 10.1.1.5 etc.  The DYNAMICXCF says this is within the Sysplex(LPAR).  The software is smart enough to generate the device and link statements automatically.  In the netstat home command above, it gave 10.1.1.5 EZASAMEMVS which is eza_SAME_ MVS, and it has found a second TCP instance in the same LPAR.  See here for dynamic XCF, and here for the IPCONFIG statement.

I think you can use any unused IP address range; so you could use 2.2.2.5 and 2.2.2.6 instead of 10.1.1.5 and 10.1.1.6.  I believe these address are only used within the Sysplex.  So as long as these addresses are consistent and not being used else where, the values are not critical.

For a TCP/IP instance TCPIP3 on my LPAR with no external connections netstat home tc tcip3 gave me

Home address list:
Address   Link       Flg
-------   ----       ---
10.1.1.7  EZASAMEMVS P
127.0.0.1 LOOPBACK

This has the ever present LOOPBACK, and a virtual connection 10.1.1.7 to a TCP/IP instance in the same LPAR because of the ezaSAMEmvs definition

A practical path to installing Liberty and z/OS Connect servers – 10 use the MQ sample

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Use the MQ sample

You need to have installed the service, and protected it.

You need to configure MQ to include the MQ support and tell JMS where the libraries are

<server> 
<!-- Enable features --> 
<featureManager> 
    <feature>zosconnect:mqService-1.0</feature> 
</featureManager> 
                                                                                                         
<wmqJmsClient nativeLibraryPath="/Z24A/usr/lpp/mqm/V9R1M1/java/lib"/> 

<variable name="wmqJmsClient.rar.location"
   value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/jca/wmq.jmsra.rar"/> 
</server> 

You could configure a variable for the mq directory so you use it once, and use

<variable name="wmq"  value="/Z24A/usr/lpp/mqm/V9R1M1/java/lib/"
<wmqJmsClient nativeLibraryPath="${wmq}”/>
<variable name="wmqJmsClient.rar.location"
   value="${wmq}wmq.jmsra.rar}">

You could also pass the mq location as a variable in STDENV, and so pass it in through JCL.

Configure the jms to define the queue manager and queues

<server> 
 <jmsConnectionFactory jndiName="jms/cf1" 
     connectionManagerRef="ConMgr1"> 
    <properties.wmqJms transportType="BINDINGS" 
         queueManager="CSQ9"/> 
 </jmsConnectionFactory> 
                                                                                                      
 <connectionManager id="ConMgr1" maxPoolSize="5"/> 
                                                                                                      
 <!-- A queue definition where request messages 
      for the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockRequestQueue"> 
    <properties.wmqJms 
       baseQueueName="STOCK_REQUEST" 
       targetClient="MQ"/> 
 </jmsQueue> 
                                                                                                      
 <!-- A queue definition where response messages from 
      the stock query application are sent. --> 
 <jmsQueue jndiName="jms/stockResponseQueue"> 
    <properties.wmqJms baseQueueName="STOCK_RESPONSE" targetClient="MQ"/> 
 </jmsQueue> 
</server>

and include these in the server.xml file.

You need to compile and run the back end service.  See here.  Take care if using cut and paste as there a long lines which wrap, and cause compilation errors.

Because the MQ path name is long I used

export HLQ="/usr/lpp/mqm/V9R1M1/java/lib"

java -cp $HLQ/com.ibm.mq.allclient.jar:. -Djava.library.path=$HLQ TwoWayBackend CSQ9 STOCK_REQUEST STOCK_RESPONSE

I set up a job to run this in back ground, so I could free up my TSO terminal.

Use the API

Once installed you should be able to use the API. For example

curl –insecure -i –cacert cacert.pem –cert adcdd.pem:password –key adcdd.key.pem https://10.1.3.10:9443/stockmanager/stock/items/999999

If the back end application was working I got

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

If the back end application was not working I got back an empty response.

The back-end application runs until Ctrl+c is used.  On my USS, the ESCape key is cent symbol ¢ (Unicode 00a2) which I do not have on my default keyboard.    See  x3270 – where’s the money key? for guidance on how to set it.

 

Use the Service

To use the API I used a web browser with

https:10.1.3.10/9443/stockmanager/stock/items/999999

and got back

{"SQRESP":{"ITEM_ID":999999,"ITEM_DESC":"A description. 00050","ITEM_COST":45,"ITEM_COUNT":0}}

or curl with

 curl --insecure -X post -H Content-Type: application/json
  -H "Content-Type:application/json" -i --cacert cacert.pem
  --cert adcdd.pem:password --key adcdd.key.pem 
  --data {"STOCKQRYOperation": {"sqreq" : { "item_id": 2033}}}
  https://10.1.3.10:9443/zosConnect/services/stockQuery?action=invoke

A practical path to installing Liberty and z/OS Connect servers – 11 WLM classifying a service

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Classify the services with WLM to give them the right priority

You can configure an API or service, for example https://10.1.3.10:9443/stockmanager/stock/items/999999, so it gets an appropriate service class in WLM, for example high priority.

You can use RMF to report on service classes to see the response time profile, and if the service class is meeting its performance goals.

See here for a good article.

This page gives the example

<wlmClassification>
  <httpClassification transactionClass="CLASS001" 
      host="127.0.0.1" 
      port="9080" 
      method="GET"
      resource="/testResource" />
</wlmClassification>

You can classify the traffic depending on the IP address and port of the server, as well as the resource name.

A practical path to installing Liberty and z/OS Connect servers – 3 Initial customisation

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Copy the PDS from the install system to where they will run.

These files can be sent using FTP or XMIT.

Copy the product ZFS to where it will be used.

You can use DFDSS to dump the ZFS and move it to the SYSPLEX where it will be used.

  1. Mount the file system read/write for example /usr/lpp/IBM/zosconnect/v3r0beta
  2. Perform Setting up the product extensions directory.  You need to do this on each LPAR, as it set up files in /var/zosconnect
  3. Mount the file system READ
  4. Change the BPXPRMxx parmlib member so the file sytsem is mounted READ it at IPL
  5. You may want to create an alias /usr/zosc to the product executables.
    • ln -s /usr/lpp/IBM/zosconnect/v3r0beta/bin /usr/zosc

Create the angel started task if needed.

If you will be using an existing Angel you will need to set up RACF permissions.

Follow the instructions the z/OS Connect  documentation

  • If you need a named Angel, specify the value in the NAME=value.  You can specify NAME when you start the task.
  • You do not need to classify this in WLM as it uses little CPU once it has been started.

Create the file system for the server

  1. See Creating the z/OS Connect EE shared directory. You will need to know the path for server’s home directory. The path /var/zosconnect can only be used on the LPAR.   The files cannot be shared.
  2. If you need to allocate a file system.
    1. Allocate a ZFS. If the initial allocation of the ZFS is too small, then the ZFS will grow on demand, or using the zfsadm grow command to expand it.
    2. Mount the file system over the directory.
    3. Change the BPXPRMxx parmlib member so the file system is mounted RW at IPL.

Create the Z/OS Connect started task

See Creating a z/OS Connect Enterprise Edition server.

Using /usr/zosc alias defined above

export  WLP_USER_DIR=/var/zosconnect
/usr/zosc/zosconnect create default --template=zosconnect:default

If you need to start again, use the following to remove the server.

rm -r  /var/zosconnect/servers/default
  1. Copy BAQSTRT to the SYS1.PROCLIB concatenation, to the name you are going to use, BAQSTRT, BAQSTRT1 etc
  2. When you edit the member, use caps off to use lower case text in the member.
  3.  If you will be using MQ, add the MQ libraries to STEPLIB
    //STEPLIB   DD DSN=CSQ913.SCSQANLE,DISP=SHR 
    //          DD DSN=CSQ913.SCSQAUTH,DISP=SHR
  4. You may find it easier to remove the _CEE_RUNOPTS= and add
    //CEEOPTS DD DISP=SHR,DSN=...(CEEOPTS)

    to the JCL. This makes it easier to add CEE options, and specify storage heap and pool sizes.

  5. In my CEEOPTS I use RPTSTG(ON), to report the storage pool usage. This provides useful information which may be useful when investigating performance problems.
  6. Despite what the Liberty documentation says you do need
      • JVM_OPTIONS=<Optional JVM parameters>
    • for example
      • JVM_OPTIONS=-Xoptionsfile=/var/zosconnect/servers/default/jvm.options
  7. You may want a JVM specific to a server, as you may want to set some service specific options such as TLS trace.
  8. You may want to specify a default server name in the JCL parms. You can specify a different one when z/OS Connect is started.
  9. Specify WLP_USER_DIR for your server’s directory. You could make this as a parameter to the started task JCL if you want to provide isolation for the servers, and give each server its own file system.

Create the JVM options file

Go to the server’s directory.

cd /var/zosconnect/servers/default

use oedit jvm.options to create an EBCDIC file.

add

-Dcom.ibm.ws.zos.core.angelRequired=true 
#-Dcom.ibm.ws.zos.core.angelName=MYZANGL

The -Dcom.ibm.ws.zos.core.angelRequired=true option stops the server from starting if the angel process is not running. You get message:

CWWKB0116I: This server is not registered with an angel process even though it is configured to require registration with an angel process. This server is attempting to stop.

The option -Dcom.ibm.ws.zos.core.angelName=MYZANGL allows you to specify a non default angel process name. The # in front of the statement makes it into a comment.

add the following, if required, with your time zone.

-Duser.timezone=Europe/London

Override TCPIP

If you are using a non standard TCPIP, edit server.env and add

_BPXK_SETIBMOPT_TRANSPORT=TCPIP2

Tailor the server.xml file

It was easy to get my configuration options mixed up, and hard to compare two server configurations. I found it much easier to manage my configuration by putting the components into their own xml file. This also makes having shared components much easier to implement- as you just include the name of the shared file.

The configuration files have to be in ASCII, and you have to take care ensure the files are in the correct code page.

There are at least three ways of creating the .xml files (I use the third option through ISPF 3.17)

  1. using omvs commands to copy and edit existing ASCII files
  2. use omvs commands to create ASCII files
  3. use ispf option 3.17

1 Using omvs commands to copy an existing ascii file,  then edit it it

For example

cp server.xml httpEndpoint.xml

edit the httpEndpoint.xml to keep the <server> ,httpEndpoint information, </server> and delete the rest.  The file should look like

<server description="new server"> 
    <httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" /> 
</server>

then edit the server.xml and replace

<httpEndpoint id="defaultHttpEndpoint" 
                  host="*" 
                  httpPort="9080" 
                  httpsPort="9443" />

with

<include location="${server.config.dir}/httpEndpoint.xml"/> 

Note: if you use the edit command source ? It should say ASCII already set.

2 Use OMVS to created and edit files

By default, if you edit a file in OMVS it becomes an EBCDIC file.

If you create a file without copying it, you need to use

touch httpEndpoint.xml 
chtag -t -c ISO8859-1 httpEndpoint.xml     
oedit httpEndpoint.xml

Make the same changes as above.

3 Use ispf option 3.17

If you use ISPF 3.17 (z/OS UNIX Directory List) you have more flexibility with editing files.

Edit server.xml

use the line commands mmmm and the primary command create to create a new member, give the new name when prompted, for example httpEndpoint.xml

Replace the lines you copied out with

	<include location="${server.config.dir}/httpEndpoint.xml"/>

and edit the file you just created to put <server> … </server> around the content.

Add more logging to the JCL’s STDOUT.

The default logging to the jobs STDOUT is only audit records. I found it much easier to include other levels of information so I did not have to keep looking in the message.log file. Note: the STDOUT content does not have time stamps so you do not know when the output was produced- so it is easier to use – but has less information!

I created logging.xml with the following content.

<server> 
 <logging 
 consoleLogLevel="INFO" 
 /> 

 <!-- this is a comment where I store trace information
   traceFormat="BASIC" traceSpecification="*=info:
       com.ibm.ws.security.*=fine:zos.native.03=fine" 
  end comment   -->  
</server>

I added

<include location="${server.config.dir}/logging.xml"/> 

to the server.xml file.

The consolLogLevel=”INFO” writes additional information to the STDOUT file, than the default AUDIT value which just outputs AUDIT records.

Define the http address and ports

Edit the httpEndpoint definitions

    <httpEndpoint id="defaultHttpEndpoint" 
                  accessLoggingRef="ALR" 
                  host="*" 
                  httpPort="9081" 
                  httpsPort="9443" /> 
    <httpAccessLogging 
       enabled="true" 
       id="ALR" 
       logFormat="%{c}a! %h %u %{t}W %q %r %s %b" 
   /> 

Check

  • host matches the TCPIP value to use. With my VIPA I specify the VIPA address host=”10.1.3.10”
  • httpPort – if you already have a Liberty instance running port 9081 will be in use by another server
  • httpsPort – if you already have a Liberty instance running port 9443 will be in use by another server

You will need the port values when you use the server.

The httpAccessLogging statement pointed to by accessLoggingRef, will create a log file with every attempt to use the server.

Example output

10.1.1.1! 10.1.1.1 ADCDC {12/Sep/2020:16:07:17 +0000} - 
GET /zosConnect/apis/stockmanager HTTP/1.1 200 296

Where 10.1.1.1 is the source IP address and ADCDC is the userid used.

By default this file is http_access.log in the …/logs directory.

See here for the format of the logFormat statement.

After tailoring

I did more packaging the content into include .xml files.

After my “repackaging” my server.xml file looked like

<?xml version="1.0" encoding="UTF-8"?> 
<server description="new server"> 
    <include location="${server.config.dir}/applicationMonitor"/> 
    <include location="${server.config.dir}/logging.xml"/> 
    <include location="${server.config.dir}/httpEndpoint.xml"/> 
    <include location="${server.config.dir}/cors.xml"/> 
    <include location="${server.config.dir}/config.xml"/> 
    <include location="${server.config.dir}/zosconect.xml"/>                                                                              
</server>

You can use ispf option 3.17 to list the files in the servers’s directory.

Once you have configured the server you can use the admin centre to manage the files.  This will require

<remoteFileAccess>  
    <writeDir>${server.config.dir}</writedir>
</remoteFileAccess>

Finally ! You can start the server.

A practical path to installing Liberty and z/OS Connect servers – 4 start the server

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Starting the server

Using the definitions above I started the server, using the default configuration directory, and used jobname=default.

s baqstart,parms=’default’,jobname=default

On my z/OS system, running on zPDT on Linux, it takes about 5 minutes to start.

During start up, check that the parameter have been accepted – with the correct case.

If you get configuration errors, fix them and use the

f ...zcon,refresh

command to get the server to pickup the changes. Note. Not all changes are picked up by the refresh command, and you have to stop and restart the server to pick up changes.

There is output in the STDOUT and possibly in STDERR, also in the /var/zosconnect/default/logs/messages.log

STDOUT will have messages like

[AUDIT] CWWKG0028A: Processing included configuration resource: 
/var/zosconnect/servers/default/httpEndpoint.xml

which tells you which configuration files you are using.

If you get any syntax errors you should fix them. The command f …,zcon,refresh  may cause it to reread the definitions.

The second +  start of server start  after  IPL may be faster than the first if Java has cached some of the executables.

Check for messages

Authorized service group … is available.

Which tells you that the Angel service is enabled.

Web application available (default_host): http://10.1.3.10:9080/

check this is what you expect. If the port is not valid check the httpEndoint.xml. You do not get an entry for the TLS port, because TLS is not enabled.

If you use a web browser to logon to the URL http:10.1.3.10:9081 (note:not https) you will just get a blank screen. If you look in logs/http_access.log you will see a code 302. In this case, 302 means you were trying to use an http session to a session that requires https.

If you use other URLs you may get a message saying you need to use https.

If you use z/OS explorer and use the http:…9081 above you will get “302, Found”. This means you were trying to use an http session to a session that requires https.

You have successfully created a z/OS connect server, and used it – even though you could not do much.

To stop the server use the z/OS P … command.

Most of the facilities in z/OS Connect require TLS, so you cannot do much until you have configured that.

A practical path to installing Liberty and z/OS Connect servers – 5 enable logon security

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Enable logon security

To enable security checking you need  to use the zosconnect_zosConnectManager tag.

See here for information, and here for the tag  syntax.

<zosconnect_zosConnectManager 
     requireAuth="true"  
     requireSecure="true" 
     setUTF8ResponseEncoding="true"/>

add to the zosconnect.xml or server.xml file.

You cannot do much with this enabled as you need TLS

Add in the SAF repository to use z/OS security services.

Within the server directory create a file called saf.xml. (I always used ISPF 3.17, and use cc…cc create.. to create a new file.

<server> 
                                                                                
  <featureManager> 
     <feature>transportSecurity-1.0</feature>   
     <feature>zosSecurity-1.0</feature> 
     <feature>appSecurity-2.0</feature> 
  </featureManager> 
  <authentication id="Basic" cacheEnabled="false" /> 
  <safAuthorization id="saf-authorization"  racRouteLog="ASIS" />                                                                                 

  <safRegistry id="saf" realm="myrealm"/> 
  <safCredentials profilePrefix="ZZZZDFLT" 
     suppressAuthFailureMessages="false" 
     unauthenticatedUser="WSGUEST"/> 
</server>

Where you need to specify the profile prefix (for example ZZZBFLT), and define the security profile.

I do not think the realm is used for SAF processing.

If you are planning on using a new profilePrefix  (YYYYDFLT) you will need to create several profiles.

RDEFINE APPL YYYYDFLT  UACC(NONE) 
PERMIT YYYYDFLT  CLASS(APPL) ACCESS(READ) ID(group1,group2)
SETROPTS raclist(APPL) refresh

RDEFINE server BBG.SECPFX.YYYYDFLT uacc(NONE)
permit  BBG.SECPFX.YYYYDFLT class(server) access(read) id(start1)
setropts raclist(server) refresh

rdefine ejbrole YYYYDFLT.zos.connect.access.roles.zosConnectAccess 
   uacc(none)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
   class(ejbrole) 
   id(Adcdc) access(READ)
permit YYYYDFLT.zos.connect.access.roles.zosConnectAccess  
    class(ejbrole) 
    id(group1,group2) access(READ)
setropts raclist(ejbrole) refresh

These profiles are only accessed once someone tries to use the server – not at startup, so try logging on using the https port.

Every one using the server needs read access to YYYYDFLT CLASS(APPL).

As an interim step towards getting your server up and secure, enable userid and password authentication.

<webAppSecurity overrideHttpAuthMethod="BASIC"/> 
<webAppSecurity allowFailOverToBasicAuth="true"/> 

overrideHttpAuthMethod=”BASIC” says used userid and password – and not client certificates.

allowFailOverToBasicAuth=”true” says if the client certificate logon fails – fail over to basic, userid and password.

Once you have configured the server, you can remove the overrideHttpAuthMethod tag, so client certificates are used.  Once these are working remove the allowFailOverToBasicAuth, to remove the userid and password option.  With these both removed, people will have to use client certificates to authenticate.

A practical path to installing Liberty and z/OS Connect servers – 6 Enabling TLS

Introduction

I’ll cover the instructions to install z/OS Connect, but the instructions are similar for other products. The steps are to create the minimum server configuration and gradually add more function to it.

The steps below guide you through

  1. Overview
  2. planning to help you decide what you need to create, and what options you have to choose
  3. initial customisation and creating a server,  creating defaults and creating function specific configuration files,  for example a file for SAF
  4. starting the server
  5. enable logon security and add SAF definitions
  6. add keystores for TLS, and client authentication
  7. adding an API and service application
  8. protecting the API and service applications
  9. collecting monitoring data including SMF
  10. use the MQ sample
  11. using WLM to classify a service

With each step there are instructions on how to check the work has been successful.

Configuring TLS

  1. You can configure the server to creates a keystore file on its first use. This creates a self signed certificate. This is good enough to provide encryption of the traffic. Certificates sent from the client are ignored as the trust store does not have the Certificate Authority certificate to validate them.
  2. You can use your site’s keystore and trust store. The server can use them to process certificate sent from the client for authentication.

Decide how you want to authenticate

Most of the functions require an https connection. This will require a keystore.

You can decide if

  1. The server uses the client’s certificate for authentication,
    1. if that does not work then use userid and password
    2. if that does not work, then fail the request; there is no fall back to userid and password.
  2. The server does not use the clients certificate.
    1. You can configure that userid and password will used for authentication
    2. There is no authentication

Have the server create a keystore.

You can get Liberty to create a keystore for you. This creates a self signed certificate and is used to encrypt the traffic between client and server. This is a good start, while you validate the set up, but is not a good long term solution.

Create keystore.xml with

<server>
<keyStore id="defaultKeyStore" password="${keystore_password}" /> 

<ssl clientAuthentication="false" 
    clientAuthenticationSupported="false" 
    keyStoreRef="defaultKeyStore" 
    id="defaultSSLSettings" 
    sslProtocol="TLSv1.2" 
/> 
</server>

Add to the bottom of the server.xml file

 <include location="${server.config.dir}/keystore.xml"/>

If you have keyStore id=”defaultKeyStore”, (it must be defaultKeyStore) and do not have a keystore defined, the the server will create the keystore in the default location (${server.output.dir}/resources/security/key.p12) with the password taken from the server.env file.  See here.

Restart the server.

I got the messages

CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started 
and is now listening for requests on host 10.1.3.10  
(IPv4: 10.1.3.10) port 9443.

Showing TLS was active, and listening on the 9443 port.

If the keystore was created, you will get messages like

[AUDIT   ] CWPKI0803A: SSL certificate created in 87.578 seconds. 
SSL key file: /var/zosconnect/servers/d3/resources/security/key.p12 
[INFO    ] Successfully loaded default keystore: 
/var/zosconnect/servers/d3/resources/security/key.p12 of type: PKCS12

The certificate has a problem (a bug). It has been generated with CN:localhost, O:ibm: ou:d3 where d3 is the server name. The Subject Alternative Name (SAN) is DNS:localhost. It should have a SAN of the server’s IP address (10.3.1.10 in my case).

Clients check the SAN and compare it with the server’s IP address.

  1. Chrome complain. “Your connection is not private NET:ERROR_CERT_AUTHORITY_INVALID”, and the option to accept it
  2. Firefox gives “Warning: Potential Security Risk Ahead”, and the option to accept it.
  3. Z/OS explorer gives a Server certificate alert pop up, saying “Host:10.1.3.10 does not match certificate:localhost” and gives two buttons Decline or Accept.
  4. With curl I got SSL_ERROR_SYSCALL.

You can accept it, and use it until you have your own keystores set up. You can also reset this decision.

Using a RACF keyring as the keystore.

You can use a file based keystore or a RACF keying.  Below are the definitions for my RACF keyrings. The started task userid is START1. The keystore (containing the private key for the server is keyring START1/KEY. The server should use key ZZZZ.

The trust store, containing the Certificate Authority certificates and any self signed certificates from clients, is START/TRUST.

The <ssl.. /> points to the different keystores, so it makes sense to keep all these definitions in one file.  You may already have a file of these definitions which you can use from another Liberty server.

<server>

<sslDefault sslRef="defaultSSLSettings"/> 
<ssl clientAuthentication="true" 
    clientAuthenticationSupported="true" 
    id="defaultSSLSettings" keyStoreRef="racfKeyStore"  
    serverKeyAlias="ZZZZ" 
    sslProtocol="TLSv1.2" 
    trustStoreRef="racfTrustStore"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfKeyStore" 
     location="safkeyring://START1/KEY" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/> 
                                                                                                                  
  <keyStore filebased="false" id="racfTrustStore" 
     location="safkeyring://START1/TRUST" 
     password="password" 
     readOnly="true" 
     type="JCERACFKS"/>                                                                                                                  
</server>

This sets clientAuthentication=”true” and clientAuthenticationSupported=”true”

Specify if you want to use a client certificate for authentication

If you specify clientAuthenticationSupported=”true”… the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.

The default keystore will not be able to validate any certificates sent from the client. When connecting to Chrome with certificates set up, I got an FFDC and messages

  • [INFO ] FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued by CN=SSCA8, OU=CA, O=SSS, C=GB is not trusted; internal cause is: java.security.cert.CertPathValidatorException:
  • [ERROR ] CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired.

If you specify clientAuthentication=”false” (the default) the server does not request that a client send a certificate during the handshake.

If you specify <webAppSecurity allowFailOverToBasicAuth=”true” />  the client certificate connection is not used or it fails,

  1. if  you specify<webAppSecurity allowFailOverToBasicAuth=”true” /> the user will be prompted for userid and password
  2. If you specify <webAppSecurity allowFailOverToBasicAuth= false > or not specified, the connection will fail.

If a userid and password can be used, the first time a browser uses the server it will be prompted for userid and password. As part of the handshake, the LTPA2 cookie is sent from the server. This has the userid and password encrypted within it. If you close down the browser and restart it (not just restart it from within the browser) you will be prompted again for userid and password. You can also be prompted for userid and password once the LPTA cookie has expired.

If you are using z/OS explorer and get a code 401, unauthorised, you may be using a certificate credential ( format userid@CertificateAuthority(CommonName)) rather than a userid and password with format of just the userid eg COLIN. Use “Set Credentials” to change credentials.

You can see what userid is being used for the requests, from the …/logs/http_access.log file.

To make it even more complex you can have different keystores for different connections or ports.  See here. But I would not try that just yet.

Map client certificates to a SAF userid

If you are using certificate authentication you will need to map the certificate to a userid using the RACDCERT MAP command.

Testing it

If the server starts successfully you can use a web browser with URL

  http:/10.1.3.10:9443/zosConnect/api-docs

and it should display json data.

If you get “Context Root Not Found” or code 404 you should wait and retry, as the https processing code is active, but the code to process the requests is not yet active.

Review the contents of …/servers/…/logs/http_access.log to see the request being issued and the http completion code.

If you have problems connecting clients over TLS add -Djavax.net.debug=ssl:handshake to the jvm.options file and restart the server.

If you connect to the z/OS Explorer, and logon to the z/OS Connect EE Server, you should have a folder for APIs and Services – which may have no elements.