Category: Liberty

Using the Java Health centre for looking into Z/OSMF, MQWEB and other Liberty products.

The Java Health centre has an agent running in the JVM of interest, and there is Eclipse plug-in to display the data.

A Java server such as Liberty ( as used in z/OSMF, z/OSMF and MQWEB) can provide information on how the server is running. I was running MQWEB with Openj9, Java 21 (Semeru).

You need to configure the Liberty server and have something to process the data such as Health Center running on Eclipse.

You can display information in graphical time line format, such as

  • CPU used, system and application as used by the JVM
  • Which classes are being used
  • The environment – such as the parameters used to start the JVM
  • Garbage collection activity
  • I/O – number of files open, and open activity
  • Method profiling
  • Threads in use.

Configure the Eclipse

I installed Health Center from the Market place.

How to collect the data

You can configure the JVM in different modes:

  • headless – data is collected and written to the local file system
  • collect from the start – and view in Eclipse, this means you get all of the Java class loading activity
  • start collecting only after Eclipse has started, and connected to the JVM. I use this method. I start my server, and run a workload to “warm up the JVM” then use Eclipse to show the activity due to my testing.

Configure the JVM server

The options are listed here.

You can specify the JVM options on the command line or the jvm.options file.

You can specify them on the -Xhealthcenter:… statement, or as

-Dcom.ibm.diagnostics.healthcenter...=...

values. For example

-Xhealthcenter:level=off,readonly=off,jmx=on,port=1972

or

-Xhealthcenter:level=off
-Dcom.ibm.java.diagnostics.healthcenter.agent.port=1972 
-Dcom.ibm.diagnostics.healthcenter.jmx=on
-Dcom.ibm.diagnostics.healthcenter.readonly=on

To run headless

In the server

I added the following to my jvm.options

-Xhealthcenter:level=headless 
-Dcom.ibm.java.diagnostics.healthcenter.headless.delay.start=2 
-Dcom.ibm.diagnostics.healthcenter.headless=on 
-Dcom.ibm.java.diagnostics.healthcenter.data.collection.level=headless 
-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/u/tmp/zowec/ 
-Dcom.ibm.diagnostics.healthcenter.readonly=on

Down load the files to your work station, and use File -> Load Data to process the files.

To run the Health centre in real time

In the server

-Xhealthcenter:level=off,readonly=off,jmx=on,port=1972 
-Dcom.ibm.diagnostics.healthcenter.logging.level=debug

Note the jmx=on and the port number. You need this for the Eclipse configuration. The level=off means do not start collecting data until the Health centre agent connects.

In Eclipse

File -> New Connection… -> Enable an application for monitoring -> Next.

On the Select connector panel I used

Once it worked, I enabled security.

Click Next

The Health Centre then starts searching at the specified port. I disable the Scan next 100 ports… When it manages to connect to the port, click Finish.

I initially had problems connecting to the server, see Why can’t I connect to a z/OS port?

It takes a few seconds to start the data collection, and start downloading the data.

Let the JVM warm up

The image below shows the CPU usage from the start of the server.

For the first 5 minutes, this is the JVM starting up with no workload. Afterwards the CPU used drops to a low value.

After 5 minutes, I started my workload. For the first 12 or so minutes the CPU is high, but after about 13 minutes it levels out. If you want to do any measurements of cost per transaction you should take them from this period. During the “warm up” period, the JVM is optimising the code etc.

The green line shows the system CPU usage. The red line (and grey area) shows the Application usage. We can see most of the CPU used is application usage.

The number of methods profiled is the JVM optimising the code. It takes the “hottest” classes and does those first… until all (most) of the classes are optimised.

Long term monitoring.

f

From this diagram you can see the JVM startup, the initial part of my test where the JVM was warming up, the remainder of the test, and the JVM overhead after the test.

You need to take all of these into consideration when running performance tests.

Running performance tests

I set up my Work Load Manager configuration to record the number of MQ transactions, and had a report class for the MQWEB server. From this I can calculate the cost per transaction.

Health centre agent logging

With

-Dcom.ibm.diagnostics.healthcenter.logging.level=finest

I had output in the STDERR output 

[06:51:52] com.ibm.diagnostics.healthcenter.Agent FINE: System receiver, version 1.0 
[06:51:52] com.ibm.diagnostics.healthcenter.Agent FINE: /usr/lpp/java/J21.0_64//lib/libhcapiplugin.so, version 1.0                                                                                                                    
[06:51:52] com.ibm.diagnostics.healthcenter.java FINE: Health Center Agent 4.0.7 
06:51:53com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean <init> 
INFO: Agent version "3.0.21.202109031203" 
06:51:56 com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent 
INFO: Health Center agent running in off mode. 
06:51:56 com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent 
INFO: Health Center agent started on port 1972.

and in STDOUT many 

com.ibm.lang.management.OperatingSystemMXBean.getTotalPhysicalMemory()

Why can’t I connect to a z/OS port?

I’ve found couple of those little problems which took me a day to resolve – but which are obvious when you understand the problem.

The problems

I was trying to connect the Health Center in Eclipse to the Health agent in Liberty on z/OS.

The first problem was the health center agent on z/OS could not connect to the port. This was due to bad TCPIP configuration

The second problem was I could not connect to it from Eclipse. I had configured the port to be on the local rather than external interface.

My setup

In my jvm.options I had

-Xhealthcenter:level=off,readonly=off,jmx=on,port=1972

Problem 1: The health center agent on z/OS could not connect to the port

In the Liberty startup output I received (after about a timeout of about a minute)

SEVERE: Health Center agent failed to start. java.io.IOException: Cannot bind to URL [rmi://S0W1:1972/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host:

Where my system is called S0W1.

It is trying to connect to system S0W1 port 1972, and failing.

TSO PING S0W1 gave

 CS 3.1: Pinging host S0W1.DAL-EBIS.IHOST.COM (172.26.1.2)
 Ping #1 timed out

This was a surprise to me – I was expecting it to be my local z/OS machine…. I do not have an interface with address 172.26.1.2. This explains why it timed out.

In my ADCD.Z31B.TCPPARMS(GBLTDATA) I had 

S0W1:   HOSTNAME   S0W1 
; 
; 
; NOTE - Use either DOMAINORIGIN/DOMAIN or SEARCH to specify your domain 
;         origin value 
; 
; DOMAINORIGIN or DOMAIN statement 
; ================================ 
; DOMAINORIGIN or DOMAIN specifies the domain origin that will be 
; appended to host names passed to the resolver.  If a host name 
; ends with a dot, then the domain origin will not be appended to the 
; host name. 
; 
DOMAINORIGIN  DAL-EBIS.IHOST.COM

Because S0W1 did not end with a dot – TCPIP put the DOMAINORIGIN on the end.

ADCD.Z31B.TCPPARMS(ZPDTIPN1)

had

172.26.1.2 S0W1.DAL-EBIS.IHOST.COM S0W1      
127.0.0.1       LOCALHOST

Which says for S0W1…. use IP address 172.26.1.2.

I changed this to

S0W1:   HOSTNAME   S0W1.
   127.0.0.1       S0W1        
   127.0.0.1       LOCALHOST

With these changes, I restarted TCPIP, and told the resolver to use the updated configuration.

F RESOLVER,REFRESH,SETUP=ADCD.Z31b.TCPPARMS(GBLRESOL)

I then got 

INFO: Health Center agent started on port 1972.

So my first success. However…

Problem 2 : I could not connect Eclipse to the port

… once I had managed to get get the server to connect to the port. When the server issues a TCPIP binds to a port, you need to specify the IP address and port. I had configured the hostname S0W1 as the local interface (127.0.0.1). When I tried to connect from Eclipse, I was trying to connect to port 1972 on interface 10.1.1.2 – which had not been configured!

The Liberty output had

WARNING: RMI TCP Accept-1972: accept loop for erverSocket[addr=0.0.0.0/0.0.0.0, localport=1972] throws java.io.IOException: EDC5122I Input/output error. (errno2=0x12B804B9)

I changed ADCD.Z31B.TCPPARMS(ZPDTIPN1) to have

10.1.1.2 S0W1
127.0.0.1 LOCALHOST

so the name S0W1 is associated with interface 10.1.1.2. I started restart TCPIP and the resolver and it manage to connect. It only took a day to resolve these problems.

Why can’t Liberty see my bootstrap.properties? The answer is in the title!

I put the osgi address and port information into the bootstrap.properties file, but it wasn’t being picked up. A couple of hours later I found out why – the answer is that Liberty could see my bootstrap.properties file.

The command

ls -lT bootstrap.properties

gave

- untagged    T=off -rwx------   1 OMVSKERN SYS1 ... bootstrap.properties

which means the started task running Liberty did not have access to it, only the owner had rwx access. Java did not process it, because it could not see it.

I used the command

chmod 744 bootstrap.properties

and next time I started the Liberty instance, it could find the file.

My file had

osgi.console = 10.1.1.2:5400

The TSO command

tso netstat allconn (port 5400

saying – show all the connections which are, or were recently active, filtered by port 5400

gave

User Id  Conn     State
-------  ----     -----
CSQ9WEB  00000139 Listen
  Local Socket:   ::ffff:10.1.1.2..5400
  Foreign Socket: ::ffff:0.0.0.0..0

Showing it is active.

What’s the best way of connecting to an HTTPS server. Pass ticket or JWT?

This blog post was written as background to some blog posts on Zowe API-ML. It provides back ground knowledge for HTTPS servers running on z/OS, and I think it is useful on its own. Ive written about an MQWEB server – because I have configured this on my system.

The problem

I want to manage my z/OS queue manager from my Linux machine.I have several ways of doing it.

Which architecture?

  • Use an MQ client. Establish a client connect to the CHINIT, and use MQPUT and MQGET administration messages to the queue manager.
    • You can issue a command string, and get back a response string which you then have to parse
    • You can issue an MQINQ API request to programmatically query attributes, and get the values back in fields. No parsing, but you have to write a program to do the work.
  • Use the REST API. This is an HTTP request in a standard format into the MQWEB server.
    • You can issue a command string, and get back a response string which you then have to parse to extract the values.
    • You can issue a JSON object where the request is encoded in a URL, and get the response back in JSON format. It is trivial to extract individual fields from the returned data.

Connecting to the MQWEB server

If you use REST (over HTTPS) there are several ways of doing this

  • You can connect using userid and password. It may be OK to enter your password when you are at the keyboard, but not if you are using scripts and you may be away from your keyboard. If hackers get hold of the password, they have weeks to use it, before the password expires. You want to give your password once per session, not for every request.
  • You can connect using certificates, without specifying userid and password.
    • It needs a bit of set up at the server to map your certificate to a userid.
    • It takes some work to set up how to revoke your access, if you leave the company, or the certificate is compromised.
    • Your private key could be copied and used by hackers. There is discussion about reducing the validity period from over a year to 47 days. For some people this is still too long! You can have your private certificate on a dongle which you have to present when connecting to a back end. This reduces the risk of hackers using your private key.
  • You can connect with a both certificate and userid and password. The certificate is used to establish the TLS session, and the userid and password are used to logon to the application.
  • You can use a pass ticket. You issue a z/OS service which, if authorised, generates a one time password valid for 10 minutes or less. If hackers get hold of the pass ticket, they do not have long to be able to exploit it. The application generating the pass ticket, does not need the password of the userid, because the application has been set up as trusted.
  • You can use a JSON Web Token (JWT). This has some similarities with certificates. In the payload is a userid value and issuer value . I think of issuer as the domain the JWT has come from – it could be TEST or a company name. From the issuer value, and IP address range, you configure the server to specify a realm value. From the userid and realm you can map this to a userid on the server. This JWT can be valid from minutes to many hours (but under a day). The userid and realm mapping to a userid is different to certificate mapping to a userid.

Setting up a pass ticket

The passticket is used within the sysplex. It cannot be used outside of a sysplex. The pass ticket is a password – so needs to be validated against the RACF database.

The application that generates the pass ticket must be authorised to a profile for the application. For example, define the profile for the application TSO on system S0W1, the profile is TSOS0W1.

 RDEFINE PTKTDATA TSOS0W1

and a profile to allow a userid to create a pass ticket for the application

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE) 

PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(COLIN) ACCESS(UPDATE) 
PERMIT IRRPTAUTH.TSOS0W1.* CLASS(PTKTDATA) ID(IBMUSER)ACCESS(UPDATE)

Userids COLIN and IBMUSER can issue the callable service IRRSPK00 to generate a pass ticket for a user for the application TSOS0W1.

The output is a one-use password which has a validity of up to 10 minutes.

As an example, you could configure your MQWEB server to use profile name MQWEB, or CSQ9WEB.

How is it used

A typical scenario is for an application running on a work station to issue a request to an “application” on z/OS, like z/OSMF, to generate a pass ticket for a userid and application name.

The client on the work station then issues a request to the back end server, with the userid and pass ticket. If the back end server matches the application name then the pass ticket will be accepted as a password. The logon will fail if a different application is used, so a pass ticket for TSO cannot be used for MQWEB.
This is more secure than sending a userid and password up with every back end request, but there is additional work in creating the pass ticket, and two network flows.

This solution scales because very little work needs to be done on the work station, and there is some one-off work for the setup to generate the pass tickets.

JSON Web Tokens

See What are JSON Web Tokens and how do they work?

The JWT sent from the client has an expiry time. This can be from seconds to hours. I think it should be less than a day – perhaps a couple of hours at most. If a hacker has a copy of the JWT, they can use it until it expires.

The back end server needs to authenticate the token. It could do this by having a copy of the public certificate in the server’s keyring, or send a request down to the originator to validate it.

If validation is being done with public certificates, because the client’s private key is used to generate the JWT, the server needs a copy of the public certificate in the server’s keyring. This can make it hard to manage if there are many clients.

The Liberty web server has definitions like

<openidConnectClient id="RSCOOKIE" 
  clientId="COLINCOO2" 
  realmName="zOSMF" 
  inboundPropagation="supported" 
  issuerIdentifier="zOSMF" 
  mapIdentityToRegistryUser="false" 
  signatureAlgorithm="RS384" 
  trustAliasName="CONN1.IZUDFLT" 
  trustStoreRef="defaultKeyStore" 
  userIdentifier="sub" 
 > 
<authFilter id="afint"> 
  <remoteAddress id="myAddress" ip="10.1.0.2" matchType="equals" /> 
</authFilter         > 
                                                                                                 
</openidConnectClient>

For this entry to be used various parameters need to match

  • The issuerIdentifier. This string identifies the client. It could be MEGABANK, TEST, or another string of your choice. It has to match what is in the JWT.
  • signatureAlgorithm. This matches the incoming JWT.
  • trustAliasName and trustStoreRef. These identify the certificate used to validate the certificate
  • remoteAddress. This is the address, or address range of the client’s IP addresses.

If you have 1000 client machines, you may need 1000 <openidConnectClient…/> definitions, because of the different certificate and IP addresses.

You may need 1000 entries in the RACMAP mapping of userid + realm to userid to be used on the server.

How is it used

You generate the JWT. There are different ways of doing this.

  • Use a service like z/OSMF
  • Use a service on your work station. I have used Python to do this. The program is 30 lines long and uses the Python jwt package

You get back a long string. You can see what is in the string by pasting the JWT in to jwt.io.
You pass this to the backend as a cookie. The cookie name depends on what the server is expecting. For example

'Authorization': "Bearer " + token

The JWT has limited access

For the server to use the JWT, it needs definitions to recognise it. If you have two back end servers

  • Both servers could be configured to accept the JWT
    • If the server specified a different REALM, then the mapped userid from the JWT could be different for each server because the userid/realm to userid mapping can be different.
  • One server is configured to accept the JWT
    • If only one server has the definitions for the JWT, then trying to use the JWT to logon to another server will fail.

Tracing input and output of the Liberty web server.

The Liberty web server is used by many IBM products on z/OS, for example z/OSMF, MQSeries and z/OSConnect (but not Zowe).

When using Zowe, I struggled finding out what data was input to the server. As usual, when you have found the answer it is easy.

Once it worked, I had

<httpAccessLogging id="accessLogging" 
  logFormat="%a %s ET=%D %r i=%i c=%C " 
  enabled="true" 
/> 
<httpEndpoint id="defaultHttpEndpoint" 
  accessLoggingRef="accessLogging" 
  httpPort="9080" 
  httpsPort="10443" 
  />

Where the httpEndpoint defines the port 10443 , and references httpAccessLogging.

It one point I had two ports defined for https. I separated the output for each port using

filepath="${server.output.dir}/logs/http_10443_access.log"

within the httpAccessLogging definition, to output the data to a specific file to match the port.

What data is output?

You can control what data is output. I used logFormat to output what I was interested in.

logFormat="%a %s ET=%D %r i=%i c=%C "

Where

  • %a is the remote IP address 10.1.0.2
  • %s is the status – if it worked the value is 200.
  • ET=%D. This is the duration of the request in microseconds. It appears as ET=667601
  • %r the first line of the request POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.
  • i=%i the header name from the request. My request did not have one so this comes out as i=-
  • c=%C gives the cookies. You can request a specific cookie. My output had c=jwtToken:eyJraWQiOiJhYVBkRzd5N…. which is the JSON Web Token. To see the contents, I took this token, and pasted it into http:jwt.io.

You can ask for the datetime, but this comes out as a long string with year,month, day hh:mm:ss.uuuuuu. I found the year month and day were not needed, but I could not find how to display just the time.

The output from the above format was

10.1.0.2 200 ET=667601 POST /ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc HTTP/1.1 i=- c=jwtToken:eyJraWQiOiJhYVBkRzd5NmZETk5UT....

Using keyrings and certificates on z/OS with Liberty

This is a work in progress… but someone asked me for this information so I’ll publish as it is.

I had a lot of playing around with MQWEB on Linux, and I thought that z/OS would be just a small step more than Linux.  After many false steps, it is very similar to Linux – but because z/OS is more secure than Linux you have to do more work to get the right permissions.

Once you have set up the keyrings and got your web browser to talk to the Liberty at the back end, you can then

  • logon with userid and password
  • logon with digital certificate, so there is now set up to map your Linux  digital certificate to a z/OS userid

Once you have got the z/OS userids set up,  you need to set up the RACF groups which determine if you are allowed to use the server, and if you are a read only user, or a full function.

z/OS certificates background.

  • You use the RACF RACDCERT command to manage certificates
  • Certificates either stored in an (encrypted) RACF data base.
  • You provide similar information as with midrange machines to define certificates, such as Organisation, Common Name, Country etc.
  • On z/OS you cannot specify Extended Key Usage (EKU) attributes.
  • A certificate has two parts,
    • a private part, this allows you to encrypt data for that certificate.  It must be kept securely.
    • a public part  – anyone can have a copy of this – allows them to decrypt a message created using the private key.
  • Certificate ownership.  The RACF documentation says
    • User certificate A certificate that is associated with a RACF user ID and is used to authenticate the user’s identity. The RACF user ID can represent a traditional user or be assigned to a server or started procedure.  These are identified by ID(…).
    • Certificate-authority certificate. A certificate that is associated with a certificate authority and is used to verify signatures in other certificates.   These are identified by CERTAUTH.
    • Site certificate. A certificate that is associated with an off-platform server or other network entity, such as a peer VPNserver. This category of certificate can also be used to share a single certificate and its private key among multiple RACF user IDs. When used for sharing, a certificate might be referred to as a placeholder certificate.  These are identified by SITE.
  • A certificate has to have the TRUST attribute to be visible in a keyring.
  • Applications access certificate via a user’s keyring.
    • A userid can have many keyrings
    • A keyring can have many certificates.  They can belong to User, Certificate Authority, or Site.
    • You can control access at an individual keyring level.
  • Your web browser will likely have two keyrings (similar to midrange)
    • A Server key ring containing the private certificate to identify the server, and has the private key.   This keyring might only be used by a group of MQWEB servers and no other applications. A web server in CICS would have a its own keyring.
    • A Trust store.  This has the public certificates and allows people to communicate with the web server.  It typically has Certificate Authority certificates.   A CA certificate can validate all users who’s certificate was  signed by the CA. So one CA in the keyring could validate 1000 users.   If the end users are using self signed certificates, then the z/OS keyring has to have a copy of every self signed certificate.  If you add a new certificate then you may have to restart the server.
    • You might share this trust store with all web servers, either native or within CICS.

Steps for setting up the certificates and keyrings to connect the browser to the z/OS web server

  • Server keyring
    • Create a z/OS certificate authority
    • Create a server keyring
    • Create a certificate for the server
    • Add the certificate to the keying
    • Add the CA to the keyring
    • Give the mqweb userid access to the keyring.
    • Export the CA certificate, download it, distribute it to all users.  Import into their web browsers.
  • Trust keyring
    • Create a (shared) keyring
    • Get the CA from your users, upload to z/OS import them into the keyring
    • Give the server userids access to the keyring

Once you have been sucessfull with setting up the keyrings you will either be logged on on, or get a logon panel.

You now have to configure the environment to allow people to logon with the correct authorities.

Detailed steps.

I found it easiest to create JCL using batch TSO  with the various commands rather than use panels, or typing commands.  It means you can repeat the steps until they work.

//IBMRABB JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
  Put your commands here
  /* this (with a space in front) is ignored 
/*

Configuration

  • The MQWEB server runs with userid START1
  • It will have a server key keyring called KEY
  • It will have a trust store keyring called TRUST
  • The IP address of the z/OS image is 10.1.1.2.   This is needed  when defining the server’s certificate because the clients compare this with web server’s location.  It can be IP(‘nn.nn.nn.nn’) or DOMAIN(‘ABC.COM’).

Create the CA

 /* Delete it in case it existed
RACDCERT CERTAUTH   DELETE(LABEL('TEMP-CA'))
 /* the CA needs size >= 2048 otherwise Chrome browser may ignore it
 /* The certificate shows up in browsers under org-TEST 
RACDCERT GENCERT  - 
  CERTAUTH - 
  SUBJECTSDN(CN('TEMPCertification Authority')- 
             O('TEMP') - 
             OU('TEST')) - 
  NOTAFTER(   DATE(2021-07-01  ) -   
  KEYUSAGE(HANDSHAKE,CERTSIGN,DOCSIGN,DATAENCRYPT          ) - 
  RSA SIZE(2048) - 
  WITHLABEL('TEMP-CA') 

RACDCERT CERTAUTH EXPORT(LABEL('TEMP-CA'))- 
       DSN('IBMUSER.CERT.TEMPCA.PEM') - 
       FORMAT(CERTB64) - 
       PASSWORD('password')

You can now download the dataset IBMUSER.CERT.TEMPCA.PEM and import it into the browser’s keystore.

Create the server keyring

RACDCERT DELRING(KEY) ID(START1)
RACDCERT ADDRING(KEY) ID(START1)

Create the certificate to identify the server

This needs the ALTNAME so the browsers  accept the certificate

RACDCERT ID(START1) DELETE(LABEL('TEMP')) 
RACDCERT ID(START1) GENCERT - 
  SUBJECTSDN(CN('TEMP') - 
             O('TEMP') - 
             OU('TEST')) - 
   ALTNAME(IP(10.1.1.2)) - 
   SIGNWITH (CERTAUTH LABEL('TEMP-CA')) - 
   RSA SIZE(2048) 
   NOTAFTER(   DATE(2021-12-29))- 
   WITHLABEL('TEMP')
 /* KEYUSAGE is not needed

RACDCERT ID(START1) ALTER(LABEL(‘TEMP’))TRUST

      
RACDCERT ID(START1) CONNECT(RING(KEY) LABEL('TEMP')) 
RACDCERT ID(START1) CONNECT(RING(KEY) - 
       LABEL('TEMP-CA') CERTAUTH) 
RACDCERT LIST(LABEL('TEMP-CA'             ))  CERTAUTH 
RACDCERT LIST(LABEL('TEMP'                ))  ID(START1) 
                                                                  
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Give the servers userid access to the server’s keyring

See here.

RDEFINE RDATALIB START1.KEY.LST UACC(NONE)
 /* READ access enables retrieving one's own private key
 /* UPDATE access enables retrieving other's.
PERMIT START1.KEY.LST CLASS(RDATALIB) ID(START1 ) ACCESS(UPDATE)
PERMIT START1.KEY.LST CLASS(RDATALIB) ID(IBMUSER) ACCESS(UPDATE)

SETROPTS RACLIST(RDATALIB) REFRESH
RLIST FACILITY START1.KEY.LST

 

Upload the CA from the midrange machine and store in the key store.

The certificate ( .pem file) looks like

—–BEGIN CERTIFICATE—–
…
—–END CERTIFICATE—–

So should be uploaded as a source file.

RACDCERT CERTAUTH DELETE(LABEL('Linux-CA2')) 
RACDCERT SITE ADD('IBMUSER.CACERT.PEM') - 
    WITHLABEL('Linux-CA2') TRUST 
RACDCERT DELRING(TRUST) ID(START1) 
RACDCERT ADDRING(TRUST) ID(START1) 


RACDCERT ID(START1) CONNECT(RING(TRUST ) - 
       LABEL('Linux-CA2') SITE 

RACDCERT LISTRING(TRUST ) ID(START1) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Of course you run this just once, and use the connect statement to add as many CAs as you have.

 

 

Passing parameters to Java programs

There is a quote by Alexander Pope

A little learning is a dangerous thing.
Drink deep, or taste not the Pierian Spring;
There shallow draughts intoxicate the brain,
and drinking largely sobers us again.”

I was having problems passing parameters to Java. I had a little knowledge about Java web servers on z/OS, so I was confident I knew how to configure a web browser; unfortunately I did not have enough knowledge.

I was trying to get keyrings working with a web server on z/OS, but what I specified was not being picked up. In this post I’ll focus on keyrings, but it applies to other parameters are well.

Setting the default values

You can give Java a default keyring name

-Djavax.net.ssl.keyStore=safkeyring://START1/MQRING

Which says, if the user does not specify a keystore then use this value. If the user does not create a keystore, then a default keystore is created using these parameters.

Specify the value yourself

You can create your own keystore, passing keystore name and keystore type. You could use you own option, such as -DCOLINKEYRINGNAME=safkeyring://START1/MQRING and use that when configuring the keystore to Java. In this case the -Djavax.net.ssl.keyStore=safkeyring://START1/MQRING is not used. This is the same for other parameters.

Web server configuation

The Apache web server, and IBM Liberty web servers, are configured using server.xml files, and the web server creates the keystores etc using the information in the file.

For example a web server may have content like 

<Connector port="${port.http}" protocol="${http.protocol}" 
     SSLEnabled="true" 
     maxThreads="550" scheme="https" secure="true" 
     clientAuth="false" sslProtocol="TLS" 
     keystoreFile="${keystoreFile}" 
     keystorePass="${keystorePass}" 
     keystoreType="${keystoreType}" 
     sslEnabledProtocols="${ssl.enabled.protocols}" />

where the ${…} is an environment variable. ${keystoreFile} would use the environment variable keystoreFile or option -DkeystoreFile=”safkeyring//START1/MYRING”

You can have multiple <Connector..> each with different parameters, so using -Djavax.net.ssl.keyStore etc would not work.

You need to know which options to use, because setting the Java defaults using -Djavax.net.ssl.keyStore… may be overridden.

You also need to know which server.xml file to use! I was getting frustrated when the changes I made to the server.xml was not the server.xml being used!

I’m sorry I haven’t a clue…

As well as being a very popular British comedy, it is how I sometimes feel about what is happening inside the Liberty Web servers, and products like z/OSMF, z/OS Connect and MQWEB. It feels like a spacecraft in cartoons – there are usually only two controls – start and stop.

One reason for this is that the developers often do not have to use the product in production, and have not sat there, head in hand saying “what is going on ?”.

In this post I’ll cover

What data values to expose

As a concept, if you give someone a lever to pull – you need to give them a way of showing the effect of pulling the level.

If you give someone a tuning parameter, they need to know the impact of using the tuning parameter. For example

  • you implement a pool of blocks of storage.
  • you can configure the number of maximum number of blocks
  • if a thread needs some storage, and there is a free block in the pool, then assign the block to the thread. When the thread has finished with it, the thread goes back into the pool.
  • if all the blocks in the pool are in-use, allocate a block. When the thread has finished with the block – free it.
  • if you specify a very large number of blocks it could cause a storage shortage

The big questions with this example is “how big do you make the pool”?

To be able to specify the correct pool size you need to know information like

  • What was the maximum number of blocks used – in total
  • How many times were additional blocks allocated (and freed)
  • What was the total number of blocks requested.

You might decide that the pool is big enough if less than1% of requests had to allocate a block.

If you find that the maximum value used was 1% of the size of the pool, you can make the pool much smaller.

If you find that 99% of the requests were allocated/freed, this indicates the pool is much to small and you need to increase the size.

For other areas you could display

  • The number of authentication requests that were userid+ password, or were from a certificate.
  • The number of authentication requests which failed.
  • The list of userid names in the userid cache.
  • How many times each application was invoked.
  • The number of times a thread had to wait for a resource.
  • The elapsed time waiting for a resource, and what the resource was.

What attributes to expose

You look at the data to ask

  • Do I have a problem now?
  • Will I have a problem in the future? You need to collect information over time and look at trends.
  • When we had a problem yesterday, did this component contribute to it? You need to have historical data.

It is not obvious what data attributes you should display.

  • The “value now” is is easy to understand.
  • The “average value” is harder. Is this from the start of the application (6 months ago), or a weighted average (99 * previous average + current value)/100. With this weighted average, a change since the previous value indicates the trend.
  • The maximum value is hard – from when? There may have been a peak at startup, and small peaks since then will not show up. Having a “reset command” can be useful, or have it reset on a timer – such as display and reset every 10 minutes.
  • If you “reset” the values and display the value before any activity, what do you display? “0”s for all of the values, or the values when the reset command was issued.

Resetting values can make it easier to understand the data. Comparing two 8 digit numbers is much harder than comparing two 2 digit numbers.

How to expose data

Java has a Java Management eXtension (JMX) for reporting management information. It looks very well designed, is easy to use, and very compact! There is an extensive document from Oracle here.

I found Basic Introduction to JMX by Baeldung , was an excellent article with code samples on GitHub. I got these working in Eclipse within an hour!

The principal behind JMX is …

For each field you want to expose you have a get… method.

You define an interface with name class| |”MBean” which defines all of the methods for displaying the data.

public interface myClassMBean {
public String getOwner();
public int getMaxSize();
}

You define the class and the methods to expose the data.

public class myClass implements myClassMBean{

…

// and the methods to expose the data

public String getOwner() {
return fileOwner;
}

public int getMaxSize() {
return fileSize;
}

}

And you tell JMX to implement it

myClass myClassInstance = new myClass(); // create the instance of myClass

MBeanServer server = ManagementFactory.getPlatformMBeanServer();
ObjectName objectName =….
server.registerMBean(myClassInstance, objectName);

Where myClassInstance is a class instance. The JMX code extracts the name of the class from the object, and can the identify all the methods defined in the class||”MBean” interface. Tools like jconsole can then query these methods, and invoke them.

ObjectName is an object like

ObjectName objectName = new ObjectName(“ColinJava:type=files,name=onefile”);

Where “ColinJava” is a high level element, “type” is a category, and “name” is the description of the instance .

That’s it.

When you use jconsole ( or other tools) to display it you get

You could have

MBeanServer server = ManagementFactory.getPlatformMBeanServer();

ObjectName bigPoolName = new ObjectName(“ColinJava:type=threadpool,name=BigPool”);
server.registerMBean(bigpoolInstance, bigPoolName);

ObjectName medPoolName = new ObjectName(“ColinJava:type=threadpool,name=MedPool”);
server.registerMBean(medpoolInstance, medPoolname);

ObjectName smPoolName = new ObjectName(“ColinJava:type=threadpool,name=SmallPool”);
server.registerMBean(smallpoolInstance,smPoolName);

This would display the stats data for three pools

  • ColinJava
    • threadpool
      • Bigpool..
      • MedPool….
      • SmallPool…

And so build up a tree like

  • ColinJava
    • threadpool
      • Bigpool..
      • MedPool….
      • SmallPool…
    • Userids
      • Userid+password
      • Certificate
    • Applications
      • Application 1
      • Application 2
    • Errors
      • Applications
      • Authentication

You can also have set…() methods to set values, but you need to be more careful; checking authorities, and possibly synchronising updates with other concurrent activity.

You can also have methods like resetStats() which show up within jconsole as Operations.

How do I build up the list of what is needed?

It is easy to expose data values which have little value. I remember MQ had a field in the statistics “Number of times the hash table changed”. I never found a use for this. Other times I thought “If only we had a count of ……”

You can collect information from problems reported to you. “It was hard to diagnose because… if we had the count of … the end user could have fixed it without calling us”.

Your performance team is another good source of candidates fields. Part of the performance team’s job is to identify statistics to make it easier to tune the system, and reduce the resources used. It is not just about identifying hot spots.

Before you implement the collection of data, you could present to your team on how the data will be used, and produce some typical graphs. You should get some good feedback, even if it is “I dont understand it”.

What can I use to display the data

There are several ways of displaying the data.

  • jconsole – which comes as part of Java can display the data in a window
  • python – you can issue a query can capture the data. I have this set up to capture the data every 10 seconds
  • other tools using the standard interfaces.

Have a good REST and save a fortune in CPU with Python

Following on from Have a good REST and save a fortune in CPU. The post gives some guidance on reducing the costs of using Liberty based servers from a Python program.

Certificate set up

I used certificate authentication from Linux to z/OS. I used

  • A certificate defined on Linux using Openssl.
  • I sent the Linux CA certificate to z/OS and imported it to the TRUST keyring.
  • I created a certificate on z/OS and installed it into the KEY keyring.
  • I exported the z/OS CA, sent it to Linux, and created a file called tempca.pem.

Python set up

Define the names of the user certificate private key, and certificate

cf=”colinpaicesECp256r1.pem”
kf=”colinpaicesECp256r1.key.pem”
cpcert=(cf,kf)

Define the name of the certificate for validating the server’s certificate

v=’tempca.pem’

Set up a cookie jar to hold the cookies sent down from the server

jar = requests.cookies.RequestsCookieJar()

Define the URL and request

geturl =”https://10.1.1.2:9443/ibmmq/rest/v1/admin/qmgr/ “

Define the headers

import base64
useridPassword = base64.b64encode(b’colin:passworm’)
my_header = {
‘Content-Type’: ‘application/json’,
‘Authorization’: useridPassword,
‘ibm-mq-rest-csrf-token’ : ‘ ‘
}

An example flow of two requests, using two connections

For example using python

s = requests
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert)

creates two session, each has a TLS handshake, issue a request, get a response and end.

An example of two requests using one session

For example using python

s = requests.Session()
response1 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert1)
response2 = s.get(geturl,headers=my_header,verify=v,cookies=jar,cert=cpcert2)

The initial request has one expensive TLS handshake, the second request reuses the session.

Reusing this session means there was only one expensive Client Hello,Server Hello exchange for the whole conversation.

Even though the second request specified a different set of certificates, the certificates from when the session was established, using cpcert1 were used. (No surprise here as the certificates are only used when the session is established).

For the authentication, in both cases the first requests received a cookie with the LtpaToken2 cookie in it.

When this was passed up on successive requests, the userid information from the first request was used.

What is the difference?

I ran a workload of a single thread doing 200 requests. The ratios are important, not the absolute values.

Shared sessionOne session per requests
TCP flows to server1 11
CPU cost1 5
Elapsed time16

Have a good REST and save a fortune in CPU

The REST protocol is a common programming model with the internet. It is basically a one shot model, which scales and has high availability, but can have a very high CPU cost. There are things you can do to reduce the CPU cost. Also, the MQWeb server, has implemented some changes to reduce the cost. See here for the MQ documentation.

The post gives some guidance on reducing the costs, for Liberty based servers.

The traditional model and the REST model

The traditional application model may have a client and a flow to the server

  • Connect to the server and authenticate
  • Debit my account by £500 within syncpoint
  • Credit your account by £500 within syncpoint
  • Commit the transaction
  • Do the next transaction etc
  • At the end of the day, disconnect from the server.

The REST model would be

  • Connect to the server and authenticate and do (Debit my account by £500, credit your account by £500), disconnect

This model has the advantage that it scales. When you initiate a transaction it can go to any one of the available back-end servers. This spreads the load and improves availability.

With the traditional model, the clients connects any available server at the start of day stays connected all day. If a new server becomes available during the day, it may get no workload.

The downside of the REST model is the cost. Establishing a connection and authenticating can be very expensive. I could not find much useful documentation on how to reduce these costs.

There are two parts of getting a REST connection.

  • Establishing the connection
  • Authentication

Establishing the connection

You can have each REST request use a new session for every REST request each of which which involves a full TLS handshake. Two requests could go to different servers, or go to the same server.

You can issue multiple REST request over the same session, to the same backend server.

On my little z/OS, using z/OSMF it takes

  • about 1 second to create a new connection and issue a request and terminate
  • about 0.1 seconds to use the shared session, per REST request.

Establishing the TLS session is expensive, as there is a lot of computation to generate the keys.

For MQWEB, the results are very similar.

Authentication

Once the session has been established each REST request requires authentication.

If you are using userid and password, the values are checked with z/OS.

If you are using client certificate authentication the Subject DN is looked up in the security manager, and if there is a DN to userid mapping, the userid is returned.

Once you have a valid userid, the userid’s access can be obtained from the security manager.

All of these values can be cached in the Liberty web server. So the first time a certificate or userid is used, it will take a longer than successive times.

Information about authentication is then encrypted and passed back in the REST response as the LtpaToken2 cookie.

If a REST request passes the cookie back to the server, then the information in the cookie is used by the server, and fewer checks need to be done.

This cookie can expire, and when it does expire the userid and password, or the certificate DN is checked as before, and the cookie will be updated.

If you do not send the LtpaToken2 cookie, this will cause the passed authentication information to be revalidated. If you want to change userid, do not send the the cookie.

Is any of this documented?

There is not a lot of documentation. There is information Configuring the authentication cache in Liberty.

There is a parameter javax.net.ssl.sessionCacheSize. If this is not set the default is 20480.