Would you lock your front door and leave the key under the mat? So why do you do it with digital keys?

Where I live it is Island Mentality. Someone said to me that they do not lock their front door. Sometimes, when they come home, they find some eggs or tray-bakes on the kitchen table. They went on a celebration cruise, but could not find the key to the front door, and so left the house unlocked the two weeks they were away.

Digital certificates and keys are used for identification authentication. Often these are stored in a key store, just a file in Windows or Unix. You typically need a password to be able to read the file. If you got hold of a keystore, you could try “password” with an “o”, “passw0rd” with zero etc. There is no limit to the number of attempts you can have. Don’t worry, the password is stored in a stash file , which is just another file. If you have the key store and the stash file you can open the keystore using standard commands. Having both the keystore and the stash file is like finding the front door unlocked.

If someone is an administrator on the machine, they can access any file and so can get the keystore and the stash file. IBM says you need superuser access to install MQ – so the MQ administrator can access these files. I heard that one enterprise was doing backups from the user’s machines to a remote site. The files were encrypted at the remote site, but not the network link to the remote site – whoops! The files could have been stolen en route.

Use external security devices.

You can get round this problem by using an external Hardware Security Module. Instead of storing the keys in a file, they are stored on an external device. You can get USB like devices. Some HSM can store keys, other HSMs can encrypt data. For example my bank gives its user’s a small machine. You put in your debit card, enter your pin. It encrypts the data and generates a one time key which you enter into the bank’s web site.

To steal the keystore you now need access to the physical machine to be able to unplug the USB.

Built in devices that cannot be removed.

On some machines, such as z hardware, they have a tamper resistant “cryptographic chip” built in. If you remove it from the machine, it is useless. When you configure it you need three keys, so you have three people each with their own key. When you install the backup machine, the three people have to go on site, and re enter their keys. They have mechanisms like three wrong passwords and it self destructs (perhaps in a cloud of smoke, as it does in the movies).

“Cloud”

One of the selling points of cloud is flexibility. You can deploy an image anywhere; you can wheel in new machines, and wheel out old machines; and you can have different “tenants” on the same hardware. This makes it difficult to use an HSM device to store your keys, as each machine needs the same keys, and the HSM could have all the keys from all the tenants. So you have the problem, of having your key store as a file with its stash file, and even more people have access to these files.

Would you lock your front door and leave the key under the mat? So why do you do it with digital keys

It is all down to the management of risk. Digital certificates do not give absolute protection. Strong encryption just means it takes longer to crack!

Certificate logon to MQWEB on z/OS, the hard way.

I described here different ways of logging on to the MQ Web Server on z/OS. This post describes how to use a digital certificate to logon. There is a lot of description, but the RACF statements needed are listed at the bottom.

I had set up my keystore and could logon to MQWEB on z/OS using certificates. I just wanted to not be prompted for a password.

Once it is set up it works well. I thought I would deliberately try to get as many things wrong, so I could document the symptoms and the cure. Despite this, I often had my head in the hands, asking “Why! – it worked yesterday”.

Can I use CHLAUTH ? No – because that is for the CHINIT, and you do not need to have the CHINIT running to run the web server.

Within one MQ Web Server, you can use both “certificate only” logon as well as using “certificate, userid and password” logon.

When using the SAF interface you specify parameters in the mqwebuser.xml file, such as keyrings, and what level of certificate checking you want.

Enable SAF messages.

If you use <safCredentials suppressAuthFailureMessage=”false” …> in the mqwebuser.xml then if a SAF request fails, there will be a message on the z/OS console. You would normally have this value set to “true” because when the browser (or REST client) reauthenticates (it could be every 10 seconds) you will get a message saying a userid does not have access to an APPL, or EJBROLE profile. If you change this (or make any change the mqwebuser.xnml file), issue the command

f CSQ9WEB,refresh,config

To pick up the changes.

Configure the server name

In the mqwebuser.xml file is <safCredentials profilePrefix=”MQWEB“…> there MQWEB identifies the server, and is used in the security profiles (see below).

SSL parameters

In the mqwebuser.xml file you specify

  • <ssl …
  • clientAuthenticationSupported=”true”|”false. The doc says The server requests that a client sends a certificate. The client’s certificate is optional
  • clientAuthentication=”true”|”false” if true, then client must send a certificate.
  • ssslProtocol=”TLSV1.2″
  • keyStoreRef=”…”
  • trustStoreRef=”…”
  • id=”…”
  • <sslDefault … sslRef=”…” this points to a particular <ssl id=…> definition. It allows you to have more than one <ssl definition, and pick one.

I think it would have been clearer if the parameters were clientAuthentication=”yes”|”no”|”optional”. See my interpretation of what these mean here.

Client authentication

The client certificate maps to a userid on z/OS, and this userid is used for access control.

The TLS handshake: You have a certificate on your client machine. There is a handshake with the server, where the certificate from the server is sent to the client, and the client verifies it. With TLS client authentication the client sends a certificate to the server. The server validates it.

If any of the following are false, it drops through to Connecting with a client certificate, and authenticate with userid and password below.

Find the z/OS userid for the certificate

The certificate is looked up in a RACDCERT MAP to get a userid for the certificate (see below for example statements). It could be a one to one mapping, or depending on say OU=TEST or C=GB, it can check on part of the DN. If this fails you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0194)
SUBJECT(CN=ADCDC.O=cpwebuser.C=GB) ISSUER(CN=SSCARSA1024.OU=CA.O=SSS.
C=GB).

Check the userid against the APPL class.

The userid is checked against the MQWEB profiles in the APPL class. (Where MQWEB is the name you configured in the web server configuration files). If this fails you get

ICH408I USER(ADCDE ) GROUP(TEST ) NAME(ADCDE ) MQWEB CL(APPL )
WARNING: INSUFFICIENT AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

Pick the EJBROLE for the userid

There are several profiles in the EJBROLES class. If the userid has read access to the class, it userid gets the attribute. For example for the profile MQWEB.com.ibm.mq.console.MQWebAdmin, if the userid has at least READ access to the profile, it gets MQWEBADMIN privileges.
If these fail you get messages in the MQWEB message logs(s).

To suppress the RACF messages use option suppressAuthFailureMessage=”false” described above.

The userid needs access to at least one profile to be able to use the MQ Web server.

Use the right URL

The URL is like https://10.1.1.2:9443/ibmmq/console/

No password is needed to logon. If you get this far, displaying the userid information (click on the ⓘ icon) gives you Principal:ADCDE – Read-Only Administrator (Client Certificate Authentication) where ADCDE is the userid from the RACDDEF MAP mapping.

Connecting with a client certificate, and authenticate with userid and password.

The handshake as described above is done as above. If clientAuthentication=”true” is specified, and the handshake fails, then the client gets This site can’t be reached or similar message.

If the site can be reached, and a URL like https://10.1.1.2:9443/ibmmq/console/login.html is used, this displays a userid and password panel.

The password is verified, and if successful the specified userid is looked up in the APPL and EJBROLES profiles as described above.

If you get this far, and have logged on, displaying the userid information (click on the ⓘ icon) gives you Principal:colin – Read-Only Administrator (Client Certificate Authentication) where colin is the userid I entered.

The short solution to implement certificate authentication

If you already have TLS certificates for connecting to the MQ Web Server, you may be able to use a URL like https://10.1.1.2:9443/ibmmq/console/ to do the logon. If you use an invalid URL, it will substitute it with https://10.1.1.2:9443/ibmmq/console/login.html .

My set up.

I set up a certificate on Linux with a DN of C=GB,O=cpwebuser,CN=ADCDC and signed by C=GB,O=SSS,OU=CA,CN=SSCARSA1024. The Linux CA had been added to the trust store on z/OS.

Associate a certificate with a z/OS userid

I set up a RACF MAP of certificate to userid. It is sensible to run these using JCL, and to save the JCL for each definition.

 /*RACDCERT DELMAP( LABEL('ADCDZXX'  )) ID(ADCDE  ) 
 /*RACDCERT DELMAP( LABEL('CA'  )) ID(ADCDZ  )   
RACDCERT MAP ID(ADCDE  )  - 
    SDNFILTER('CN=ADCDC.O=cpwebuser.C=GB') - 
    WITHLABEL('ADCDZXX') 
                                                 
 RACDCERT MAP ID(ADCDZ  )  - 
    IDNFILTER('CN=SSCARSA1024.OU=CA.O=SSS.C=GB') 
    WITHLABEL('CA       ') 
                                                 
 RACDCERT LISTMAP ID(ADCDE) 
 RACDCERT LISTMAP ID(ADCDZ) 
 SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH 

This mapped the certificate CN=ADCDC.OU=cpwebuser.C=GB to userid ADCDE. Note the “.” between the parts, and the order has changed from least significant to most significant. For other certificates coming in with the Issuer CA of CN=SSCARSA1024.OU=CA.O=SSS.C=GB they will get a userid of ADCDZ.

You do not need to refresh anything as this change becomes visible when the SETROPTS RACLIST REFESH is issued.

First logon attempt

I stopped and restarted my Chrome browser, and used the URL https://10.1.1.2:9443/ibmmq/console. I was prompted for a list of valid certificates. I chose “Subject:ADCD: Issuer:SSCARSA1024 Serial:0194”.

Sometimes it gave me a blank screen, other times it gave me the logon screen with username and Password fields. It had a URL of https://10.1.1.2:9443/ibmmq/console/login.html.

On the z/OS console I got

ICH408I USER(ADCDE ) GROUP(TEST ) NAME(ADCDE ) MQWEB CL(APPL )
WARNING: INSUFFICIENT AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

I could see the the userid(ADCDE) from the RACDCERT MAP was being used (as expected). To give the userid access to the MQWEB resource, I issued the commands

 /* RDEFINE APPL MQWEB UACC(NONE)
PERMIT MQWEB CLASS(APPL) ACCESS(READ) ID(ADCDE)
SETROPTS RACLIST(APPL) REFRESH

And tried again. The web screen remained blank (even with the correct URL). There were no messages on the MQWEB job log. Within the MQWEB stdout (and /u/mqweb/servers/mqweb/logs/messages.log) were messages like

[AUDIT ] CWWKS9104A: Authorization failed for user ADCDE while invoking com.ibm.mq.console on
/ui/userregistry/userinfo. The user is not granted access to any of the required roles: [MQWebAdmin, MQWebAdminRO, MQWebUser].

Give the userid access to the EJBroles

In my mqwebuser.xml I have <safCredentials profilePrefix=”MQWEB”. The MQWEB is the prefix of the EJBROLE resource name. I had set up a group MQPA Web Readonly Admin (MQPAWRA) to make the administration easier. Give the group permission, and connect the userid to the group.

 /* RDEFINE EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO  UACC(NONE) 
PERMIT MQWEB.com.ibm.mq.console.MQWebAdminRO CLASS(EJBROLE) - 
  ACCESS(READ) ID(MQPAWRA) 
CONNECT ADCDE group(MQPAWRA)
SETROPTS RACLIST(EJBROLE) REFRESH

Once the security change has been made, it is visible immediately to the MQWEB server. I clicked the browser’s refresh button and successfully got the IBM MQ welcome page (without having to enter a userid or password). When I clicked on the ⓘ icon it said

Principal:ADCDE – Read-Only Administrator (Client Certificate Authentication)

Logoff doesn’t

If you click the logoff icon, you get logged off – but immediately get logged on again – that’s what certificate authorisation does for you. You need to go to a different web site. If you come back to the ibmmq/console web site, it will use the same certificate as you used before.

Ways of logging on to MQWEB on z/OS.

There are different ways of connecting to the MQ Web Server on z/OS (this is based on the z/OS Liberty Web server). Some ways use the SAF interface. This is an interface to the z/OS security manager. IBM provides RACF, there are other security managers such as TOP SECRET, and ACF2. Userid information is stored in the security manager database.

The ways of connecting to the MQ Web server on z/OS.

  • No security. Use no_security.xml to set up the MQ Web Server.
  • Hard coded userids and passwords in a file. Using the basic_registry.xml. This defines userid information like <user name=”mqadmin” password=”mqadmin”> . This is suitable only for a sandbox. The password can be obscured or left in plain text.
  • Logon by z/OS userid and password. Use zos_saf_registry.xml. Logon is by userid and password and checked by a SAF call to the z/OS security manager. The userid is checked for access to a resource like MQWEB.com.ibm.mq.console.MQWebAdmin in class(EJBROLE) and MQWEB in class(APPL).
  • Connect with a client certificate, and authenticate using userid and password. This uses zos_saf_registry.xml plus additional configuration. The userid, password and access to the EJBROLE and APPL resources is checked by the SAF interface. The certificate id is not used to check access, it is just used to do the TLS handshake.
  • Certificate authentication, a password is not required. Connecting use a client certificate. This uses zos_saf_registry.xml plus additional configuration. Using the SAF interface, the certificate maps to a z/OS userid; this ID is used for checking access to the EJBROLE and APPL resource.

The configuration for using TLS is not clear.

I found the documentation for the TLS configuration to be unclear. Two parameters are <ssl clientAuthentication clientAuthenticationSupported…/> The documentation says

  • If you specify clientAuthentication="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake does not succeed.
  • If you specify clientAuthenticationSupported="true", the server requests that a client sends a certificate. However, if the client does not have a certificate, or the certificate is not trusted by the server, the handshake might still succeed.
  • If you do not specify either clientAuthentication or clientAuthenticationSupported, or you specify clientAuthentication="false" or clientAuthenticationSupported="false", the server does not request that a client send a certificate during the handshake.

I experimented with the different options and the results are below.

  1. I used a web browser with several possible certificates that could be used for authentication. I was given a pop up which listed them. Chrome remembers the choice. With Firefox, you can click an option “set as default“. If this is unticked you get prompted every time.
  2. I used a browser with no certificates for authentication.

When a session was not allowed, I got (from Firefox) Secure Connection Failed. An error occurred during a connection to 10.1.1.2:9443. PR_END_OF_FILE_ERROR

Client AuthenticationClient Authentication SupportedBrowser with certificatesBrowser without certificates
trueignoredPick certificate, userid and password NOT requiredPR_END_OF_FILE_ERROR
falsetruePick certificate, userid and password NOT requiredA variety of results. One of
  1. PR_END_OF_FILE_ERROR,
  2. Blank screen
  3. Userid and password required
falsefalseUserid and password requiredUserid and password required

When using certificates, you can chose to specify userid and password instead of client authentication, by using the appropriate URL with https://10.1.1.2:9443/ibmmq/console/login.html, instead of https://10.1.1.2:9443/ibmmq/console .

Note well.

The server caches credential information. If you change the configuration and refresh the server, the change may not be picked up immediately.

Once you have logged on successfully, a cookie is stored in your browser. This may be used to authenticate, until the token has expired. To be sure of clearing this token I restarted my browser.

Are you going crazy with Chrome giving fatal: certificate_unknown, bad_certificate NET::ERR_CERT_INVALID? Me too!

I came back to using Chrome with my MQ Web browser. It was working last week, but yesterday and today it stopped working. In debugging it, I’ve learned even more ways of checking TLS handshakes to see why they fail!

Using Wireshark packet trace on Linux, and TLS trace in the web server, I could see the Client Hello, Server hello, worked; but the response was Chrome giving NET::ERR_CERT_INVALID, and the traces showing Alert Level: Fatal, Description: Certificate Unknown.

  • I had been through the checks to make sure the CA was in the client key store.
  • I double checked, and tripled checked to make sure it was the right CA.
  • I exported the z/OS CA certificate, downloaded it and imported it.
  • I displayed the z/OS version, and the Chrome keystore’s version and they matched ( ok – the not-after and not-before times were different due to different time zones).
  • I shut every thing down and restarted it the next day.
  • I felt like screaming “AHH -it worked last week it works on FireFox – it should all work on Chrome”

After a day workingb in the garden, I had time to consider a different approach.

How I traced the problem down.

From your browser you can display the “problem” certificate by clicking on the icon in front of the URL. You get into “certificate viewer”. There is a “General” display which shows you useful information about the certificate. There is also the “Details”. At the bottom of the Details is a button labelled “Export”. Click it and export the certificate to a file such as SERVER.PEM.

You can now use openssl on this. For example

openssl x509 -in ~/Downloads/SERVER.cert.bad -text -noout|less

This shows you all the details of the certificate, so you can check them again!

You can also export the CA you think is being used, from the browser keystore, eg CA.pem

You can use the openssl verify command to do its validation of the server certificate and the CA certificate.

openssl verify -CAfile ca.pem -show_chain ~/Downloads/SERVER.cert

worked, it gave

OK
Chain:
depth=0: O = ZZZZ, OU = SSS, CN = SERVER (untrusted)
depth=1: O = TEMP, OU = TEST, CN = TEMP4Certification Authority

Which shows the server certificate, and the CA certificate were OK. However the same command with -x509_strict gave

openssl verify -CAfile ca.pem -show_chain -x509_strict ~/Downloads/SERVER.cert

gave

O = ZZZZ, OU = SSS, CN = SERVER
error 24 at 0 depth lookup: invalid CA certificate
error /home/colinpaice/Downloads/SERVER.cert.bad: verification failed

I felt I was on the trail of the problem.

Depth 0 is the server certificate. Depth 1 is the signers certificate, Depth 2 is the next level up. It was strange that it said the server certificate was an invalid CA.

Looking in the openssl source openssl/crypto/x509/x509_vfy.c showed several lines giving the error code X509_V_ERR_INVALID_CALL.

It turns out that the server’s certificate had been configured to to “certsign” (I had mis-copied a line in the certificates definition from an earlier test). Certsign means the certificate is a CA because it can sign things – but the CA flag was not set in the certificate – so clearly it was a bad_certificate. The validation failed – as it failed the consistency check it reported unknown certificate, and I was getting very frustrated.

What keyusage is required?

Any keyusage from no keysusage to KEYUSAGE( DATAENCRYPT, DOCSIGN, HANDSHAKE) works.

Just do not use KEYUSAGE(CERTSIGN) .

Recreate the certificate, and add it to the key store, and use

f CSQ9WEB,refresh,keystore

to get the MQ web server to pick up the change to the keystore. You may need to restart your web browser.

I think this is a useful technique which I will use in the future when I stumble over the next TLS set up problem.

What’s the difference between MQ Web, and z/OS Connect MQ support?

With MQ Web

  1. You can issue commands to administer MQ  for example display, define, delete MQ objects.
  2. You can put and get messages to and from a queue.  The message is what you specify – typically a character string.

With Z/OS Connect MQ support

  1. You can put and get messages to and from a queue, and do transformations on the message.  For example mapping a COBOL structure to JSON.  
  2. You can do field validation.
  3. You can covert HTTP code “200” to “great it worked”.

What is common?

They both use z/OS WebSphere Liberty to provide the basic web server.

Planning your Liberty – this is not an escape plan

With the web being the new front end to z/OS, most z/OS products are using Liberty as their web server to deliver web content.   Each product seems to be documented as if it is the only Liberty instance on the z/OS image, for example they all default to use http port 9080.

This blog post helps identify what planning you need to do before you can configure a Liberty instance, be it z/OSMF, MQWEB, CICS, or zOS Connect EE.

Roles

Often the person installing and configuring the server is part of the “installation team”, and may not be familiar with the  detailed use of the product, or product based tooling, for example the eclipse based  z/OS tooling.   This person’s role is to configure the server so it meets the enterprise requirements, and the configuration within the server is down to the team who requested it.

Update proclib

You need to decide if each instance will need its own procedure in proclib, or one procedure can start multiple servers.

You will need an Angel process – you can have one Angel started task across all of your Liberty instances on an LPAR, or have multiple Angel tasks.  To be shared, an Angel task needs to be at the highest service level.

If you want isolation you may want to set up an test instance with a different started task userid to the production instance.

Where do the product ZFS libraries go

Usually the product code goes in /usr/lpp/IBM/product_name. Typically you make a directory /usr/lpp/IBM/product_name, then mount the product file system over this directory.  Some times this file system needs to be mounted RW during customisation.   When you upgrade you can just mount the new file system instead of the old file system.

Where do the configuration files go?

The Liberty configuration files can go in /var/… or /u/… . If you intend for the server to be started on another LPAR, then the configuration files need to be available on the other LPAR.  Having a variable with the LPAR name as part of the directory will not work.   The configuration files are defined with the WLP_USER_DIR environment variable. You may want shell scripts which define this variable.  For example the shell script prodmqweb could have export WLP_USER_DIR=/u/mqweb/production/MQPA.   You then use sh prodmqweb to define the variable, or pass commands to it, such as sh prodmqweb dspmqweb so you can be sure you are using the right configuration file.

UNIX Directory List Utility ISPF 3.17 has space for 56 characters in the default directory name, but only 40 when working with files, such as new file or rename.   You may want to have a short prefix, or use an alias. For example /u/zoscA/servers/stockManager/server.xml instead of /MVSA/var/zosconnect/servers/stockManager/server.xml

What configuration files will be shared?

If you already have Liberty running in your environment you may be able to reuse some files, and include them in your server.xml. For example if your keystores are defined in a file you could use <include location=”…./keystore.xml”/> .

If you are providing duplicate servers for availability, you can put you common definitions in one file and share this, and server specific definitions in a different file.

It is easier to manage the configuration files if you provide small function specific files.  For example saf.xml, trace.xml,applications.xml, andr keystores.xml .

What TCPIP ports will be used

Each product will need its own ports, typically one port for http, and another port for https.   You can define multiple ports with different characteristics.  You use httpEndpoint and can specify for this port log this information, for that port log different information to a different place.

If you want to have two instances running on an LPAR using the same port,  the port needs to be defined as SHAREDPORT.

You may want to have the same port defined on each LPAR, so no matter which LPAR is used, use port 9443.

You may want to use VIPA so you have one external IP address into your SYSPLEX, and  z/OS SYSPLEX Distributor to route connection requests or distribute connection requests to available servers. If you want to do this you will need a configured VIPA TCPIP address.

You may need to specify which TCPIP instance to use if you have more than one TCPIP instance on an LPAR.

What keystores will be used

You need two keystores

  1. To identify the server – the keystore
  2. To validate certificates passed into the instance – the trust store.  Typically this has Certificate Authority certificates, and any self signed certificates.

These can be file based or SAF based using z/OS keyrings.  For example <keyStore filebased=”false” id=”racfKeyStore”
location=”safkeyring://START1/KEY” password=”password” readOnly=”true” type=”JCERACFKS”/> 

You might have enterprise keystores available to every one, or provide isolation so you have keystores for bank1 servers, and different keystores for bank2 servers.

Defining the APPL and SERVER resource

The Liberty default  APPL definition is BBGZDFLT. This allows people to access the server (the front door).  If you already have a Liberty installed then you may be able to use the existing definition.

If you want isolation, for example test and production, or between two major applications you will need to select and define different APPL resources.

You will need

<safCredentials profilePrefix="ZZZZDFLT"

and

RDEFINE APPL ZZZZDFLT UACC(NONE) 
PERMIT ZZZZDFLT CLASS(APPL) ACCESS(READ) ID(START1) 
PERMIT ZZZZDFLT CLASS(APPL) ACCESS(READ) ID(GRALL) 
SETROPTS RACLIST(APPL )REFRESH 

RDEFINE SERVER BBG.SECPFX.ZZZZDFLT UACC(NONE) 
PERMIT BBG.SECPFX.ZZZZDFLT CLASS(SERVER) ACCESS(READ) ID(START1) 
SETROPTS RACLIST(SERVER,APPL)REFRESH

   /* for z/OS Connect
RDEFINE EJBROLE ZZZZDFLT..zos.connect.access.roles.zosConnectAccess   +
   UACC(NONE) 
PERMIT ZZZZDFLT..zos.connect.access.roles.zosConnectAccess + 
   CLASS(EJBROLE)ACCESS(READ) ID(IBMUSER) 
SETROPTS RACLIST(EJBROLE )REFRESH 

   /* for MQ Web
RDEFINE EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin UACC(NONE)

These tend to be mixed case, so take care when defining them.

Starting the instance

If you use

S BAQSTRT,PARM='server1'
S BAQSTRT,PARM='server2'

If you use STOP BAQSTRT then both servers will stop.

If you use

S BAQSTRT.BAQ1,PARMS='server1' 
S BAQSTRT.BAQ2,PARMS='server2'

You can use STOP BAQ1 to stop just server1.

You can also use

S BAQSTRT,PARMS=’server1′,jobname=BAQ1
S BAQSTRT,PARMS=’server2′,jobname=BAQ2

Then you can use P BAQ1, and also have WLM give BAQ1 and BAQ2 different service classes, and so give them different priorities

Set up monitoring

There may be SMF data available, which you can collect if you enable the SMF collection classes.

You may have data from JMX which you collect and report.

Accessing the server

You will need to set up a profile and give permission to groups of people, just to be able to use the server.

You may need to protect individual applications, for example ability to start or stop applications, or to invoke the application.  This can be done once the basic setup has been done, and the system handed over.

For example in server.xml for z/OS connect

<zosconnect_service> 
<service name="stockquery" 
    serviceDescription="stockQueryServiceDescriptionColin" 
    id="stockQueryService" 
adminGroup="a3Admin,a4Admin" 
invokeGroup="a3Invoke" 
operationsGroup="a3Ops" 
readerGroup="a3Reader" 
/> 
</zosconnect_service>

It is good policy to only grant access to groups, and not individual ids as it simplified userid administration.  You would define a3Admin, a4Admin, a3Invoke as groups

HA Liberty web server – implementing VIPA with distributing connections.

Overview of VIPA solutions

You can implement VIPA, where you give your application its own IP address, across multiple TCPIP images.   This solves the problem of certificates not matching the host IP address.

You can have

  • One TCPIP image processing the connection requests. You have multiple TCPIP images – but only one TCPIP image at a time processes the connections.   If the TCPIP image stops, another can take over.
  • Multiple TCPIP stacks can process connection requests. A front end TCPIP image takes the connection requests and distributes them to TCPIP instances where the application is running.   You can use load balancing across multiple TCPIP images such distributing the connection techniques such as Round Robin, or Hot Standby.  This is based on Sysplex Distributor technology.

This blog post discusses the second case.

To provide background information, I created

Sysplex Distributor background

Sysplex Distributor is like having a router inside your TCPIP on z/OS; it can route traffic transparently to other TCP images in the environment.  The Sysplex Distributor can distribute connection requests for a  VIPA requests to TCPIP images where the application is running.    This set up is called Distributed VIPA (DVIPA).

It took me about 2 weeks to get a Sysplex DVIPA working – about a week understanding the documentation on VIPA, and the other week trying to understand why it didn’t work – and the simple configuration error I had.

I’ll break it down into simple stages which should help you understand the documentation.

The scenario

The scenario I used was going from my laptop, to z/OS running under zPDT on my laptop.  In effect there was an OSA connection between Ubuntu and my z/OS LPAR.  If you do not know what an OSA is, think of it as an Ethernet connection which can plug into multiple z/OS LPARs.

  1. My Ubuntu had an address of 10.1.1.1 over the tunnel connection
  2. I had one LPAR  with three TCPIP images
    1. TCPIP, host address 10.1.1.1 for the primary “front end”
    2. TCPIP2, host address 10.1.1.2 for the backup “front end” and where a server instance was running
    3. TCPIP3, host address 10.1.1.3 with a server instance running.
  3. I used a VIPA address of 10.1.3.10

The steps are

  1. Connect from Ubuntu to the z/OS
  2. Configure the LPAR(s)
  3. Define the VIPA configuration
  4. Define the “routing” to where the server was running.
  5. Getting the server to use the VIPA
  6. Commands to see what is going on (or not as the case may be)

Connect from Ubuntu to the z/OS

I had an existing tunnel connection from Ubuntu to z/OS.

I used the ip route command

sudo ip r add 10.1.3.10 link tap0

to define the route to 10.1.3.10 via the tunnelling device tap0 .  This looks like an OSA connection into z/OS.

Configure the LPAR(s)

The Sysplex Router uses XCF communications between LPARs and TCPIP images on the LPARs.

You configure each TCPIP with a statement

IFCONFIG DYNAMICXCF 172.1.2.x

My “frontend” TCP/IP had  IFCONFIG DYNAMICXCF 172.1.2.1, the other two TCP/IP images had 172.1.2.2 and 172.1.2.3.

The 172.1.2.x address can be any address not used by your enterprise.  It is internal to the Sysplex configuration.

Define the VIPA configuration

You define the configuration once, in the front end TCPIP.  It is visible from the other TCP/IP images because the information is shared via the DYNAMICXCF.

You define the VIPA in the front end TCP/IP image with a VIPADEFINE netmask address.  I used

VIPADYNAMIC
  VIPADEFINE 255.255.255.0 10.1.3.10
...
ENDVIPADYNAMIC

You can define VIPABACKUP in another TCPIP image, so if the main front end TCP/IP is not available then a backup can take the traffic and distribute it to the other TCP/IP stacks.

When the main front end TCP/IP image is restarted, you can have it take back the routing.

Define the “routing” to where the server was running.

You can define a variety of ways of routing the work

  • A Hot Standby – where the input to the front end TCP/IP image is routed to a single “backend” application’s TCP/IP image.  If this fails, the work is routed to a running backup application.
  • A Round Robin – where requests are routed in turn to each TCPIP with an active application.
  • Routing depending on WLM or other load characteristics.

This routing is done by the VIPADISTRIBUTE command in the front end TCP/IP image.

The definitions for the front end TCPIP

IPCONFIG SYSPLEXROUTING 
    DYNAMICXCF 172.1.1.1 255.255.255.0 3 

VIPADYNAMIC 
   VIPADEFINE 255.255.255.0 10.1.3.10 

   VIPADISTRIBUTE DEFINE DISTM ROUNDROBIN 10.1 .3.10 PORT 8443 
      DESTIP 
         172.1.1.2 
         172.1.1.3 
ENDVIPADYNAMIC 

This routes connection requests to the TCPIP images with the DYNAMICXCF of 172.1.1.2 (TCPIP2) and 172.1.1.3 (TCPIP3)

The definitions for TCPIP2

IPCONFIG SYSPLEXROUTING 
DYNAMICXCF 172.1.1.2 255.255.255.0 3 

The definitions for TCIP3

IPCONFIG SYSPLEXROUTING 
DYNAMICXCF 172.1.1.3 255.255.255.0 3

Use of VIPABACKUP

If the backup TCPIP front end image it used, it can have its own VIPADISTRIBUTE statement, or just use the same statement shared from the main front end TCP/IP image.  It is better to have the VIPADISTIBUTE statements, for the case when the backup TCPIP is started before the front end TCPIP.   The backup needs the VIPADISTRIBUTE statements. (These statements can be put into a PDS, and included using the INCLUDE dataset(member) statement in both primary and backup environments.)

To define TCPIP2 as a backup I used

VIPADYNAMIC 
    VIPABACKUP MOVEABLE IMMEDIATE 255.255.255.0 10.1.3.10 
    VIPADISTRIBUTE DEFINE DISTM ROUNDROBIN 10.1.3.10 PORT 8443 
        DESTIP 
        172.1.1.2 
        172.1.1.3 
ENDVIPADYNAMIC 

Getting the server to use the VIPA

The TCP/IP images hosting the applications just have the IFCONFIG DYNAMICXCF aa.bb.cc.dd statement.  They do not have any VIPADYNAMIC … ENDVIPADYNAMIC statements unless they are the main or backup front end TCP/IP images.

The application can connect using the VIPA address, for example create the SSLSOCKET Listener passing the VIPA address.   You can also configure TCP/IP so when a port is used, it binds to a particular IP address for example

PORT 8443 BIND 10.1.3.10

So an application using port 8443 to listen, will get IP address 10.1.3.10 – which in my case is a VIPA address.

You can use

PORT
9443 TCP * SHAREPORT BIND 10.1.3.7

to allow the port to be shared by many applications on a TCPIP Instance.

How are the connections distributed?

The VIPADISTRIBUTE  has many routing options. I used Hot Standby and Round Robin.

With RoundRobin, I had

  • the front end TCPIP
  • TCPIP2 with two Liberty servers
  • TCPIP3 with  one Liberty server

I ran some workload and found that the server on TCPIP3 had half the requests, and each of the two servers on TCPIP2 had a quarter of the overall requests.  This shows the routing is done at the TCPIP level – not the number of servers.

HA Liberty web server – implementing VIPA using the simpler VIPARANGE technology

Overview of VIPA solutions

You can implement VIPA, where you give your application its own IP address, across multiple TCPIP images.   This solves the problem of certificates not matching the host IP address.

  • One TCPIP image processes the connection requests. You have multiple TCPIP images – but only one TCPIP image at a time processes the connections.   If the TCPIP image stops, another can take over.
  • Multiple TCPIP stacks can process connection requests. This uses Sysplex Distributor;  a front end TCPIP image takes the connection requests and distributes them to TCPIP instances where the application is running.   You can use load balancing such as Round Robin, or Hot Standby.

This blog post discusses the first case.

To provide background information, I created

Using VIPARANGE configuration

The technique uses the VIPARANGE configuration statement.

The concept is that many LPARs can be attached to an OSA adapter, one, just one,  TCPIP stack (I dont know which of the available images) takes the connection requests and passes them on to the application on that TCPIP image.

You allocate a range of TCPIP address for your applications, with the same network prefix, for example 9.4.6.x   Allocate a host id to a Liberty, for example 9.4.6.7.   The Liberty instance keeps this address for whereever it runs.  You configure your routers so that  9.4.5.* are routed to the OSA adapter.

For each TCPIP image where you want to run Liberty, add to the  TCPIP startup configuration (or to an OBEY file)

VIPADYNAMIC 
   VIPARANGE DEFINE 255.255.255.0 9.4.6.7 
ENDVIPADYNAMIC

The 255.255.255.0 is the  subnet mask.  If your organisation uses a different subnet mask, it affects the IP addresses used.

These instructions say that they are defining a range of IP addresses on this LPAR, for  9.4.6.1 to 9.4.6.254

If an application connects to TCPIP, and the bind specifies a value in this range (9.4.6.1 to 9.4.6.254) then it is considered a VIPA address.  If the application used 9.4.6.7 this would count as a VIPA address.

When your application (Liberty) connects to TCP and uses an address in the VIPARANGE,  the TCPIP instance will create a dynamic IP address.   When I started my server application,   I got a z/OS console message

EZD1205I DYNAMIC VIPA 9.4.6.7 WAS CREATED USING BIND BY jobname ON TCPIP2.

When I shut down the server I got

EZD1298I DYNAMIC VIPA  9.4.6.7 DELETED FROM TCPIP2
EZD1207I DYNAMIC VIPA 9.4.6.7 WAS DELETED USING CLOSE API BY jobname ON TCPIP2

If the VIPA address is active on more than one TCPIP image, just one image will get all of the requests.  If you stop this image, another TCPIP image can take over.

If you have a different server using the same IP address, but a different port number, because they use the same IP address, the same LPAR will process the requests.

With VIPAROUTE you do not get connections distributed to more than one TCPIP image.

In your browser use  9.4.6.7:9443 address, the network router, routes this to the OSA, a TCPIP captures this and passes it to the application (Liberty).   As part of the handshake Liberty sends down its certificate, which has a SAN of  9.4.6.7 which matches the IP address, so this works.

On another day, when a different z/OS image is capturing the VIPA address connections,  the TCPIP address is 9.4.6.7 as before, so this matches the SAN in the certificate.

Testing it

In a test I used “ping -R 9.4.6.7 ” to the VIPA address.
This reported it was sent to TCPIP stack with 10.1.1.2. When I shut this TCPIP image down, ping reported the request was sent to 10.1.1.3.  It did this with no manual intervention.

 

Setting up a highly available web server is a “yes – but” problem

I’ve been setting up a Liberty web server, as used in MQWEB, z/OS Connect, z/OSMF and so on, and was looking into how to make this available, so I could move the web server to a different LPAR or TCP/IP instance.

Moving it should be easy – it is  – but …  but there are things you need to think about. It is a bit like going around a maze trying to find the solution.

How do I get to the fail over system?

You start the web server on a different LPAR in the sysplex. How can you support this to allow your browser to get to the backend, without changing the URL?

You have two choices.

  1. You change your DNS look up, or router so your request goes via a different connection (think different bit of wire) to the failover LPAR. These change can be automated to some extent.
  2. Multiple z/OS images can listen for an IP address.

These work but…

The certificate sent down from the web server contains the address of the LPAR as part of the SAN.  When the browser processes it, it compares the LPAR address in the certificate with the address in the certificate.  If they do not match the browser produces an error message.

How do I get over the certificate SAN and the IP address difference?

You have a couple of choices

  1. Use a unique certificate on each LPAR.    Yes this works, but there is more administrative work to set up.  You could set up two web servers and only use one at a time.   This work, but it is unnecessary work.
  2. Use a Virtual IP address.   In TCP networking the end of every connection is a “device” or system with its unique IP address.   You can give the web server its own IP address which is “virtual” as it is not  device or system.  With this, when you start your web server on a different LPAR, it has the same IP address.  To use this you have to configure z/OS to support this.  You can set this up
    1. To support multiple web servers, and distribute the work to them
    2. Have a hot standby
    3. To route traffic to where the web server has started.

Yes, these work, but – is not easy to set up.  I’ll be blogging how to do this.

Defining a second TCPIP stack on z/OS on zPDT

I wanted a second TCPIP stack on my z/OS because I wanted to test it with MQWEB.   There is no good documentation in one place, there is good documentation hidden away, but not all in one place.
This took me about half a day to set up -including several IPLs , but I am on my own z/OS zPDT image so this was not a problem.  It take a while to understand the definitions – it is another one of “this point to that which points to something else…”.   You need to be able to copy a definition rather than use the books to create it from nothing.

I’ll describe setting up TCPIP2.

Overall I was surprised at how easy this bit was to set up.

The work breaks into

  • setting up the connectivity from Linux to z/OS
  • setting up the second TCP stack
    • Configure sys1.parmlib memmber and IPL
    • Define the new TCPIP procedure
    • Configure the new TCPIP configuration
    • Allowing people to use the TCPIP stack

Both of these need an IPL of z/OS, so you could do all of the customising and IPL afterwards at the end.

I’ll cover sharing an existing OSA adapter and setting up a new OSA adapter.

Sharing an existing OSA adapter.

Copy ADCD.Z24A.VTAMLST(OSATRL2) to USER.Z24A.VTAMLST(OSATRL2) and make the changes in bold

OSATRL1 VBUILD TYPE=TRL 00010000
OSATRL1E TRLE LNCTL=MPC,READ=(0400),WRITE=(0401), X00020007
               DATAPATH=(0402,0404,040,0406),     X00021013
               PORTNAME=PORTA,                    X00022004
               MPCLEVEL=QDIO                       00023005
*SATRL2E TRLE LNCTL=MPC,READ=(0404),WRITE=(0405),DATAPATH=(0406), X00024011
* PORTNAME=PORTB, X00025011
* MPCLEVEL=QDIO 00026011

I changed

  • DATAPATH=(0402) to DATAPATH=(0402,0404,0406)  – note every other address.    With 0402,0403 etc in the list, the second TCP failed to work, with messages like
    • EZZ4310I ERROR: CODE=80100040 REPORTED ON DEVICE PORTA. DIAGNOSTIC CODE: 03
    • EZZ4309I ATTEMPTING TO RECOVER DEVICE PORTA
    • IST1222I DATA DEVICE 0403 IS INOPERATIVE, NAME IS PORTA
    • IST1578I DEVICE INOP DETECTED FOR PORTA BY ISTTSCMA CODE = 104
  • Commented out/deleted the second TRLE definition

The zPDT devmap needs to have OSA definitions for these

name awsosa 0009 --path=A0 --pathtype=OSD --tunnel_intf=y # QDIO mode
device 400 osa osa --unitadd=0
device 401 osa osa --unitadd=1
device 402 osa osa --unitadd=2
device 403 osa osa --unitadd=3
device 404 osa osa --unitadd=4
device 405 osa osa --unitadd=5
device 406 osa osa --unitadd=6

I created a file USER.Z24A.TCPPARMS(T2OSA)

DEVICE PORTA  MPCIPA 
LINK ETH1  IPAQENET PORTA 
START PORTA 
HOME 10.1.1.3 ETH1 

and put

include USER.Z24A.TCPPARMS(T2OSA)

into my tcpip2 startup.

By putting the definitions in a PDS member, means I can use

V TCPIP,TCPIP2,OBEY,USER.Z24A.TCPPARMS(T2OSA)

to activate them.

I reipled the system to pick up VTAM changes.

Once I had stared TCPIP and TCPIP2 the command d net,id=OSATRL1E gave

D NET,ID=OSATRL1E
IST097I DISPLAY ACCEPTED
IST075I NAME = OSATRL1E, TYPE = TRLE 466
IST486I STATUS= ACTIV, DESIRED STATE= ACTIV
IST087I TYPE = LEASED , CONTROL = MPC , HPDT = YES
IST1954I TRL MAJOR NODE = OSATRL2
IST1715I MPCLEVEL = QDIO MPCUSAGE = SHARE
IST1716I PORTNAME = PORTA LINKNUM = 0 OSA CODE LEVEL = 7617
IST2337I CHPID TYPE = OSD CHPID = A0 PNETID = **NA**
IST1577I HEADER SIZE = 4096 DATA SIZE = 0 STORAGE = ***NA***
IST1221I WRITE DEV = 0401 STATUS = ACTIVE STATE = ONLINE
IST1577I HEADER SIZE = 4092 DATA SIZE = 0 STORAGE = ***NA***
IST1221I READ DEV = 0400 STATUS = ACTIVE STATE = ONLINE
IST924I -------------------------------------------------------------
IST1221I DATA DEV = 0403 STATUS = ACTIVE STATE = N/A
IST1724I I/O TRACE = OFF TRACE LENGTH = *NA*
IST1717I ULPID = TCPIP ULP INTERFACE = PORTA
...
IST1221I DATA DEV = 0404 STATUS = ACTIVE STATE = N/A
IST1724I I/O TRACE = OFF TRACE LENGTH = *NA*
IST1717I ULPID = TCPIP2 ULP INTERFACE = PORTA
IST2310I ACCELERATED ROUTING DISABLED
IST924I -------------------------------------------------------------
IST1221I DATA DEV = 0405 STATUS = RESET STATE = N/A
IST1724I I/O TRACE = OFF TRACE LENGTH = *NA*
IST924I -------------------------------------------------------------
IST1221I DATA DEV = 0406 STATUS = RESET STATE = N/A
IST1724I I/O TRACE = OFF TRACE LENGTH = *NA*
IST924I -------------------------------------------------------------
IST1500I STATE TRACE = OFF

Setting up the connectivity from Linux to z/OS using a second OSA adapter

You need to set up an interface from Linux to z/OS via an Open Systems Adapter (OSA).

TCP/IP Interfaces are used to tunnel from Linux to z/OS.  These have names like tap0, tap1;  they tie up with z/OS paths and devices.  The Linux device drivers implement the QDIO protocol, a simpler and faster protocol than traditional z/OS channels.

Identify the path and devices to be used.

The zPDT find_io command gave me

 FIND_IO for "colin@colin-ThinkCentre-M920s" 

      I/face Cur           MAC      IPv4         IPv6 
Path  Name   State         Address  Address      Address 
----  ----   ---- -------- -------- -------       ----------------- ---------------- -------------- 
F0    eno1    UP, RUNNING 00:d8:... 10.1.0.3     fe80:...%eno1 
F1    wlxd..  UP, RUNNING d0:37:... 192.168.1.67 2a00:...6cab 
.  
A0   tap0     UP, RUNNING 9e:30:... 10.1.1.1     fe80:... %tap0 
A1   tap1     UP, RUNNING 7e:66:... 0.1.2.1      fe80:... %tap1 
A2   tap2   DOWN 02:a2:a2:a2:a2:a2  *            *

We can see from this the IP addresses being used;  channel paths A0, A1 are in use by tunneling; channel path A2 is available.

In the zPDT devmap I set up

[manager] # tap0 define network adapter (OSA) for communication with Linux
name awsosa 0009 --path=A0 --pathtype=OSD --tunnel_intf=y # QDIO mode
device 400 osa osa --unitadd=0
device 401 osa osa --unitadd=1
device 402 osa osa --unitadd=2

[manager] # tap1 define network adapter (OSA) for communication with Linux
name awsosa 0010 --path=A1 --pathtype=OSD --tunnel_intf=y --tunnel_ip=10.1.2.1 --tunnel_mask=255.255.255.0 # QDIO mode
device 408 osa osa --unitadd=0
device 409 osa osa --unitadd=1
device 40a osa osa --unitadd=2

Where the paths tie up with the output from the find_io.

Each connection needs 3 consecutive devices, for example 408,409,40a.

On z/OS use the command D U,CTC to find which devices are available.  I think (I am not sure) that the first device has to end in 0, or 8 .

I have

UNIT TYPE STATUS 
0400 OSA A-BSY 
0401 OSA A 
0402 OSA A-BSY 
0403 OSA OFFLINE 
0404 OSA OFFLINE 
0405 OSA OFFLINE 
0406 OSA OFFLINE 
0407 OSA OFFLINE 
0408 OSA A-BSY 
0409 OSA A 
040A OSA A-BSY

Once you have selected the OSA addresses to use, and configured the devmap file, you will need to restart zPDT with the updated devamp – but you need to customise z/OS and IPL – so do not IPL just yet.

Z/OS work for setting up the second TCP stack

Some basic terminology and concepts.

  • There is an network domain AF_INET which programmers use via sockets to communicate with the network.   (There is another network domain AF_UNIX for Unix programming).
  • You have to configure the domain, for example how many concurrent sessions it can support.
  • Originally you could have only one TCP stack in the environment.   This used an interface called INET.  This did not support more than one TCP/IP stacks.
  • A new interface was developed Common INET ( CINET). Conceptually this sits in front of TCP/IP and routes packets to the TCPIP subsystems.
  • To be able to use multiple stacks, CINET needs to be used instead of INET.
  • These are customised in SYS1.PARMLIB(BPXPRMxx).

Customise sys1.parmlib(BPXPRMxx) member

For example

FILESYSTYPE TYPE(INET) ENTRYPOINT(EZBPFINI) 

SUBFILESYSTYPE NAME(TCPIP) 
     TYPE(INET) 
     ENTRYPOINT(EZBPFINI) 

NETWORK DOMAINNAME(AF_INET) 
     DOMAINNUMBER(2) 
     MAXSOCKETS(64000) 
     TYPE(INET) 
     INADDRANYPORT(5555) 
     INADDRANYCOUNT(1000)

Change TYPE(INET) to TYPE(CINET) in 3 places, and change ENTRYPOINT(EZBPFINI) to ENTRYPOINT(BPXTCINT)

Add the new TCPIP address space

SUBFILESYSTYPE NAME(TCPIP2) 
     TYPE(CINET) 
     ENTRYPOINT(EZBPFINI)

This change needs a IPL to activate (or possibly a SETOMVS RESET=(xx).   I do not know what else the change from INET to CINET affects, so check with IBM before implementing it.

Define the TCPIP2 procedure

  • I defined a new profile in the STARTED class to map TCPIP2 to a userid.   I used the same userid as for TCPIP.
  • I copied the TCPIP procedure from TCPIP to TCPIP2.
  • The TCPIP procedure refers to TCP configuration,
    • //PROFILE  … DSN=SYS1.TCPPARMS(PROF) and
    • //SYSTCPD …. DSN=SYS1.TCPPARMS(TCPDATA).
  • Create your own copies of these, for example copy them to USER.TCPPARMS, and rename the members to PROF2, and TCPDATA2

Create VTAM definition for the tunnelling connection for the a second OSA adapter

 

If you are using a second OSA adapter, you need to create a VTAM member to map from the OSA device to a TCP/IP name using MPC.  This is Multi Protocol Channel, using protocol QDIO which is simpler and faster protocol than traditional z/OS channels.

Create a member in VTAMLST for a Transport Resource List major node, for example OSACOLIN.

----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8
OSA5 VBUILD TYPE=TRL 
OSATRL5E TRLE LNCTL=MPC,READ=(0408),WRITE=(0409),DATAPATH=(040A),      X
               PORTNAME=PORTZ,                                         X
               MPCLEVEL=QDIO

Note the format, continuation ‘x’ in column 72, and continuation text in column 16.

You can use V NET,ACT,ID=OSACOLIN  to activate it. If you use D NET,IS=OSACP,E it should find it, and report it is active.

You can use D NET,TRL  to display the status of the  links.

Configure TCPPARMS(PROF2)

I used a copy of the TCPIP(PROF) as my starting configuration.

I commented out all of the lines between AUTOLOG and ENDAUTOLOG.

I went down to the DEVICE and BEGINROUTE section and used

DEVICE PORTZ MPCIPA
LINK ETHZ IPAQENET PORTZ
; end of link and device definitions
;
HOME 10.1.2.2 ETHZ
;
BEGINRoutes
;            Destination SubnetMask FirstHop LinkName Size
ROUTE        DEFAULT               10.1.2.1  ETHZ     MTU 1492
ENDRoutes
; start it when TCP/IP starts
START PORTZ

Where

  • DEVICE PORTZ MPCIPA  –  MPCIPA says this is an OSA QDIO, and uses the PORTZ definition.  PORTZ was defined above in the VTAMLST(OSACP).
  • LINK ETHZ IPAQENET PORTZ –  this create a LINK ETHZ associated with DEVICE PORTZ in the line above. It uses the interface type IPAQENET, which is for IP V4 and device OSA QDIO.   (There is IPAQENET6 for IP V6 for OSA QDIO).
  • HOME 10.1.2.2 ETHZ –  for traffic coming in over ETHZ (via PORTZ, and back to the tap1 which was defined with –tunnel_ip=10.1.2.1).   A ping 10.1.2.2 should come in over this interface.  For the first OSA adapter this had 10.1.1.2.
  •  BEGINRoutes
    ;                Destination SubnetMask FirstHop LinkName  Size
    ROUTE  DEFAULT                               10.1.2.1          ETHZ    MTU 1492
    ENDRoutes

    •  Any traffic going to 10.1.2.1 go via link ETHZ and use a packet size of 1492 bytes.
  • START PORTZ – get it working

Edit the TCPDATA

For sharing an OSA or using a new OSA, I edited the TCPDATA2 file and added

TCPIPJOBNAME TCPIP2
S0W1: HOSTNAME S0W1COL
DOMAINORIGIN COLIN.HOST.COM
DATASETPREFIX TCPIP
NSPORTADDR 53
RESOLVEVIA UDP
LOOKUP LOCAL
ALWAYSWTO YES

I dont know which of these are important.  I changed the bold lines, to match my name.

RACF profile changes

You have to set up a security  profile before an application can connect to TCPIP and listen on a socket.  MQWEB got EDC5112I Resource temporarily unavailable. (errno2=0x74610296)

rdefine SERVAUTH EZB.INITSTACK.*.TCPIP2  from(EZB.INITSTACK.*.TCPIP)

Using the model… above copies the permission from the base object.   You can allow more users using

permit EZB.INITSTACK.*.TCPIP2 class(SERVAUTH) id(START1) access(READ)

The “*” is for any system in the sysplex, so you could have EZB.INITSTACK.MVSA.TCPIP2 and allow access to TCPIP2 on system MVSA, but not from another MVS system.

You can protect TCPIP2 for example protect the NETSTAT command

RDEFINE SERVAUTH (EZB.NETSTAT.*.TCPIO2.*) UACC(NONE)
PERMIT (EZB.NETSTAT.*.TCPIP2.*) ACCESS(READ) CLASS(SERVAUTH) ID(TCPADMIN)
SETROPTS GENERIC(SERVAUTH) REFRESH 

Check it out

You can use the Linux netstat -i command to display the interfaces defined to Linux.  On my Linux  I got

colin@colin-ThinkCentre-M920s:/home/zPDT$ netstat -i 
Kernel Interface table
Iface     MTU   RX-OK ... Flg
eno1     1500   84758 ... BMRU
lo      65536  188855 ... LRU
tap0     1500       6 ... BMRU
tap1     1500      25 ... BMRU
wlxd0374 1500   10545 ... BMRU

z/OS commands

D TCPIP – displays the TCP address spaces in the LPAR

D tcpip,tcpip2,netstat,home gave
EZZ2500I NETSTAT CS V2R4 TCPIP2 540
HOME ADDRESS LIST:
ADDRESS LINK FLG
10.1.2.2 ETHZ P
127.0.0.1 LOOPBACK

Using TCPIP2 from Liberty web server

I added

_BPXK_SETIBMOPT_TRANSPORT=TCPIP2

to the server.env file, and restarted Liberty

I connected from my web browser to MQWEB using 10.1.2.2:9443, and got the messages

Your connection is not private
Attackers may be trying to steal your information from 10.1.2.2
NET:ERR_CERT_COMMON_NAME_INVALID

The NET:ERR_CERT_COMMON_NAME_INVALID message is because the certificate had a Subject Alternative Name of a different IP address 10.1.1.2.  It traffic flow was sent from 10.1.2.2.

This was what I expected.