Tracing Liberty logon on z/OS – is difficult

I had a few problems logging on to the MQWEB server using certificates, and found there was no documentation to help resolve problems. The debug information you can get is often not very helpful!

As an extra twist, a userid having no access and getting a “not authorised to the resource” is OK. For example my userid may have access to MQWebAdmin, but not to MQWebAdminRO – it may be wrong to have access to both, so you will get at least one “not authorised to the resource” violation.

I looked at

MQWEB traces not enough information provided
RACF traces – looks wrong to me
RACF audit data in SMF – this is all you need

The only way of getting out the data, is to enable RACF audit for the profile, and set an option in the mqwebuser.xml file.

To make it even more difficult to resolve problems. When a request arrives at the web server, it encrypts the information, and sends down an LTPA token. The next request from the browser sends this token, and bypasses some of the initial checks, so you will not see trace entries. After the LTPA token has expired, the next request will do the full logon again.
To prevent this from happening, clear your browser history and cache before retesting.

MQWEB trace provides information – none of it usable.

I used the trace string

traceSpecification=”*=info:zos.native.03=fine”

I also included :UserRegistry=all:Credentials=all which gave more information, not all of it useful.

This provides information like

Description: Entry: checkAuthorizationFast 
serviceResults: 00000050868103e7 
suppresMessages: 0 
logOption: 3 
requestor: 
raco_cb: 0000005082d08290 
acee: 0000000000000000 
accessLevel: 2 
applName: MQWEB 
className: EJBROLE 
entityName: MQWEB.com.ibm.mq.console.MQWebAdminRO 
...
Description: RACROUTE REQUEST=FASTAUTH return 
   returnCode: 0 
   safReturnCode: 0 
   racfReturnCode: 0 
   racfReasonCode: 0

But does not tell you which userid the request was being made for!

Sometime it gives you full control blocks, other times truncated like MQWEB.com.ibm.mq so you do not know if this is for Admin, AdminRO, or User.

MQWEB safAuthorization racRouteLog

I enabled RACF AUDIT for the MQWEB.com.ibm.mq.console.MQWebAdminRO and MQWEB.com.ibm.mq.console.MQWebAdmin.

In the mqwebuser.xml you can have to display audit messages about EJBROLE

See here – which says

Specifies the types of access attempts to log.

ASIS Records the event in the manner specified in the profile that protects the resource, or by other methods such as the SETROPTS option.
NOFAIL If the authorization check fails, the attempt is not recorded. If the authorization check succeeds, the attempt is recorded as in ASIS.
NONE The attempt is not recorded.
NOSTAT The attempt is not recorded. No logging occurs and no resource statistics are updated.

With AUDIT enabled, and with racRouteLog=”ASIS” I got the following “failures” (every 10 seconds) due to the web server doing auto-refresh. The checks to MQWebUser worked, and were not recorded in the joblog.

15.51.54 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC) MQWEB.com.ibm.mq.console.MQWebAdmin CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
15.51.54 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC ) MQWEB.com.ibm.mq.console.MQWebAdmin CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
15.51.55 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC ) MQWEB.com.ibm.mq.console.MQWebAdminRO CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
15.51.55 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC )MQWEB.com.ibm.mq.console.MQWebAdminRO CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
15.51.55 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC )MQWEB.com.ibm.mq.console.MQWebAdmin CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
15.51.55 ICH408I USER(ADCDC ) GROUP(TEST ) NAME(ADCDC )MQWEB.com.ibm.mq.console.MQWebAdminRO CL(EJBROLE ) INSUFFICIENT ACCESS AUTHORITY ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

When I changed it to racRouteLog=”NONE” the messages were not produced on the joblog. There were still records produced in the SMF audit data with the profiles having AUDIT ALL(READ) specified.

I think you should usually run with racRouteLog=”NONE” , and change it to racRouteLog=”ASIS” when you have a problem – but be careful not to generate a flood of messages.

To display SAF messages about other violations use

RACF Trace

This gives data as to what was accessed, but reports the userid of the web server, not the userid being checked – so not very useful.

I used the command

#set trace(RACROUTE(ALL),JOBNAME(CSQ9WEB))
traced to GTF with USRP(F44)
formatted it with the IPCS command GTF USR(ALL)

This had data like

Trace Type: RACFPOST – this is the “AFTER” request
Service number: 00000002 – this is RACROUTE 2, verify
RACF Return code: 00000008
RACF Reason code: 00000004
MQWEB.com.ibm.mq.console.MQWebAdmin – this profile
EJBROLE – this class
MQWEB – this application
ACEE ( userid block) with userid=START1, Group=SYS1, Jobname=CSQ9WEB. This had a userid of START1 (from the job), but the userid being tested was for ADCDC which was not in the control blocks – so no good for telling you which userid had accces or not.

So all we can tell is, that for the profile EJBROLEMQWEB.com.ibm.mq.console.MQWebAdmin someone got a ‘no access’ return code.

Turn off the RACF trace using

#SET TRACE(NORACROUTE,NOJOBNAME)

RACF Auditing – this worked and gave me most of the information

I turned on RACF auditing using

RALTER EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin AUDIT(ALL,READ)
RALTER EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO AUDIT(ALL,READ)
RALTER EJBROLE MQWEB.com.ibm.mq.console.MQWebUser AUDIT(ALL,READ)
SETROPTS RACLIST(EJBROLE)

This writes a record to SMF for ALL(failed and successful) request which were READ or above.

I used the RACF provided exits to the SMF dump program(IFASMFDP). This produces readable file with all of the data.

I wrote an ISPF edit macro in rexx to take the data, and extract the key fields.

Below are the records produced for

an SSL connection using a certificate which mapped to userid ADCDB,
a logon with a userid and password with userid ADCDC

RESULT  USERID  WANT ALLOWED CLASS   RESOURCE
SUCCESS START1  READ /READ   SERVER  BBG.SECPFX.MQWEB 
SUCCESS ADCDB   READ /UPDATE APPL    MQWEB 
SUCCESS ADCDB   READ /UPDATE APPL    MQWEB 
SUCCESS ADCDC   READ /READ   APPL    MQWEB 
SUCCESS WSGUEST READ /READ   APPL    MQWEB 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO 
SUCCESS ADCDC   READ /READ   EJBROLE MQWEB.com.ibm.mq.console.MQWebUser 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO 
SUCCESS ADCDC   READ /READ   EJBROLE MQWEB.com.ibm.mq.console.MQWebUser 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO 
SUCCESS ADCDC   READ /READ   EJBROLE MQWEB.com.ibm.mq.console.MQWebUser 
SUCCESS ADCDC   READ /READ   EJBROLE MQWEB.com.ibm.mq.console.MQWebUser 
SUCCESS ADCDC   READ /READ   EJBROLE MQWEB.com.ibm.mq.console.MQWebUser 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdmin 
INSAUTH ADCDC   READ /NONE   EJBROLE MQWEB.com.ibm.mq.console.MQWebAdminRO

From the RESULT column, the userid ADCDC had

only READ access to MQWEB.com.ibm.mq.console.MQWebUser,
no access to MQWEB.com.ibm.mq.console.MQWebAdmin and MQWEB.com.ibm.mq.console.MQWebAdminRO, as we can see from the INSufficientAUTHority in the RESULT column. The resource wanted READ – but had NONE. This is OK, as we want the MQWebUser permissions.

From SUCCESS ADCDB READ /UPDATE APPL MQWEB we can see that userid ADCDB (from the certificate) wanted READ access, but had UPDATE access to to MQWEB in the class APPL.

It does not look very optimised code, it looks like the logic is like

hmm it looks like this userid does not have access to this EJBROLE, let me check again
and again
and again
and again
and again
and again
OK give up, and try another resource
hmm it looks like you do have access to this other resource -let me check again..
and again…
… ok you still have access – let’s go with this.

I expect is because a high level java program called a class to do some work, which checked the access; it called another class which did its own checks etc. Understandable, but not efficient coding.

This all worked, I could see all of the access requests, but sadly I did not get a record saying “this certificate…. was mapped to userid ADCDB”.

What no one tells you about defining your RACF resources – and how to do it for MQ.

Introduction

The RACF documentation has a lot of excellent reference materials describing the syntax of the commands, but I could not find much useful information on how to set up RACF specifically for products like MQ, CICS, Liberty etc.

It is bit like saying programming has the following commands, load, store, branch; but fails to tell you that you can do wonderful things like draw Mandelbrot pictures using these instructions.

I set up MQ on my z/OS system as an enterprise user – though I have an enterprise with just one person in it – me! With this view it shows what you need to configure.

I am not a RACF expert – but have learned as I go along. I believe this blog post is accurate – but I may have missed some set up considerations.

In this blog post I’ll cover

Security roles
RACF concepts – class and profile
Controlling who can create profiles and how to limit what they can create
MQ profiles
Planning for MQ
How do you copy the security profiles for a new queue manager?

A typical enterprise – from a security perspective

In a typical enterprise there are different departments

The security team responsible for the overall set up of security, ensuring that configurations are up to date (userids which are no longer needed are deleted).
product teams are responsible for defining the security profiles within their products, protecting resources, and giving people access to facilities.
- most developers do not have the authority to define profiles or give access. Many developers do not have a z/OS logon.

What do you need to protect?

There are four types of resources you can define

commands
resources
“logging on” or “connecting to”
turning off security, or making powerful commands generally available

For example, for z/OS

commands
- being able to issue z/OS commands
- being able to issue TSO commands
resources
- data sets
logging on
- which systems you can logon to
turning off security
- changing the RACF configuration

for MQ

commands
- being able to issue MQ commands to configure the queue manager
- being able to issue commands to define MQ queues etc
resources
- queues, channels etc
logging on
- which queue managers you can use, connect from batch, but not from CICS
defining the switches to disable parts of MQ security checking.

How do you protect a resource?

Resources are defined in classes. For example

class OPERCMDS define z/OS console commands
class MQCMDS for MQ commands
class MQQUEUE for MQ queues
class SERVER for managing access to servers such as Liberty and WAS

You need to go down a level, and protect resources within a class. You may want to allow one group of people to define resources for production and another group allow to define resources for test. You may have MQOPS allowed to define profiles for PROD… and TEST…. and TESTOPS only allowed to define resources TEST…..

You need CLass AUTHorisation (CLAUTH) on a userid to be able to define a resource. The CLAUTH does not exist for a group.

ALTUSER ADCDA CLAUTH(MQCMDS)

With this command userid ADCDA can now use commands like

RDEFINE MQCMDS MQPA.DISPLAY.** UACC(NONE) OWNER(MQM)

This says

Create an entry for class MQCMDS
Queue manager MQPA, any DISPLAY command, so DISPLAY USAGE, and DISPLAY QLOCAL would be covered
No universal access
The resource is owner by MQM. If this is a group, anyone with group special in group MQM can issue the PERMIT command on the resource

How specific a profile do I need?

For harmless commands, such as DISPLAY you can have a general profile MQPA.DISPLAY.* to cover all DISPLAY commands.

For commands that can change the system, you should use specific profiles, for example

RDEFINE MQCMDS MQPA.DEFINE.PSID UACC(NONE) OWNER(MQM)
RDEFINE MQCMDS MQPA.DEFINE.QLOCAL UACC(NONE) OWNER(MQM)
PERMIT MQPA.DEFINE.PSID CLASS(MQCMDS) ACCESS(READ) ID(MQOP1)
PERMIT MQPA.DEFINE.QLOCAL CLASS(MQCMDS) ACCESS(READ) ID(MQAMD1)

If you use the DEFINE.** then administrators can give themselves access to the operator DEFINE commands.

Limiting what profiles a user can manage

If you have RACF GENERICOWNER enabled (this is a system wide option) you can create profiles and grant people access within that group.

Turn on GENERIC OWNER

SETROPTS GENERICOWNER

Create a top level, catch-all case

RDEFINE MQCMDS ** UACC(NONE) OWNER(SYS1)

Create a profile limiting people in group ADCD to define resources with names MQPC.**

RDEFINE MQCMDS MQPC.** UACC(NONE) OWNER(ADCD)

If userid ADCDA in group ADCD tries to create a profile

RDEFINE MQCMDS MQPC.AA3 UACC(NONE)

it works, but

RDEFINE MQCMDS MQPZ.AA UACC(NONE)

gives ICH10103I NOT AUTHORIZED TO DEFINE MQPZ.AA.

The owner of a profile can give authority to anyone, there are no limits or checks.

Creating profiles for MQ

Using the categories described above

MQ commands
- being able to issue MQ commands to configure the queue manager
- being able to issue commands to define MQ queues etc
MQ resources
- queues, channels etc
connecting to MQ
- which queue managers you can use
defining the switches to disable parts of MQ security checking.

MQ Commands

Commands can be issued from

the operator console (SDSF)
with the MQ ISPF panels, messages are put to the SYSTEM.COMMAND.INPUT.QUEUE
Applications putting messages to the SYSTEM.COMMAND.INPUT.QUEUE
Applications using PCF to the SYSTEM.COMMAND.INPUT.QUEUE

If command checking is enabled then command are checked using the MQCMDS class.

Other commands, via the SYSTEM.COMMAND.INPUT.QUEUE, need to have permission to put to the queue, and the command is checked by the MQCMDS class.

MQResources

The queuing resources are have the following classes – MX… are for MiXed case names. A completely UPPER case queue name can still be protected if you choose to use the MXQUEUE class. That is “upper case” names are a subset of the “mixed case” names, and MYQUEUE is different from MyQueue.

MQQUEUE,MXQUEUE queue resources
MQPROC, MXPROC process (for example triggering)
MQNLIST, MXNLIST name list
MXTOPIC topics – Topics are always mixed case.

Connecting to MQ

MQCONN and you define resources like MQPA.BATCH CLASS(MQCONN)

Defining switches to disable parts of MQ security checking, and subset checks

MQADMIN, MXADMIN, Profiles:

Used mainly for holding profiles for administration-type functions. For example:

- Profiles for IBM MQ security switches
- The RESLEVEL security profile
- Profiles for alternate user security
- The context security profile
- Profiles for command resource security

For example the following turns off all RACF checking for the queue manager

REFINE MQADMIN MQPA.NO.SUBSYS.SECURITY

You can set up security so people are authorised to only a subset of objects.

You can set up

RDEFINE MQADMIN MQPA.QUEUE.TEST* OWNER(MQPAOPS)

to allow people access to a subset of queues – in this case queues beginning with TEST on queue manager MQPA. A user would need to be authorised to use RDEFINE MQCMDS MQPA.DEFINE.QLOCAL or (hlq.DEFINE.**) and authorised to RDEFINE MQADMIN MQPA.QUEUE.TEST*.

A thought on the MQ profile design.

It feels like the security was not well defined in this area. You want to allow someone to restrict someone’s access to only use a subset of queues, but the person may have the authority to turn MQ security off by giving them authority to create MQADMIN MQPA.NO.SUBSYS.SECURITY!

You can solve this using GENERICOWNER (which is optional) and

RDEFINE MQADMIN MQPA.NO.** UACC(NONE) OWNER(THEBOSS)

Looking back, rather than depending on the GENERICOWNER facility, I would have set up a class MQSWITCH to allow only the site RACF coordinator to define a switch and so turn off security.

Planning for security

You need to identify

the classes of profiles ( MQCMDS, MQQUEUES, z/OS OPERCMDS)
the subsystems being protected ( MQ, DB2)
the areas of profiles, TEST queues, Production tables for PAYROLL application
the roles of people and what they are expected to do – map each role to a group
- For each subsystem and class of profile what can each role do?
  - Production, Read Only operator commands, roles: all roles
  - Production, DEFINE PAGESET commands, roles: members of ZOPER group
  - Production, DEFINE QUEUE commands, roles: members of PRODADMN group
  - Test, DEFINE PAGESET commands, roles: members of ZOPER and TESTOPER groups
The hierarchy of groups. If you have defined a profile with owner TESTOPER, people can create resources in this group
- if they are in the TESTOPER group,
- or a user who has group-SPECIAL authority over the group which owns the TESTOPER profile
Define the profiles, the general MQPA.DISPLAY.**, and the specific MQPA.DEFINE.PAGESET, MQPA.DEFINE.QLOCAL

Another thought of MQ security design.

At the beginning of MQ 25+ years ago, this was before Sysplex, there was only a single LPAR, and typically only one queue manager, DB2 etc on each LPAR. These days people have many “identical” queue managers – which may be in a QSG or not.

When you create a new queue manager you have to replicate the security profiles, so copy all the profiles from MQPA…. to MQPB….

With hindsight it may have been better to

define profiles with a generic name prefix, eg MQHLQ, so you would have MQHLQ.DEFINE.**
have a queue manager option SECPFX=MQHLQ which points to these profiles
have a class SERVER profile MQ.MQHLQ and grant the queue manager userid access to it.

How do you copy the security profiles for a new queue manager?

I could find no easy way of doing this. When I worked for IBM I had some rexx code which used the IRRXUTIL to extract information from the RACF database and rebuild the RDEFINE and PERMIT statements.

You could also use the RACF Unload Database program into a file, but most people are not likely to have access to the this.

What’s the difference between RACDCERT MAP and RACMAP?

I was trying to set up digital certificate authentication into RACF and was having problems. I had used a command

RACDCERT MAP ID(ADCDC ) - 
   SDNFILTER('CN=colinpaicesECp256r1.O=cpwebuser.C=GB') - 
   WITHLABEL('colinpaicesECp256r2')

but it was hard to find out why I could not connect. I started looking into this and got confused because the MQWEB liberty trace talked about userid in realms, but I did not have a realm.

I took a couple of days to write a program to use the RACF callable service to query the userid given a DN, but it kept reporting the certificate was not found.

Eventually I found that RACF has two ways of mapping a DN string to a userid

RACDCERT MAP ID(ADCDC ) SDNFILTER(‘CN=colinpaicesECp256r1.O=cpwebuser.C=GB’) WITHLABEL(‘colinpaicesECp256r2’)
RACMAP MAP ID(ADCDC ) USERDIDFILTER( NAME(‘CN=colinpaicesECp256r1.O=cpwebuser.C=GB’)) REGISTRY(NAME(‘ADCDPL’))
WITHLABEL(‘COLIN5’)

RACDCERT MAP

This is used so that when someone logs on using a certificate, the certificate DN is looked up in the RACDCERT MAP, and if found, the matching userid is returned.

This command is not very usable.

You can map a DN string to a user.
You can list the DN string associated with a userid
You cannot query to see if a DN string exists, and which userid it is mapped to
If you try to add it, and it already exists, it just reports that it exists, and does not tell you which userid it is mapped to. So you cannot easily delete it
For an application to query the userid, you need to use the initACEE interface which is complex and requires your code to run authorised.

If the system is unable to map a certificate to a userid you get a message…

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0162)
SUBJECT(CN=colinpaicesECp256r1.O=cpwebuser.C=GB) ISSUER(CN=SSCA8.OU=CA.O=SSS.C=GB).

RACMAP MAP

This has been designed for enterprise identity propagation. You can have userid information in different realms, for example in RACF or in one oe more LDAPs.

You can map a DN string to a userid
You can list the DN strings associated with a userid
You can query a DN string and get the associated userid
You can use the r_usermap (IRRSIM00) callable service to map a DN string to a user. You need access to some RACF profiles.

They are not interchangable

You cannot define a mapping using RACDCERT MAP and use the r_usermap interface, or the other way around.

You need a SAF Angel for Liberty Web Server on z/OS

I spent a week trying to get MQWEB on z/OS to work using digital certificates using RACF as a repository, and had lots of frustrations, and some successes. I found that to get security working you need an Angel process.

Some parts are well document, some other parts are not, so I’ll try to fill some of the gaps.

How does MQWEB do security?

The Liberty server has a couple of ways of Authentication and Authorisation.

The basic repository where you define users, passwords and their role (MQWebAdmin, MQWebAdminRO or MQWebUser) within the xml configuration file. This provides the most basic levels of protection – for example the MQ administrator has to create and maintain the passwords. It is easy to set up, but not very secure.
Using the System Authorization Facility (SAF) interface. This is an interface which applications can use to get to the security back end. There is a choice of back-ends, for example RACF from IBM, and Top Secret from CA. This can be configured to be very secure, but has more administrator work to set up.

How does SAF work?

There are RACF profiles which control access to restricted facilities. If you have access to the profile you can perform the function. For example when you logon to the a server, the server thread has to run as your userid, to allow your userid access to resources. You set up a profile, and explicitly give the server access to the profile permitting the server to “run on behalf of another userid”.

There is a C function on z/OS that allows my userid to check to see if your userid is authorised to a resource. If my userid has access to the correct profile, my userid can get the information about your userid.

Some “system-like” functions assume the requester is authorised, and bypass some of the checks. For example the RACROUTE FASTAUTH check request. These have another level of control in that they need to be in a restricted, Supervisor (kernel) state, to be able to issue the requests.

What is the angel process?

Instead of giving Java programs access to this restricted Supervisor mode, there is an Angel address space which can execute the restricted requests, and the Java program can call the code running in the Angel address space. (Under the covers there is a Program Call to execute the code, in the same way that MQ and DB2 requests do.) Of course, there is a RACF profile to control which address spaces can access the Angel, and other profiles to configure what restricted functions the Angel process can execute.

If the Angel process was not running, I could not logon to MQWEB. There were messages saying my userid did not have access to MQWebAdmin, MQWebAdminRO or MQWebUser. The checking is done within the LIberty and is very simple, and restrictive – and does not work because it looks for group names MQWebAdmin etc… but MQWebAdmin is too long for a z/OS group name which can have up to 8 characters in it.

Setting up the Angel process.

The Angel process can be shared by all of the web servers on the LPAR.
In a highly available environment you need more than one Angel on an LPAR.
MQ ships Angel code, but it may be at a different level to other Liberty servers’s Angel code. You should run on the latest level of code, but I expect if they are reasonably current( within 1 year) they should all work ok.
This gives a good overview of the Angel.
This also gives a good overview and instructions
See here for configuration instructions.
I think the only Angel interface used by MQWEB is SAFCRED (SAF Credentials) and PRODMGR, but it does no harm having access to services you do not use.
You cannot stop an Angel process if it has servers “connected” to it.
If you cancel the Angel, the Web Server stops working, it may get CEE3250C The system or user abend S0D6 R=00000027 was issued in the message.log file, and abend.
For availability you may want two queue managers on an LPAR, so you need two Angel process.
There is no configuration you do to an Angel, so the only reason why you may want to shut down the Angel during normal running is to apply fixes.
You need to ensure the Angel process is started before the Web Server is started, as the Web Server only connects at start up, see below.
Each Angel running in an LPAR needs a unique name. You can have a default, unnamed Angel, or give your Angel a name.

I called my angel process ANGEL. From WAS and z/OS Connect they talk about their Angels with names like BBZANGL, BAQZANGL, which are not very memorable, hard to remember, and easy to mistype.

You start an angel process (once the JCL has been configured) using a command like

S ANGEL
S ANGEL,NAME=ANGEL1
S ANGEL.ANGEL1,NAME=ANGEL1
S ANGEL.ANGEL2,NAME=ANGEL2

The first command starts the default, unnamed, Angel.

Using the S ANGEL.name command is useful if you have more than one angel as it means you can use the STOP name command to that particular instance.

Configure your web server to end if the angel is not available

You can configure your Liberty server to end if the Angel process is not running. Without it, the Web Server would start, and requests to use it would fail, because the unauthorised interface would be used. Using this option means you find out Sunday night and not when your business starts at 0900 on Monday morning!

You configure it by editing the server jvm.options file by adding

-Dcom.ibm.ws.zos.core.angelRequired=true

If the Angel is not available, this statement prevents the Web Server from starting

You can specify the angel name using

-Dcom.ibm.ws.zos.core.angelName=MYANGEL

If you are using the default angle name, comment this out

# -Dcom.ibm.ws.zos.core.angelName=

When your Web Server starts you should get messages in the message.log like

CWWKB0103I: Authorized service group SAFCRED is available.

Tracing violations.

You can start an Angel using

S ANGEL,SAFLOG=YES

I gave my MWEB userid no access to SAFCRED, and restarted MQWEB. With SAFLOG=YES, I got the following on the joblog

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
BBG.AUTHMOD.BBGZSAFM.SAFCRED CL(SERVER )
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

and in the MQWEB directory/logs/message.log file

CWWKB0104I: Authorized service group SAFCRED is not available.

With SAFLOG=NO, there was no message on the joblog, but the same message in MQWEB directory/log/message.log file

CWWKB0104I: Authorized service group SAFCRED is not available.

From this we can see that you should always specify SAFLOG=YES.

Angel commands

You can use

F ANGEL,DISPLAY,SERVERS

This gives a message like

CWWKB0052I ACTIVE SERVER ASID 3f JOBNAME CSQ9WEB

You can use

F ANGEL,TRACE=Y

This writes a trace to //STDOUT in the Angel job.

The output is like

Trace: 2020/07/25 15:38:07.877974 t=8D5E88 key=S2 (04002008)
Description: write_to_operator_location, entry
message_p: CWWKB0057I WEBSPHERE FOR Z/OS ANGEL PROCESS ENDED NORMALLY

Which is not very helpful, as it traces what the Angel process is doing – rather than the Web Servers using it.

How to use RACF callable services from C

Trying to use the RACF callable services was like trying to find treasure with an incomplete map. I found it hard to create a C program to query a repository for ID information. This post is mainly about using the RACF callable services.

I was trying to understand the mapping of a digital certificate to a z/OS userid, but with little success. I found a RACF callable service which appeared to do what I wanted – but it did not give the answers – because, like many treasure maps, I was looking in the wrong place.

RACF has two repositories for mapping identities to userid.

RACDCERT MAP which was the original way of mapping names. As far as I can tell, the only way of getting the certificate to userid mapping programmatically, is to use the certificate to logon, and then find the userid! This is used by Liberty Web Server.
RACMAP MAP which is part of Enterprise wide identification. It maps identity strings, as you may get from LDAP, to a userid. You can use the r_usermap callable service to get this information.

It took me some time to realise that these are different entities, and explains why there was no documentation on getting Liberty to work with RACMAP to handle certificates. I found out RACMAP does not map certificate, after I got my program working.

The r_usermap service documentation is accurate – but incomplete, so I’ll document some of the things I learned in getting this to work.

The callable service to extract the userid from identity information is documented here. In essence you call the assembler routine r_usermap or IRRSIM00.

Building it

When you compile it, you need to provide the stub IRRSIM00 at bind time. I used JCL

//S1 JCLLIB ORDER=CBC.SCCNPRC 
//DOCLG EXEC PROC=EDCCBG,INFILE='ADCD.C.SOURCE(C)', 
// CPARM='OPTF(DD:COPTS)' 
//COMPILE.COPTS DD * 
LIST,SSCOMM,SOURCE,LANGLVL(EXTENDED) 
TEST 
/* 
//COMPILE.SYSIN DD * 
  Source program goes here
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB 
//BIND.SYSIN DD * 
INCLUDE CSS(IRRSIM00) 
/*

You need code like

#pragma linkage(IRRSIM00, OS) 
int main(){...
...
char * workarea ; 
workarea = (char *) malloc(1024)   ; 
long ALET1= 0; 
...
long SAF_RC,RACF_RC,RACF_RS; 
...
rc=  IRRSIM00(workarea, // WORKAREA 
             &ALET1  , // ALET 
             &SAF_RC, // SAF RC 
...

Some fields are in UTF-8.

To covert from EBCDIC to UTF-8, (it looks like ASCII ) I used

cd = iconv_open("UTF-8", "IBM-1047"); 
...
struct 
{ 
  short length ; // length of string following or 0 if ommitted 
  char value[248]; 
} DN; 
char * sDN= "CN=COLIN.C=GB"; 
size_t llinput= strlen(sDN); 
size_t lloutput= sizeof(DN.value); 
char * pOutValue= &DN.value[0]; 
rc = iconv(cd,        // from  iconv_open
           &sDN,      // input string
           &llinput,  // length of input 
           &pOutValue,// output 
           &lloutput);// length of output 
if (rc == -1) // problem
{ 
  perror("iconv"); 
  exit(99); 
} 
DN.length  =sizeof(DN.value) - ll2; // calculate true length

What access do I need?

You need

permit IRR.RUSERMAP class(FACILITY) access(READ) ID(....)
SETROPTS RACLIST(facility ) REFRESH

Output

Once I had got the program to compile and bind, and got the authorisation it worked great.

It only works with the RACFMAP … command, not the RACFDCERT command, obvious now I know! To get the information from the RACDCERT MAP, you need to use initACEE.

What no one tells you about setting up your RACF groups – and how to do it for MQ.

Introduction

It is bit like saying programming has the following commands, load, store, branch; but fails to tell you that you can do wonderful things like draw Mandelbrot pictures using these instructions.

You need to plan your group structure before you try to implement security, as it is hard to change once it is in place.

The big picture

You can set up a hierarchy of groups so the site RACF person can set up a group called MQ, and give the MQ team manager authority to this group.

The manager can

define groups within it
connect users to the group
give other people authority to manage the group.

We can set up the following group structure

MQM
- MQOPS – for the MQ operators
  - MQOPSR for operators who are allow to issue only Read (display) commands
  - MQOPSW for operators who can issue all command, display and update
- MQADMS – for the MQ administrators
  - MQADMR – for MQ administrators who can only use display commands
  - MQADMW – for MQ administrators who can use all commands
- MQWEB….

You should place an operator’s userid in only one group MQOPSR or MQOPSW as these are used to control access. MQM, MQOPS, MQADMS, MQWEB are just used for administration.

You permit groups MQOPSR and MQOPSW to issue a display command, but only permit group MQOPSW to issue the SET command.

Setting up groups to make it easy to administer

A group needs an owner which administers the group. The owner can be a userid or a group.

A group has been set up called MQM, and my manager has been made the owner of it.

My manager has connected my userid PAICE to the MQM group with group special.

CONNECT PAICE GROUP(MQM) SPECIAL

I can define a new group MQOPS for example

ADDGROUP MQOPS SUPGROUP(MQM) OWNER(MQM) DATA(‘MQ operators’)

The SUPGROUP says it is part of the hierarchy under MQM. I can create the group under MQM because I am authorised, If I try to create a group with SUPGROUP(SYS1) this will fail because I am not authorised to SYS1.

The OWNER(MQM) says people in the group MQM with group special can administer this new group.

Because my userid (PAICE) has group special for MQM, I can now connect users to the new group, for example

CONNECT ADCDB GROUP(MQMD ) AUTHORITY(USE ).

I can create another group under MQMD called MQMX, and connect a userid to it.

ADDGROUP MQMX SUPGROUP(MQMD) OWNER(MQMD) DATA(‘MQ Bottom group’)
CONNECT ADCDE GROUP(MQMX ) AUTHORITY(USE )

My userid PAICE can administer this because of the OWNER() inheritance up to GROUP(MQM)

If I list the groups I get

LISTGRP MQM 
INFORMATION FOR GROUP MQM 
    SUPERIOR GROUP=SYS1 OWNER=IBMUSER 
    SUBGROUP(S)= MQM2 MQMD  
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS= 
       PAICE    JOIN        000000              NONE 
         CONNECT ATTRIBUTES=SPECIAL 

LISTGRP MQMD 
INFORMATION FOR GROUP MQMD 
    SUPERIOR GROUP=MQM OWNER=MQM 
       SUBGROUP(S)= MQMX 
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS=  
       ADCDB USE 000000 NONE CONNECT ATTRIBUTES=NONE
 
LISTGRP MQMX 
    INFORMATION FOR GROUP MQMX 
    SUPERIOR GROUP=MQMD OWNER=MQMD 
    NO SUBGROUPS 
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS= 
       ADCDE     USE        000000              NONE 
         CONNECT ATTRIBUTES=NONE

All SUPGROUP() does is to define the hierarchy as we can see from the LISTGRP. We can display the groups and draw up a picture of the hierarchy. You can use the LISTGRP command repeatedly, or use the DSMON program(EXEC PGM=ICHDSM00) and use option
USEROPT RACGRP to get a picture like

 LEVEL GROUP 
1 SYS1 (IBMUSER ) 
2 | MQM (IBMUSER ) 
3 | | MQMD 
4 | | | MQMX 
3 | | MQM2 (IBMUSER )

Using OWNER(group) instead of OWNER(userid)

If you have OWNER(groupname) it is easy to administer the groups. When someone joins or leaves the department, you add or remove the userid from groupname. One change.
If you have OWNER(userid), then you have to explicitly connect the userid to each group with group special. When there is a new person you have to add the userid to each group individually. When someone leaves the team you have to remove the persons userid from all of the groups. This could be a lot of work.

Delegation.

You could define an operator MQOP1 and give the userid group-special for group MQOPS. This userid (MQOP1) can be used to add or remove userids in the MQOPSR and MQOPSW groups.

Looking at the MQOPS groups we could have groups and connected userids

MQM with MQ security userids PAICE, BOB having group-special
- MQOPS with the operations manager and deputy MQOP1, MQOP2 having group special
  - MQOPSR with STUDENT1, STUDENT2 who are only allowed to issue display commands
  - MQOPSW with PAICE, TONYH, CHARLIE
- MQADMS….

and similarly for the MQ administration eam.

Userid PAICE can connect userids to all groups. MOP1 can only connect userids to the MQOPSR and MQOPSW, and not connect to the MQ ADMIN groups.

You use groups MQOPSR and MQOPSW for accessing resources. Groups MQM and MQOPS have no authority to access a resource, they are just to make the administration easier.

You may also want to consider having a group for application development. The group called PAYRDEVT is under MQM, is owned by the manager of the payroll development team.

When the annual userid validation check is done, the development manager does the checks, and tells the security department it has been done.

Permissions

There is no inheritance of permissions. If a userid needs functions available to groups MQMD and MQMX, the user needs to be connected to both groups.

You only connect userids to groups, you cannot have groups within groups. There may be many groups of userids which are allowed to issue an MQ display command, but only one group who can issue the SET command.

Suggested MQ groups

You need to consider

production and test environments
resources shared by queue managers, queue managers with the same configurations in a sysplex which can share definitions
queue managers as part of a Queue Sharing Group
queue manages that need isolation and so may have common operations groups, but different administration and programming groups.

You might define

Group MQPA for the queue manager super group. (MQ, Production system, A)
Groups for MQWEB. The Web server roles are described here.
Groups for controlling MQ, operations and administrations, read only or update
Groups for who can connect via batch, CICS etc
Groups for application usage, who can use which queues

Groups for MQWEB

For MQWEB the MQ documentation describes 4 roles: MQWebAdmin, MQWebAdminRO, MQWebUser, MFTWebAdmin; and there is console and REST access.

Each role should have its own group. The requests from “Admin” and “Read Only” run with the userid of the MQWEB started task. The request from “User” run with the signed on user’s authority.

You might set up groups

MQPAWCO MQPA – MQWebAdminRO Console Read Only.
MQPAWCU MQPA – MQWebUser Console User only. The request operates under the signed on userid authority.
MQPAWCA MQPA – MQWebAdmin Console Admin.
MQPAWRO MQPA – MQWebAdminRO REST Read Only.
MQPAWRU MQPA – MQWebUser REST User only. The request operates under the signed on userid authority.
MQPAWRA MQPA – MQWebAdmin REST Admin Only.
MQPAWFA MQPA – MFTWebAdmin MFT REST Admin.
MQPAWFO MQPA- MFTWebAdmin MFT REST Read Only.

I would expect most people to be in

MQPAWCU MQPA – MQWebUser Console User only. The request operates under the signed on userid authority.
MQPAWRU MQPA – MQWebUser REST User only. The request operates under the signed on userid authority.

so you can control who does what, and get reports on any violations etc. If people use the MQWEB ADMIN you do not know who tried to issue a command.

Groups for operations

The operations team may be managing multiple queue managers, so you may need groups

PMQOPS for Production
- PMQOPSR
- PMQOPSW
TMQOPS for Test
- PMQOPSR
- PMQOPSW

If some operators are permitted to manage only a subset of the queue managers you will need a group structure that can handle this, so have a special group XMQOPS for this.

XMQOPS for the special queue manager
- XMQOPSR
- XMQOPSW

Groups for administration.

This will be similar to operations.

Groups for end users.

This is for people running work using MQ.

Usually there are checks to make sure a userid can connect to the queue manager, using the MQCONN resource. Some customers have a loose security set up, and rely on the CICS to check to see if the userid is allowed to use a CICS transaction, rather than if the userid is allowed to access a queue.

Linux where’s my window? Use the keyboard to bring favourite window to the front

Using z/OS from my Linux laptop, I use X3270 to define the 3270 screens. I use multiple screens ( master console, ISPF with main userid, ISPF with test userid). It can be hard to find the right session. I also try to use the keyboard rather than then mouse – how do I do it?
There is a good program wmctrl which can help you manage your all you windows, but I use it mainly for x3270 sessions.

For example wmctrl -a colin@ brings a window with “colin@” in the title, and gives it focus. You then associate this with a keyboard action and the problem is solved.

wmctrl can do many things including

switch desktops
list desktops and windows managed by wmctrl
hide windows or make them full screen

To associate a keyboard action and a command,

display the settings (for example from the down arrow at top of the screen)
Keyboard – this lists your keyboard shortcuts.
go to the bottom of the list and click on +
enter
- name:my name
- command:wmctrl -a colin@
and click on Set Shortcut…
a small window pops up – type the key combination for example Ctrl 1

I have Ctrl+1 for the master console, Ctrl+2 for my main ISPF session, and Ctrl+3 for my “end user” ISPF session, and Ctrl+4 for Firefox

You can create an x3270 session using a command like

x3270 -model 5 colin@localhost:3270

which creates a session to port 3270 on my localhost, and puts -model 5 colin@localhost:3270 in the title bar. You could create another session x3270 -model 5 super@localhost:3270 so so use “super@” to identify it in the wmctrl command.

x3270 – where’s the money key?

On z/OS the default escape character for OMVS is the cent symbol ¢ (Unicode 00a2) which I do not have on my keyboard.

I set up in ~/.x3270pro

! Use the 'mine' keymap, defined below
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
   <Key>Escape: Clear()\n\
   <Key>End: FieldEnd()\n\
   Ctrl<Key>Delete: EraseEOF()\n\
   Ctrl<Key>Right: NextWord()\n\
   Ctrl<Key>Left: PreviousWord()\n\
   Ctrl<Key>Up: Home()\n\
   <Key>Control_L: Reset()\n\
   <Key>Control_R: Reset()\n\
   <Key>Prior: PF(7)\n\
   <Key>Next: PF(8)\n\
   <Btn3Down>: PA(1)\n\
   Ctrl<Key>backslash: Key(U+00a2)\n\
   Alt<KeyPress>4: Key(U+00a2)

So now I can use Alt 4 or Ctrl \ to enter the value.

Making x3270 green screens blue or red, or yellow with green bits.

I have several ISPF screens on z/OS, and I found it confusing knowing which one I was on. I solved it by making the screen with the all powerful userid displayed in red, and the “normal” screen in multi colours, and the screen with no authority in green.

With x3270 you can use Options -> Colour Scheme -> Default 3279 |Green, to use a predefined colour scheme.

I created my own colour scheme by editing ~./x3270pro and adding

! define the list of colour schemes
x3270.schemeList: Default 3279: default\n\
    Bright: bright\n\
    Reverse: reverse\n\
    Green Screen: greenScreen\n\
    Colin: colin 
! now define my list
x3270.colorScheme.colin: \
    black deepSkyBlue pink red \
    green turquoise yellow white \
    black blue3 orange purple \
    paleGreen paleTurquoise2 grey white \
    white black dimGray \
    4 2 1 15

This gave a good description, and a good list of options is here.

You can now use the command to start a session with this scheme.

x3270 -model 5 -scheme colin tso@localhost:3270

Playing twister with Liberty and falling over

Twister is a game where you have you put your left foot here, your left hand there, your right foot here, and in trying to put your right hand over there you fall over. This is how I felt when I was trying to understand the SSL definitions in MQWEB. In the end I printed off the definitions and used coloured pens to mark the relevant data.

Let’s start with the easy bit.

When mqweb starts, it reads configuration information from a file server.xml
This includes two files the “IBM” stuff in /usr/lpp/mqm/V9R1M1/web/mq/etc/mqweb.xml and the user stuff in /u/mqweb/servers/mqweb/mqwebusr.xml .
SSL parameters are defined with and <ssl… id=”thisSSLConfig” keyStoreRef=”defaultKeyStore” .. /> tag. This points to the keystore to use.
The keystore has <keyStore id=”defaultKeyStore” ….>. There is a simple link from SSL to the keystore. You could have multiple keystores, if so you just change the keyStoreRef= to point to a different one.
You can have more than one <ssl…/> definition. You might have one, and there is one in the “IBM stuff”, so you need <sslDefault sslRef=”thisSSLConfig”/> to point to the ssl statement to use.

That should all be clear, and make sense. A bit like saying you have a right foot, a left foot and two hands.

The zos_saf_registry.xml used when you want to use the SAF interface on z/OS has some SSL definitions. I was trying to understand them. This one here(put a finger on it) points to that one, (put a finger on it), which points to this other one (put a finger on it), which has an end-comment. Whoops that didn’t work. As I said a bit like playing Twister.

<sslDefault sslRef=”mqDefaultSSLConfig”/> in the user mqwebuser.xml points to content in the “IBM stuff”. By the various levels of indirection this points to <keyStore id=”defaultKeyStore” location=”key.jks” type=”JKS” password=”password”/> . This keystore has a self signed certificate provided by IBM. If you find your browser complains about using a self signed certificate, this may well be the cause.

In the zos_saf_registry.xml are commented statements

<keyStore id=”defaultKeyStore” location=”safkeyring://userId/keyring” …/>
<ssl id=”thisSSLConfig” keyStoreRef=”defaultKeyStore” …/>
<sslDefault sslRef=”thisSSLConfig”/>

To me these have been defined upside down, sslDefault should come first.

As these are after the sslDefault sslRef=”mqDefaultSSLConfig statment, if you uncomment them, they will be picked up and the “IBM stuff” will not be processed.

You can uncomment these statements and use them to add your definitions.

My definitions are

<sslDefault sslRef="defaultSSLConfig"/> 
<ssl id="defaultSSLConfig" keyStoreRef="racfKeyStore" 
   sslProtocol="TLSv1.2" 
   clientAuthenticationSupported="true" 
   clientAuthentication="true" 
   serverKeyAlias="LABELMQWEBHSCEKE"/> 

<keyStore filebased="false" id="racfKeyStore" 
   location="safkeyring://START1/MQRING" 
   password="password" 
   readOnly="true" 
   type="JCERACFKS"/> 

<webAppSecurity allowFailOverToBasicAuth="false"/>