Customising for MQWEB Liberty on z/OS, things the documentation does not tell you about

This post covers the customising you need to consider enterprise use of the Liberty MQWEB server.  It covers

  • Setup the USS path and defining an alias for the mq executable’s  directory
  • Do you have common configuration across mqwebuser.xml files?
  • Decide if you want to use setmqweb.
  • Setting up the server’s certificate and keyring
  • Setting up the trust store
  • Setting up the Angel process(es)
  • Reserving the TCP/IP Port number
  • Customising the jvm.option
    • To prevent the web server coming up if the Angel process is missing
    • Setting the time zone
  • Customising the mqwebuser.xml
    • SAF definitions
    • Setting the log sizes so the logs can be viewed
  • Letting requests in from outside of the LPAR
  • dspmqweb/setmqweb – which instance to use?
  • Selecting which IP stack to use
  • Customising ISPF option 3.17 – Unix Directory List

Setup the USS path and defining an alias for the mq executable’s directory

To be able to use the dspmweb and setmqweb commands you need to point to the command location.

You can add to the user’s .profile file, or the /etc/profile the statement

export PATH=/usr/lpp/mqm/V9R1M1/web/bin:$PATH

If you have multiple releases of MQ in your environment you could set up shell commands like

/usr/lpp/mqm/V9R1M3/web/bin/dspmqweb "$@"

But this causes extra work when you need to migrate to the new release.  It might be better to set up an alias

ls -s  /usr/lpp/mqm/V9R1M3/web/bin /v913
ls -s  /usr/lpp/mqm/V9R1M3/web/bin /mqcur

so you just need to type /v913/dspmweb or /mqcur/setmqweb

As part of the migration to a newer release you just change the alias.

Do you have common configuration across mqwebuser.xml files?

If you have multiple mqweb instances, either because you have multiple LPARs in a sysplex, or you have to support different release of MQ concurrently, you may want to put common configuration in an include file. For example created a file common.xml to hold the configuration and put

<include location=”common.xml” optional=”false”/>

in the mqwebuser.xml file.

Decide if you want to use setmqweb.

You can update your *.xml configuration files, or use setmqweb to update mqwebuser.xml for you.

Some organisations do not allow manual changes to configuration. You have to change a configuration file, have it reviewed, and use automation to deploy it.

For test systems it may be ok to use the setmqweb command and change things dynamically.

If you make a change using setmqweb, it updates the mqwebuser.xml file, by adding/changing a <variable name=”…” value=”..”/>  statement.

If you are using SAF authentication and certificate authentication

You will need keyring with the certificate to identify the server (the key store).  You will need a keyring to identify the certificates you trust (the trust store).  You could use the same keyring for both – but this is not good practice.

The server’s certificate and key store keyring

You need to decide if the MQWEB server uses the same certificate as CICS, WAS and z/OS Connect etc. on the same LPAR.  You could have a common certificate to simplify administration. The certificate needs a Subject Alternative Name, to identify the machine the certificate came from. This can be the DNS name or the dotted address ( depending on your set up.  It might be easier to define both. Note the RACF command


accepts multiple entries, but only uses the last one. The above command gave produced a certificate with

Subject's AltNames: 

This means you many only be able to use the certificate only on the LPAR that has been defined, (if you move the server to a different LPA, it will have a different IP address, and your clients will complain – see below).   You may be able to something clever things with VIPA (Virtual IP addressing) where your Sysplex has one IP address and this maps to different IP addresses on each LPAR.

If you have the wrong IP or Domain then the browser gets  a message like “Your connection is not private. Attackers may try to steal your information from  NET:ERR_COMMON_NAME_INVALID”

The trust store keyring.

The trust store keyring has the certificates to authenticate what has been sent from the client.  For example, a copy of any self signed certificate, or the Certificate Authorities of the Web Browser’s certificate.

This keyring could be sysplex wide, and shared by CICS, WAS, Z/OS connect etc – assuming they have the same people connecting to them.

The certificates may have been configured with owner CERTAUTH rather than an userid.

My definitions

<sslDefault sslRef="defaultSSLConfig"/> 
<ssl id="defaultSSLConfig" 

<keyStore filebased="false" id="racfKeyStore" 

 <keyStore filebased="false" id="racfTrustStore" 

<webAppSecurity allowFailOverToBasicAuth="false"/>
  • The sslDefault  points to the ssl with the same ID
  • The ssl points to
    • the key store with the servers certificate with the id racfKeyStore
    • the trust store to validate connecting clients, with the id racfTrustStore

Create an angel

You need an Angel process to handle the SAF (RACF) security requests – the MQ documentation tells you this.

Typically the Angel started task is started at IPL, and shut down at system shut down.
All instances of Liberty Web Server running on an LPAR can all use the same Angel, for example the z/OSMF angel IZUANG1.

You cannot shut down the Angle process if it is in use, but if you cancel it, the servers using it will stop working (hang) and may abend.

You may want to consider more than one Angel process, and not share it.

When the Angel process has started, it uses no CPU, as the Web Servers execute code within the  Angel address space, on the Web Server’s threads – just like MQ, DB2 etc.

Customise  jvm.options

Stop if there is no Angel  process

If the Angel process is not running at Liberty startup,  then the Web Server may continue to come up.  People will not be authorised to access it, but the Web Server will be running.   This is pretty useless.

You can specify an option so the liberty server (MQWEB) does not start if the Angel task is not running.

I use

If the angel process is not available then the MQWEB stops when it detects the angel is not available.

If you are using a names Angel, uncomment this and specify the Angel name.

If you are using the unnamed Angel, leave this commented.

Set the time zone

The time zone is picke up from TZ in /etc/profile, but you can override it by specifying


This sets the time-zone of the messages in the message.log and trace.log files.

Reserve the TCP/IP port number

It is a good idea to talk to the networking team and get them to update the TCP/IP configuration for example

    21 TCP OMVS ; FTP Server 
    22 TCP SSHD* ; port for sshd daemonrver 
    23 TCP TN3270 ; Telnet 3270 Server 
    1414 TCP CSQ9CHIN ; CSQ9 MQ TCP Listener  
    9443 TCP MQWEB ; Colin Paice MQWEB


Customise mqwebuser.xml

Message log and trace file settings

If the trace or message files are too big, you cannot view them. You have to use edit to look at them, but if the file is too large, browse is substituted and browse does not do code page conversion, so you are looked at raw ascii characters in an EBCDIC browser.

<variable name=”maxTraceFileSize” value=”20″/>
<variable name=”maxTraceFiles” value=”20″/>
<variable name=”maxMsgTraceFileSize” value=”20″/>
<variable name=”maxMsgTraceFiles” value=”20″/>

The file size values are in MB.

You should consider keeping you messages.log files for a week or so, so make the number of files large enough.

SAF – Access to RACF

If you are using SAF (RACF or other z/OS security manager) to manage access and authorisation you will have a default entry like

Example SAF Registry 
<safAuthorization racRouteLog="NONE" id="saf" /> 

<safRegistry id="saf" /> 
<safCredentials unauthenticatedUser="WSGUEST" profilePrefix="MQWEB" 
suppressAuthFailureMessages="false" /> 

I use <safAuthorization racRouteLog=”ASIS”… to get RACF violation messages on the joblog during set up.  See here.

<safRegistry suppressAuthFailureMessages=”false”…  prints out violation messages.  See here.

Let request in from outside z/OS

For this to work you have to edit the mqwebuser.xml file and uncomment

<variable name="httpHost" value="*"/> 

By default it only allows request from the same z/OS system – so not allowing browsers access.

dspmqweb/setmqweb – which instance to use?

This page  says you must use

export WLP_USER_DIR=WLP_user_directory

This is fine when you have one mqweb instance on one LPAR.  You might want a shell program to set this every time.  For example,  the program

export WLP_USER_DIR=/u/mqmweb/MQPA
/usr/lpp/mqm/V9R1M1/web/bin/dspmqweb "$@"

Then you can use /usr/lpp/mqm/V9R1M1/web/bin/dspmqweb as before.

If you have multiple releases of MQ in your environment you might want to point to the command in the script, so might have

export WLP_USER_DIR=/u/mqmweb/MQPA
/usr/lpp/mqm/V9R1M1/web/bin/dspmqweb "$@"

Though it might be better to have a shell script mq911 with an optional queue manager parameter

Selecting which IP stacks to use.

There is an article from IBM, which gives two ways of configuring it.  Changing the httpEndpoint, or specifying an environment variable

Customise ISPF z/OS UNIX Directory List

In the MWEB directory are message logs and trace logs.  When the file fills up, it renames the old file to include the date and time, for example messages_20.07.29_16.49.29.0.log , and creates a new message.log or trace.log

If you are using ISPF 3.17 (z/OS UNIX Directory List) to use the files, it only displays the first 15 characters of the file name, so you get lots of files with a name like “messages_20.07.” where 20 is the year, and 07 is the month.

The default layout for the z/OS UNIX Directory List  displays by default some unhelpful fields.   You can arrange the fields, (but not make the filename field wider).
If you go to the OPTIONS on the top line, and select “2. Directory List Column Arrangement… ” you can change what fields are displayed, and the order.  I set the widths of all fields to 0, except for

  • Type 04
  • Modified 19 (if you specify a smaller value you only get the YYYY-MM…  not the time)
  • Size 10

The documentation says

  • Modified The date and time the file was last changed.
  • Changed The date and time the status of the file was last changed.

I do not know the difference between these two.

Controlling what is displayed

In the directory list you can use sort commands

  • sort file A
  • sort mod D

Looking at a log or trace file

If you sort by Modified A the newer files will be at the top, so you can look at the “modified” column to look for the time the file was created, and so get the order of the files.

You can use the line command / to display the options.

You can use e to edit, or V to use edit in browse mode.

Browse displays a mess because it does not do conversion



Liberty on z/OS: Mapping an incoming certificate to a z/OS userid for client certificate authentication – and don’t forget the cookies!

I thought I understood how this worked, I found I didn’t, then had a few days hunting around for the problem

The basics

You can use a digital certificate from a web browser ( curl, or other tools) to authenticate to z/OS.  You need to map the certificate to a userid.

A certificate coming in can have a Distinguished Name like CN=adcdd.O=cpwebuser.C=GB  (Note the ‘.’not ‘,’ between elements).

Your userid needs to have SPECIAL define to be able to use the RACDCERT command (SPECIAL, not just GROUP-SPECIAL).

You will need a definition like (see here for the command)

    SDNFILTER('CN=adcdd.O=cpwebuser.C=GB') - 

or a general definition for those certificate with  O=cpwebuser.C=GB, ignoring the CN part

   SDNFILTER('O=cpwebuser.C=GB') - 

or using the Issuing Distinguished Name (the Certificate Authority)


Using a generic


does not work.

If you attempt to use a certificate which is not mapped you get


It is worth defining these using JCL, because if you try to add it, and it already exists then you get a message saying it exists already.  If you know the userid, you can list the maps associated with it.   If you do not know the userid, there is no practical way of finding out – you have to logon with the certificate, and display the userid from the web browser, or extract the list of all users, and use LISTMAP on all of them.

Once you have set up the userid, you can connect them to the group to give them access to the EJBROLE profiles.  For example use group names

  • MQPAWCO MQPAMQWebAdminRO Console Read Only.
  • MQPAWCU MQPAMQWebUser  Console User only.  The request operates under the signed on userid authority.
  • MQPAWCA MQPAMQWebAdmin Console Admin.

for queue manager MQPA, Web  Console (rather than REST) and the access.

You may want to set up  userids solely for client authentication.  If the userid has NOPASSWORD, it cannot be used to logon with userid and password, and of course the lack of password means the password will not expire.

Having a set of userids just for certificate access makes it easier to manage the RACDCERT MAPping.    You have a job with


and search the output for the certificate of interest.

It gets more complicated…

Often the user’s certificate is in the form CN=Colin Paice,o=SSS,C=GB so if you want to allow all people in the MQADMIN team access, you will need to to specify them individually.  It would be easier if DN had CN=Colin Paice,OU=MQADMIN,o=SSS,C=GB, then you can filter on the OU=MQADMIN.   These could map to a userid MQADM1.

It gets more complication if someone can work with MQ, and CICS or z/OS Connect, and you have to decide a userid – MQADM1 or CICSADM1?

Setting up a one to one mapping may be the best solution, so CN=Colin Paice,o=SSS,C=GB maps to CPAICE (or GB070594).   This userid is then added to the appropriate RACF groups to give access to the EJBROLEs, to give access to the servers.

How do I tell what is being used?

I could not get Liberty to record an audit record for the logon/matching.   I tried altering the userid to have UADIT – but it did not work either.

If you have audit defined on the class EJBROLE profile, you will get a audit record in SMF.   This has many fields including

  • Date
  • Time
  • SUCCESS – or INSACC (INSufficient Access)
  • ADCDC – userid being used
  • READ – Requested access
  • READ – permitted access
  • EJBROLE – the class
  • – the profile
  • CN=adcdd.O=cpwebuser.C=GB – the Distinguished Name of the certificate
  • CN=SSCA8.OU=CA.O=SSS.C=GB – the Issuers (Certificate Authority) of the certificate

From this you can see the userid being used ACDC, and the certificate DN CN=adcdd.O=cpwebuser.C=GB.

And to make it more complicated

I deleted the RACDCERT MAP entry, but the web browser continued to work with the user.  I had a cup of tea and a cookie, and the web browser stopped working.   Was problem this connected to a cup of tea and a cookie?

Setting up the initial handshake is expensive.  The system has to do a logon with the certificate to get the userid from the RACDCERT mapping.  It then checks the userid has access to the SERVER profile, then it checks to see if it is MQWebAdmin, MQWebAdminRO, or  MQWebUser.

Once it has done this it it takes the userid and information, encrypts it, and creates the LTPA cookie.   This is sent down to the web browser.

The next time the web browser sends some data, it also sends the cookie. The MQWEB server decrypts the cookie, checks the time stamp to make sure the information is current, and if so, uses it.  The timeline I had was

  • create the RACDCERT mapping from certificate DN to userid
  • use browser to logon to mqweb, using the certificate with the DN
  • it works, mqweb sends down the cooke
  • delete the RACDCERT mapping for the DN
  • restart the browser, logon to mqweb, using the certificate with the DN.  The cookie is passed up – the logon works
  • clear the browser’s cookies – and retry the logon.  It fails as expected.

So ensure the browser cookie is cleared if you change the mapping or ejbrole access for the user.

Tracing Liberty logon on z/OS – is difficult

I had a few problems logging on to the MQWEB server using certificates, and found there was no documentation to help resolve problems.  The debug information you can get is often not very helpful!

As an extra twist, a userid having no access and getting a “not authorised to the resource” is OK.  For example my userid may have access to MQWebAdmin, but not to MQWebAdminRO – it may be wrong to have access to both, so you will get at least one “not authorised to the resource” violation.

I looked at

  • MQWEB traces not enough information provided
  • RACF traces – looks wrong to me
  • RACF audit data in SMF – this is all you need

The only way of getting out the data, is to enable RACF audit for the profile, and set an option in the mqwebuser.xml file.

To make it even more difficult to resolve problems. When a request arrives at the web server, it encrypts the information, and sends down an LTPA token.    The next request from the browser sends this token, and bypasses some of the initial checks, so you will not see trace entries.  After the LTPA token has expired, the next request will do the full logon again.
To prevent this from happening, clear your browser history and cache before retesting.

MQWEB trace provides information – none of it usable.

I used the trace string


I also included :UserRegistry=all:Credentials=all which gave more information, not all of it useful.

This provides information like

Description: Entry: checkAuthorizationFast 
serviceResults: 00000050868103e7 
suppresMessages: 0 
logOption: 3 
raco_cb: 0000005082d08290 
acee: 0000000000000000 
accessLevel: 2 
applName: MQWEB 
className: EJBROLE 
   returnCode: 0 
   safReturnCode: 0 
   racfReturnCode: 0 
   racfReasonCode: 0

But does not tell you which userid the request was being made for!

Sometime it gives you full control blocks, other times truncated like so you do not know if this is for Admin, AdminRO, or User.

MQWEB safAuthorization racRouteLog

I enabled RACF AUDIT for the and

In the mqwebuser.xml you can have to display audit messages about EJBROLE

<safAuthorization racRouteLog=”NONE” id=”saf”
reportAuthorizationCheckDetails=”false” />

See here – which says

Specifies the types of access attempts to log.

  • ASIS Records the event in the manner specified in the profile that protects the resource, or by other methods such as the SETROPTS option.
  • NOFAIL If the authorization check fails, the attempt is not recorded. If the authorization check succeeds, the attempt is recorded as in ASIS.
  • NONE The attempt is not recorded.
  • NOSTAT The attempt is not recorded. No logging occurs and no resource statistics are updated.

With AUDIT enabled, and with racRouteLog=”ASIS”  I got the following “failures” (every 10 seconds) due to the web server doing auto-refresh. The checks to MQWebUser worked, and were not recorded in the joblog.


When I changed it to racRouteLog=”NONE” the messages were not produced on the joblog.   There were still records produced in the SMF audit data with the profiles having AUDIT ALL(READ) specified.

I think you should usually run with racRouteLog=”NONE” , and change it to racRouteLog=”ASIS”  when you have a problem – but be careful not to generate a flood of messages.

To display SAF messages about other violations use

<safCredentials unauthenticatedUser=”WSGUEST” profilePrefix=”MQWEB”
suppressAuthFailureMessages=”false” mapDistributedIdentities=”false”/>

RACF Trace

This gives data as to what was accessed, but reports the userid of the web server, not the userid being checked – so not very useful.

I used the command

  • traced to GTF with USRP(F44)
  • formatted it with the IPCS command GTF USR(ALL)

This had data like

  • Trace Type: RACFPOST – this is the “AFTER” request
  • Service number: 00000002 – this is RACROUTE 2, verify
  • RACF Return code: 00000008
  • RACF Reason code: 00000004
  • – this profile
  • EJBROLE – this class
  • MQWEB – this application
  • ACEE ( userid block) with userid=START1, Group=SYS1, Jobname=CSQ9WEB.   This had a userid of START1 (from the job),  but the userid being tested was for ADCDC which was not in the control blocks – so no good for telling you which userid had accces or not.

So all we can tell is, that for the profile someone got a ‘no access’ return code.

Turn off the RACF trace using


RACF Auditing – this worked and gave me most of the information

I turned on RACF auditing using


This writes a record to SMF for ALL(failed  and successful) request which were READ or above.

I used the RACF provided exits to the SMF dump program(IFASMFDP).   This produces readable file with all of the data.

I wrote an ISPF edit macro in rexx to take the data, and extract the key fields.

Below are the records produced for

  • an SSL connection using a certificate which mapped to userid ADCDB,
  • a logon with a userid and password with userid ADCDC

From the RESULT column, the userid ADCDC had

  • only READ access to,
  • no access to and, as we can see from the INSufficientAUTHority in the RESULT column.  The resource wanted READ – but had NONE.    This is OK, as we want the MQWebUser permissions.

From SUCCESS ADCDB READ /UPDATE APPL MQWEB  we can see that userid ADCDB (from the certificate) wanted READ access, but had UPDATE access to to MQWEB in the class APPL.

It does not look very optimised code, it looks like the logic is like

  • hmm it looks like this userid does not have access to this EJBROLE, let me check again
  • and again
  • and again
  • and again
  • and again
  • and again
  • OK give up, and try another resource
  • hmm it looks like you do have access to this other resource -let me check again..
  • and again…
  • … ok  you still have access – let’s go with this.

I expect is because a high level java program called a class to do some work, which checked the access; it called another class which did its own checks etc.  Understandable, but not efficient coding.

This all worked, I could see all of the access requests, but sadly I did not get a record saying “this certificate…. was mapped to userid ADCDB”.

How to use RACF callable services from C

Trying to use the RACF callable services was like trying to find treasure with an incomplete map.  I found it hard to create a C program to query a repository for ID information.   This post is mainly about using the RACF callable services.

I was trying to understand the mapping of a digital certificate to a z/OS userid, but with little success.  I found a RACF callable service which appeared to do what I wanted – but it did not give the answers – because,  like many treasure maps, I was looking in the wrong place.

RACF has two repositories for mapping identities to userid.

  • RACDCERT MAP which was the original way of mapping names.  As far as I can tell, the only way of getting the certificate to userid mapping programmatically, is to use the certificate to logon, and then find the userid!   This is used by Liberty Web Server.
  • RACMAP MAP which is part of Enterprise wide identification.   It maps identity strings, as you may get from LDAP,  to a userid. You can use the r_usermap callable service to get this information.

It took me some time to realise that these are different entities, and explains why there was no documentation on getting Liberty to work with RACMAP to handle certificates.  I found out RACMAP does not map certificate, after I got my program working.

The r_usermap service documentation is accurate – but incomplete, so I’ll document some of the things I learned in getting this to work.

The callable service to extract the userid  from identity information is documented here.  In essence you call the assembler routine r_usermap or IRRSIM00.

Building it

When you compile it, you need to provide the stub IRRSIM00 at bind time.  I used JCL

  Source program goes here

You need code like

#pragma linkage(IRRSIM00, OS) 
int main(){...
char * workarea ; 
workarea = (char *) malloc(1024)   ; 
long ALET1= 0; 
rc=  IRRSIM00(workarea, // WORKAREA 
             &ALET1  , // ALET 
             &SAF_RC, // SAF RC 

Some fields are in UTF-8.

To covert from EBCDIC to UTF-8, (it looks like ASCII )  I used

cd = iconv_open("UTF-8", "IBM-1047"); 
  short length ; // length of string following or 0 if ommitted 
  char value[248]; 
} DN; 
char * sDN= "CN=COLIN.C=GB"; 
size_t llinput= strlen(sDN); 
size_t lloutput= sizeof(DN.value); 
char * pOutValue= &DN.value[0]; 
rc = iconv(cd,        // from  iconv_open
           &sDN,      // input string
           &llinput,  // length of input 
           &pOutValue,// output 
           &lloutput);// length of output 
if (rc == -1) // problem
DN.length  =sizeof(DN.value) - ll2; // calculate true length

What access do I need?

You need

permit IRR.RUSERMAP class(FACILITY) access(READ) ID(....)


Once I had got the program to compile and bind, and got the authorisation it worked great.

It only works with the RACFMAP …  command, not the RACFDCERT command, obvious now I know!  To get the information from the RACDCERT MAP, you need to use initACEE.

What no one tells you about setting up your RACF groups – and how to do it for MQ.


The RACF documentation has a lot of excellent reference materials describing the syntax of the commands, but I could not find much useful information on how to set up RACF specifically for products like MQ, CICS, Liberty etc.

It is bit like saying programming has the following commands, load, store, branch; but fails to tell you that you can do wonderful things like draw Mandelbrot pictures using these instructions.

You need to plan your group structure before you try to implement security, as it is hard to change once it is in place.

The big picture

You can set up a hierarchy of groups so the site RACF person can set up a group called MQ, and give the MQ team manager authority to this group.

The manager can

  • define groups within it
  • connect users to the group
  • give other people authority to manage the group.

We can set up the following group structure

  • MQM
    • MQOPS – for the MQ operators
      • MQOPSR for operators who are allow to issue only Read (display) commands
      • MQOPSW for operators who can issue all command, display and update
    • MQADMS – for the MQ administrators
      • MQADMR – for MQ administrators who can only use display commands
      • MQADMW – for MQ administrators who can use all commands
    • MQWEB….

You should place an operator’s userid in only one group MQOPSR or MQOPSW as these are used to control access.  MQM, MQOPS, MQADMS, MQWEB are just used for administration.

You permit groups MQOPSR and MQOPSW to issue a display command, but only permit group MQOPSW to issue the SET command.

Setting up groups to make it easy to administer

A group needs an owner which administers the group.  The owner can be a userid or a group.

A group has been set up called MQM, and my manager has been made the owner of it.

My manager has connected my userid PAICE to the MQM group with group special.


I can define a new group MQOPS for example


The SUPGROUP says it is part of the hierarchy under MQM.  I can create the group under MQM because I am authorised,  If I try to create a group with SUPGROUP(SYS1) this will fail because I am not authorised to SYS1.

The OWNER(MQM) says people in the group MQM with group special can administer this new group.

Because my userid (PAICE) has group special for MQM, I can now connect users to the new group, for example


I can create another group under MQMD called MQMX, and connect a userid to it.


My userid PAICE can administer this because of the OWNER() inheritance up to GROUP(MQM)

If I list the groups I get

       PAICE    JOIN        000000              NONE 

       ADCDE     USE        000000              NONE 

All SUPGROUP() does is to define the hierarchy as we can see from the LISTGRP.    We can display the groups  and draw up a picture of the hierarchy.   You can use the LISTGRP command repeatedly,  or use the DSMON program(EXEC PGM=ICHDSM00) and use option
USEROPT RACGRP to get a picture like

3 | | MQMD 
4 | | | MQMX 
3 | | MQM2 (IBMUSER )

Using OWNER(group) instead of OWNER(userid)

  • If you have OWNER(groupname) it is easy to administer the groups.  When someone joins or leaves the department, you add or remove the userid from groupname.  One change.
  • If you have OWNER(userid), then you have to explicitly connect the userid to each group with group special.  When there is a new person you have to add the userid to each group individually.  When someone leaves the team you have to remove the persons userid from all of the groups. This could be a lot of work.


You could define an operator MQOP1 and give the userid group-special for group MQOPS.   This userid (MQOP1) can be used to add or remove userids in the MQOPSR and MQOPSW groups.

Looking at the MQOPS groups we could have groups and connected userids

  • MQM with MQ security userids PAICE, BOB having group-special
    • MQOPS with the operations manager and deputy MQOP1, MQOP2 having group special
      • MQOPSR with STUDENT1, STUDENT2 who are only allowed to issue display commands
    • MQADMS….

and similarly for the MQ administration eam.

Userid PAICE can connect userids to all groups.  MOP1 can only connect userids to the MQOPSR and MQOPSW, and not connect to the MQ ADMIN groups.

You use groups MQOPSR and MQOPSW for accessing resources. Groups MQM and MQOPS have no authority to access a resource, they are just to make the administration easier.

You may also want to consider having a group for application development.  The group called PAYRDEVT is under MQM, is owned by the manager of the payroll development team.

When the annual userid validation check is done, the development manager does the checks, and tells the security department it has been done.


There is no inheritance of permissions.  If a userid needs functions available to groups MQMD and MQMX, the user needs to be connected to both groups.

You only connect userids to groups, you cannot have groups within groups.  There may be many groups of userids which are allowed to issue an MQ display command, but only one group who can issue the SET command.


Suggested MQ groups

You need to consider

  • production and test environments
  • resources shared by queue managers, queue managers with the same configurations in a sysplex which can share definitions
  • queue managers as part of a Queue Sharing Group
  • queue manages that need isolation and so may have common operations groups, but different administration and programming groups.

You might define

  • Group MQPA for the queue manager super group. (MQ, Production system, A)
  • Groups for MQWEB. The Web server roles are described here.
  • Groups for controlling MQ, operations and administrations, read only or update
  • Groups for who can connect via batch, CICS etc
  • Groups for application usage, who can use which queues

Groups for MQWEB

For MQWEB the MQ documentation describes 4 roles: MQWebAdmin, MQWebAdminRO, MQWebUser, MFTWebAdmin; and there is console and REST access.

Each role should have its own group.  The requests from “Admin” and “Read Only” run with the userid of the MQWEB started task.   The request from “User” run with the signed on user’s authority.

You might set up groups

  • MQPAWCO MQPAMQWebAdminRO Console Read Only.
  • MQPAWCU MQPA – MQWebUser  Console User only.  The request operates under the signed on userid authority.
  • MQPAWCA MQPA – MQWebAdmin Console Admin.
  • MQPAWRO MQPA – MQWebAdminRO REST Read Only.
  • MQPAWRU MQPA – MQWebUser  REST User only.   The request operates under the signed on userid authority.
  • MQPAWRA MQPA – MQWebAdmin REST Admin Only.
  • MQPAWFO MQPA-  MFTWebAdmin MFT REST Read Only.

I would expect most people to be in

  • MQPAWCU MQPA – MQWebUser  Console User only.  The request operates under the signed on userid authority.
  • MQPAWRU MQPA – MQWebUser  REST User only.   The request operates under the signed on userid authority.

so you can control who does what, and get reports on any violations etc.  If people use the MQWEB ADMIN you do not know who tried to issue a command.

Groups for operations

The operations team may be managing multiple queue managers, so you may need groups

  • PMQOPS for Production
  • TMQOPS for Test

If some operators are permitted to manage only a subset of the queue managers you will need a group structure that can handle this, so have a special group XMQOPS for this.

  • XMQOPS for  the special queue manager

Groups for administration.

This will be similar to operations.

Groups for end users.

This is for people running work using MQ.

Usually there are checks to make sure a userid can connect to the queue manager, using the MQCONN resource.  Some customers have a loose security set up, and rely on the CICS to check to see if the userid is allowed to use a CICS transaction, rather than if the userid is allowed to access a queue.

Playing twister with Liberty and falling over

Twister is a game where you have you put your left foot here, your left hand there, your right foot here, and in trying to put your right hand over there you fall over.  This is how I felt when I was trying to understand the SSL definitions in MQWEB.  In the end I printed off the definitions and used coloured pens to mark the relevant data.

Let’s start with the easy bit.

  • When mqweb starts, it reads configuration information from a file server.xml
  • This includes two files the “IBM” stuff in  /usr/lpp/mqm/V9R1M1/web/mq/etc/mqweb.xml and the user stuff in /u/mqweb/servers/mqweb/mqwebusr.xml .
  • SSL parameters are defined with and <ssl… id=”thisSSLConfig”   keyStoreRef=”defaultKeyStore” .. /> tag.    This points to the keystore to use.
  • The keystore has <keyStore id=”defaultKeyStore”  ….>.   There is a simple link from SSL to the keystore.   You could have multiple keystores,  if so you just change the  keyStoreRef= to point to a different one.
  • You can have more than one <ssl…/> definition.  You might have one, and there is one in the “IBM stuff”, so you need <sslDefault sslRef=”thisSSLConfig”/>  to point to the ssl statement to use.

That should all be clear, and make sense.   A bit like saying you have a right foot, a left foot and two hands.

The zos_saf_registry.xml used when you want to use the SAF interface on z/OS has some SSL definitions.  I was trying to understand them.   This one here(put a finger on it) points to that one, (put a finger on it), which points to this other one (put a finger on it), which has an end-comment.  Whoops that didn’t work.    As I said a bit like playing Twister.

<sslDefault sslRef=”mqDefaultSSLConfig”/> in the user mqwebuser.xml  points to content in the “IBM stuff”.  By the various levels of indirection this points to <keyStore id=”defaultKeyStore” location=”key.jks” type=”JKS” password=”password”/> .  This keystore has a self signed certificate provided by IBM. If you find your browser complains about using a self signed certificate, this may well be the cause.

In the zos_saf_registry.xml are commented statements

  • <keyStore id=”defaultKeyStore” location=”safkeyring://userId/keyring” …/>
  • <ssl id=”thisSSLConfig”  keyStoreRef=”defaultKeyStore” …/>
  • <sslDefault sslRef=”thisSSLConfig”/>

To me these have been defined upside down, sslDefault should come first.

As these are after the sslDefault sslRef=”mqDefaultSSLConfig statment, if you uncomment them, they will be picked up and the “IBM stuff” will not be processed.

You can uncomment these statements and use them to add your definitions.

My definitions are

<sslDefault sslRef="defaultSSLConfig"/> 
<ssl id="defaultSSLConfig" keyStoreRef="racfKeyStore" 

<keyStore filebased="false" id="racfKeyStore" 

<webAppSecurity allowFailOverToBasicAuth="false"/>

Why cant I logoff from mqconsole?

If you are using mqweb using certificates to identify yourself, if you logoff, or close the tab, then open a new tab, you will get a session using the same certificate as before.

This little problem has been a tough one to investigate, and turns out to be lack of function in Chromium browser.

The scenario is you connect to mqweb using a digital certificate. You want to logoff and logon again with a different certificate, for example you do most of your work with a read only userid, and want to logon with a more powerful id to make a change.  You click logoff, and your screen flashes and logs you on again with the same userid as before.

At first glance this may look like a security hole, but if someone has access to your web browser, then the can click on the mqweb site, and just pick a certificate – so it is no different.

Under the covers,  the TLS handshake can pass up the previous session ID.   If the server  recognises this, then it is a short handshake instead of a full hand shake, so helping performance.

To reset the certificate if you are using Firefox

To clear your SSL session state in Firefox choose History -> Clear Recent History… and then select “Active Logins” and click “OK”. Then the next time you connect to your SSL server Firefox will prompt for which certificate to use, you may need to reset the URL.

You should check Firefox preferences, certificates, “Ask you every time” is selected, rather than “Select one automatically”.

Chrome does not support this reset of the certificate.

There has been discussion over the last 9 years along the lines of, seeing as Internet Explorer, and Firefox have there, should we do it to met the end user demand?

If you set up an additional browser instance, you get the same problem. With Chrome you have to close down all instances of the browser and restart chrome to be able to select a different certificate.

It looks like there is code which has a cache of url, and certificate to use.   If you open up another tab using the same IP address you will reuse the same certificate.

If you localhost instead of – it will prompt for certificate, and then cache it, so you can have one tab open with one certificate, and another tab, with a different URL and another certificate.

facepalm: Why my mqconsole was refreshing every 60 seconds

I had just upgraded MQ to v915 and was trying out the new mqconsole via the web stuff.  It was really annoying because every 60 seconds the web page would refresh, and I would have to start again.

I checked my mqwebuser.xml file for “60” seconds, but there were no hits.

Gwydion suggested I check the ltpaExpiration value.   This specifies how long a logon token is valid for; after it has expired you have to logon. again .  There were several problems here:

  • I had <variable name=”ltpaExpiration” value=”30″/> .   This says your session expires after 30 minutes.  Because I was using certificates to logon, I automatically get relogged on after this interval expires, this was 30 minutes not 1 minute so not my problem.
  • I used grep ltpa mqwebuser.xml and found I had several  <variable name=”ltpaExpiration” value=…./> statements. The last one had a value of 01 – one minute.   This was the statement causing my problems.

What did I learn from this?

  • Use the command  dspmqweb properties -a  to display all of the parameters.   If they are not as you expect, check the mqwebuser.xml file.
  • The setmqweb command deletes all the entries except for the last one, and updates the last entry with the specified value.   This solved my duplicate entry problem.
  • I specified setmqweb properties -k ltpaExpiration -v 015 to set it to 15 minutes.
  • You can put a comment  <!– unit in minutes –> before(near) the statement.
  • I also added unit=minutes into the statement <variable name=”ltpaExpiration” unit=”minutes” value=”016″/>.  setmqweb, and startup ignore this.

I tend not to use the setmqweb command as I pretend to be an enterprise where every change has to have a change record and where configurations are created centrally and deployed to the machines.

Whoops my certificate has expired – what do I need to do?

The amount of work depends on what has expired.

  • If the root CA has expired then you need to reissue all certificate which include the root CA, and update trust stores
  • If the enterprise CA has expired you need to renew all certificates signed by the enterprise CA, and update trust stores with the new certificate
  • If your personal certificate has expired, you need to renew that certificate.

The amount of work also depends on the size of your enterprise.

I’ll list the work need if the certificate is going to expire next week – as this is more work than if it has already expired

If your personal certificate is going to expire

You need to

  • renew it, create a new private key and certificate request
  • or extend the date of the existing certificate  – by re-signing it
  • update the keystores which include it.  This could be those used by java programs ( .p12, or .jks) as well as any browser keystore (.nssdb)

To renew your personal certificate

You can just recreate it.  If you are considering to use a stronger algorithms, such as Elliptic Curves, bear in mind that the servers need to be able to support what you specify.

Depending on how your openssl has been configured, if you have unique_subject = yes you may need to use openssl revoke… to remove the old personal certificate from the configuration.   This is to ensure there is only one certificate with the same distinguished name, and online checking of certificate validity (Certificate Revocation Lists and Online Status Certificate Protocol) can be used.  If you have unique_subject = no, you can recreate the certificate without revoking it.

  • openssl genpkey  to generate a new private key
  • openssl req to create a request

You do not need to recreate it – you can just sign it again.

You then need to do normal process of update the users of the certificate.

  • openssl ca to sign the request
  • openssl pkcs12 -export to create the .p12 file
  • You may want to deploy it only when the machine is attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • certutil -D $sql -n $shortName to remove the certificate from the browser’s keystore
  • pk12util -i $shortName.p12 $sql -W password to add the certificate to the browser’s keystore
  • /opt/mqm/bin/runmqckm -cert -delete  to remove a certificate from a keystore (.jks)
  • /opt/mqm/bin/runmqckm -cert -import to add a certificate to a keystore(.jks)
  • restart the servers to pick up the new certificate

Rather than deleting the certificate from the keystore and adding the new one with the same short name,  you may want to make a copy of the old certificate from the keystore, or simply backup the whole keystore – as it should not change once frequently.  If there are problems, restore from the backup.

To renew your enterprise certificate

This requires a lot more work than updating a personal certificate.

Just like the personal certificate you can recreate it, or just sign it again.

You need to phase in the new certificate as it may take days or weeks to deploy it successfully.

The client’s trust store needs the CA certificate matching the CA used by the server’s certificate.

If you have two servers, and change servers to use the new CA certificate, the client’s trust store content will change over time.

  • The old CA certificate – before any work is done
  • Both old and new CA certificates during the migration period.  The server using the old CA will use the old CA in the client.   The server which has been migrated to use the new CA will use the new CA in the client.
  • Just the new CA certificate.  Once the migration has finished. The old certificate can be removed when all of the servers have been migrated, or the certificate has expired.

A key store and a trust store can have certificate with the same distinguished names, but with a different short name or alias name.  My enterprise is called SSS, and the CA for my enterprise is CASSS… .  My trust store has

  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2016 and
  • the CA with C=GB,O=SSS,OU=CA,CN=CASSS and short name CASSS2020 and

When you maintain a keystore you use the short name, for my example CASSS2016 or CASSS2020.

During the TLS handshake, when the server’s certificate is sent to the client, it is checked against all of the trusted CA certificates in the keystore, so having two certificates with the same DN does not matter.

The steps to deploy your new CA are

  • Get the new CA.  Either create a new CA using the latest algorithms and cipher suites (the better solution), or have the old certificate resigned.
  • For each user and server
    • Deploy the CA certificate into the trust stores with a new name – eg CASSS2020
  • For each user and server
    • Renew or recreate each personal and server certificate and sign it with the new CA
    • Deploy it securely to each user and server and replace the old personal certificate
    • You may want to deploy the private key only when attached to the local network.  If you download it over wireless, or email, these may be compromised, and the bad guys can just use the new copy.  Downloading it over a wired connection eliminates this risk.
  • For the server create rules to disable weaker algorithms and cipher specs.   It is easy to undo this change.
    • This should identify any user not using the latest cipher suites and certificates.  These rules can be relaxed while problems are fixed.
  • For each user and server
    • Make a copy of the old CA certificate prior to deletion
    • Delete the old CA certificate from the trust stores eg CASSS
    • After a validation period delete the copy of the old certificate

Start all over again!

You can see how much work you need to do when you renew your enterprise CA, so make sure you  renew it in plenty of time – months rather than days.

mqweb – who did what to what, when, and how long did it take?

You can provide an audit trail of the http requests coming into your server.  This is described in the base liberty document, and works in mqweb.

Within the httpEndoint tag you can add

<httpEndpoint host=”${httpHost}” httpPort=”${httpPort}” httpsPort=”${httpsPort}” id=”defaultHttpEndpoint”>
<httpOptions removeServerHeader=”false”/>
<accessLogging enabled=”true” filePath=”${server.output.dir}/logs/http_access.log”
logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U X:%{X}W r:”%r” s:%s’


See here for information on the accessLogging,  here for the syntax of the <accessLogging…> and here for the logFormat format options.

From the logFormat page

Remote IP address
Local IP address
Response size in bytes excluding headers
Response size in bytes excluding headers. 0 is printed instead of – if no value is found.
The request cookie specified within the brackets, or if the brackets are not included, prints all of the request cookies.
The elapsed time of the request – millisecond accuracy, microsecond precision
Remote host
HeaderName header value from the request
Request method
HeaderName header value from the response
Output the query string with any password escaped
First line of the request
Service time of the request from the moment the request is received until the first set of bytes of the response is sent – millisecond accuracy, microsecond precision
Status code of the response
NCSA format of the start time of the request
The current time when the message to the access log is queued to be logged in normal NCSA format
Remote user according to the WebSphere Application Server specific $WSRU header
URL Path, not including the query string
Cross Component Tracing (XCT) Context ID


logFormat=’a:%a A:%A D:%D h:%h HeaderHost:%{Host}i HeaderOrigin:%{Origin}i m:%m R:%{R}W t:%{t}W u:%u U:%U  r:”%r” s:%s’

gave me

  • a: remote IP address
  • A: local host
  • D:225960 duration in millisecond
  • h: remote host
  • HeaderHost: the Host header
  • HeaderOrigin:- the Origin header (missing in this case)
  • m:GET request method
  • R:186930 Service time of the request from the moment the request is received until the first set of bytes of the response is sent
  • t:[27/Feb/2020:17:04:27 +0000] NCSA format of the start time of the request
  • u:colinpaice remote user
  • U:/ibmmq/console url path
  • r:”GET /ibmmq/console HTTP/1.1″ First line of the request
  • s:302 Status code of the response

There are other log formatting options available, I picked those I thought were most useful.

Note when using the MQConsole from a browser, the interface is chatty. I had 20 request to refresh one window.

Other ways of formatting the data

  • I separated each field with a ‘,’ and could read it into a spread sheet.
  • You could configure your log format string to produce the output in JSON format, to make it easier to post process.