Some of the mysteries of Java shared classes

When Java executes a program it read in the jar file, breaks it into the individual classes, converts the byte codes into instructions, and when executing it may replace instructions with more efficient instructions (Jitting). It can also convert the byte codes into instructions ahead of time, so called Ahead Of Time (AOT) compilation.

With shared classes, the converted byte codes, any Jitted code, and any AOT code can be saved in a data space.

  • When the java program runs a second time, it can reuse the data in the dataspace, avoid the overhead of the reading the jar file from the file system, and coverting the byte codes into instructions.
  • The data space can be hardened to a file, and restored to a data space, so can be used across system IPLs.

Using this, it reduced the start-up time of my program by over 20 seconds on my slow zPDT system. The default size of the cache is 16MB – one of my applications needed 100 MB, so most of the benefits of he shared classes could not be exploited if the defaults were used.

This blog post describes more information about this, and what tuning you can do.

Issuing commands to manage the shared classes cache

Commands to manage the shared classes cache are issued like

java -Xshareclasses:cacheDir=/tmp,name=client6,printStats

which can be done using JCL

// SET V=’listAllCaches’
// SET V=’printStats’
// SET C=’/tmp’
// SET N=’client6′
//S1 EXEC PGM=BPXBATCH,REGION=0M,
// PARM=’SH java -Xshareclasses:cacheDir=&C,name=&N,verbose,&V’
//STDERR DD SYSOUT=*
//STDOUT DD SYSOUT=*

Enabling share classes

You specify -Xsharedclasses information as a parameter to the program, for example in the command line or in a jvm properties file.

To use the shared classes capability you have to specify all of the parameters on one line, like

-Xshareclasses:verbose,name=client6,cacheDirPerm=0777,cacheDir=/tmp

Having it like

-Xshareclasses:name=client6,,cacheDirPerm=0777,cacheDir=/tmp
-Xshareclass:verbose

means the name, etc all take their defaults. Only shareclass:verbose would be used.

Changing share classes parameters

You can have more than one cache; you specify a name. You specify a directory were an image is stored when the cache is hardened to disk.

Some of the options like name= and cacheDir= are picked up when the JVM starts, Other parameters like cacheDirPerm are only used when the cache is (re-)created.

You can delete the cache in two ways.

Delete the cache from your your Java program

When you are playing around, you can add reset to the end of the -Xshareclasses string to caused the cache to be deleted and recreated.This gives output like

JVMSHRC010I Shared cache “client6” is destroyed
JVMSHRC158I Created shared class cache “client6”
JVMSHRC166I Attached to cache “client6”, size=20971328 bytes

This was especially useful when tuning the storage allocations.

Delete the cache independently

java -Xshareclasses:cacheDir=/tmp,name=client6,destroy

How to allocate the size of the cache

You specify the storage allocations using -Xsc.. (where sc stands for shareclasses)

If you have -Xsharedcache:verbose… specified then when the JVM shuts down you get

JVMSHRC168I Total shared class bytes read=11660. Total bytes stored=5815522
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0.
Unstored AOT bytes due to the setting of -Xscmaxaot is 1139078.
Unstored JIT bytes due to the setting of -Xscmaxjitdata is 131832.

This shows the values of maxaot and maxjitdata are too small they were

-Xscmx20m
-Xscmaxaot2k
-Xscmaxjitdata2k

Whem the values were big enough I got

JVMSHRC168I Total shared class bytes read=12960204. Total bytes stored=8885038
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0.
Unstored AOT bytes due to the setting of -Xscmaxaot is 0.
Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0.

How big a cache do I need?

If you use -Xshareclasses:verbose… it will display messages

for example

JVMSHRC166I Attached to cache “client6”, size=2096960 bytes
JVMSHRC269I The system does not support memory page protection

JVMSHRC096I Shared cache “client6” is full. Use -Xscmx to set cache size.
JVMSHRC168I Total shared class bytes read=77208. Total bytes stored=2038042

Message JVMSHRC096I Shared cache “client6” is full. Use -Xscmx to set cache size, tells you the cache is full – but no information about how big it needs to be.

You can use

java -Xshareclasses:cacheDir=/tmp,name=client6,printStats

to display statistics like

-Xshareclasses persistent cache disabled]                                         
[-Xshareclasses verbose output enabled]                                            
JVMSHRC159I Opened shared class cache "client6"                                    
JVMSHRC166I Attached to cache "client6", size=2096960 bytes                        
JVMSHRC269I The system does not support memory page protection                     
JVMSHRC096I Shared cache "client6" is full. Use -Xscmx to set cache size.          
                                                                                   
Current statistics for cache "client6": 
cache size                           = 2096592                       
softmx bytes                         = 2096592                       
free bytes                           = 0                             
ROMClass bytes                       = 766804                        
AOT bytes                            = 6992                          
Reserved space for AOT bytes         = -1                            
Maximum space for AOT bytes          = 1048576                       
JIT data bytes                       = 212                           
Reserved space for JIT data bytes    = -1                            
Maximum space for JIT data bytes     = 1048576                       
Zip cache bytes                      = 1131864                       
Startup hint bytes                   = 0                             
Data bytes                           = 13904                         
Metadata bytes                       = 12976                         
Metadata % used                      = 0%                            
Class debug area size                = 163840                        
Class debug area used bytes          = 119194                        
Class debug area % used              = 72%

Cache is 100% full  
                                                                             

This show the cache is 100% full, and how much space is used for AOT and JIT. The default value of -Xscmx I had was almost 16MB. I made it 200MB and this was large enough.

I could not find a way of getting my program to issue printStats.

How do I harden the cache?

You can use use the

java -Xshareclasses:cacheDir=/tmp,name=zosmf,verbose,snapshotCache

command to create the cache on disk. Afterwards the listAllCaches command gave

Cache name level        cache-type     feature 
client6    Java8 64-bit non-persistent cr
client6    Java8 64-bit snapshot       cr

Showing the non persistent data space, and the snapshot file.

You can use the restoreFromSnapshot to restore from the file to the data cache; before you start your Java program. You would typically do this after an IPL.

How can I tell what is going on and if shared classes is being used?

The java options “-verbose:dynload,class

reports on the

  • dynamic loading of the files, and processing them,
  • what classes are being processed.

For example

<Loaded java/lang/reflect/AnnotatedElement from /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar>
< Class size 3416; ROM size 2672; debug size 0>
< Read time 1196 usec; Load time 330 usec; Translate time 1541 usec>
class load: java/lang/reflect/AnnotatedElement from: /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar
class load: java/lang/reflect/GenericDeclaration from: /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar

dynload gave

<Loaded java/lang/reflect/AnnotatedElement from /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar>
< Class size 3416; ROM size 2672; debug size 0>
< Read time 1196 usec; Load time 330 usec; Translate time 1541 usec>

this tells you a jar file was read from the file system, and how long it took to process it.

class gave

class load: java/lang/reflect/AnnotatedElement from: /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar
class load: java/lang/reflect/GenericDeclaration from: /Z24A/usr/lpp/java/J8.0_64/lib/rt.jar

This shows two classe were extracted from the jar file.

In a perfect system you will get the class load entries, but not <Loaded….

Even when I had a very large cache size, I still got dynload entries. These tended to be loading class files rather than jar files.

For example there was a dynload entry for com/ibm/tcp/ipsec/CaApplicationProperties. This was file /usr/lpp/zosmf./installableApps/izuCA.ear/izuCA.war/WEB-INF/classes/com/ibm/tcp/ipsec/CaApplicationProperties.class

If you can make these into a .jar file you may get better performance. (But you may not get better performance, as it may take more time to load a large jar file).

I also noticed that there was dynload for com/ibm/xml/crypto/IBMXMLCryptoProvider which is in /Z24A/usr/lpp/java/J8.0_64/lib/ext/ibmxmlcrypto.jar, so shared classes has some deeper mysteries!

What happens if the .jar file changes?

As part of the class load, it checks the signature of the file on disk, matches the signature on the data space. If they are different the data space will be updated.

“Why were the options I passed to Java ignored” – or “how to tell what options were passed to my Java?”

I was struggling to understand a problem with shared classes in Java and I found the options being used by my program were not as I expected.

I thought it would be a very simple task to display at start up options used. It may be, but I could not find how to do it. If anyone knows the simple answer please tell me.

I found one way – take a dump! This seems a little extreme, but it was all I could find. With Liberty you can take a javacore dump (F IZUSVR1,JAVACORE) and display it, or you can take a dump at start up.

In the jvm.options I specified

-Xdump:java:events=vmstart

This gave me in //STDERR

JVMDUMP039I Processing dump event “vmstart”, detail “” at 2021/05/20 13:19:06 – please wait.
JVMDUMP032I JVM requested Java dump using ‘/S0W1/var/…/javacore.20210520.131906.65569.0001.txt’
JVMDUMP010I Java dump written to /S0W1/var…/javacore.20210520.131906.65569.0001.txt
JVMDUMP013I Processed dump event “vmstart”, detail “”.

In this file was list of all the options passed to the JVM

1CIUSERARGS UserArgs:
2CIUSERARG -Xoptionsfile=/usr/lpp/java/J8.0_64/lib/s390x/compressedrefs/options.default
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/Ma
2CIUSERARG -Xjcl:jclse29
2CIUSERARG -Djava.home=/usr/lpp/java/J8.0_64
2CIUSERARG -Djava.ext.dirs=/usr/lpp/java/J8.0_64/lib/ext
2CIUSERARG -Xthr:tw=HEAVY
2CIUSERARG -Xshareclasses:name=liberty-%u,nonfatal,cacheDirPerm=1000,cacheDir=…
2CIUSERARG -XX:ShareClassesEnableBCI
2CIUSERARG -Xscmx60m
2CIUSERARG -Xscmaxaot4m
2CIUSERARG -Xdump:java:events=vmstart
2CIUSERARG -Xscminjitdata5m
2CIUSERARG -Xshareclasses:nonFatal
2CIUSERARG -Xshareclasses:groupAccess
2CIUSERARG -Xshareclasses:cacheDirPerm=0777
2CIUSERARG -Xshareclasses:cacheDir=/tmp,name=zosmf2
2CIUSERARG -Xshareclasses:verbose
2CIUSERARG -Xscmx100m

the storage limits

1CIUSERLIMITS User Limits (in bytes except for NOFILE and NPROC)
NULL ------------------------------------------------------------------------
NULL         type            soft limit  hard limit
2CIUSERLIMIT RLIMIT_AS       1831837696   unlimited
2CIUSERLIMIT RLIMIT_CORE        4194304     4194304
2CIUSERLIMIT RLIMIT_CPU       unlimited   unlimited
2CIUSERLIMIT RLIMIT_DATA      unlimited   unlimited
2CIUSERLIMIT RLIMIT_FSIZE     unlimited   unlimited
2CIUSERLIMIT RLIMIT_NOFILE        10000       10000
2CIUSERLIMIT RLIMIT_STACK     unlimited   unlimited
2CIUSERLIMIT RLIMIT_MEMLIMIT 4294967296  4294967296

and environment variables used.

1CIENVVARS Environment Variables
2CIENVVAR LOGNAME=IZUSVR
2CIENVVAR _CEE_ENVFILE_S=DD:STDENV
2CIENVVAR _EDC_ADD_ERRNO2=1
2CIENVVAR _EDC_PTHREAD_YIELD=-2
2CIENVVAR WLP_SKIP_MAXPERMSIZE=true
2CIENVVAR _BPX_SHAREAS=NO
2CIENVVAR JAVA_HOME=/usr/lpp/java/J8.0_64

All interesting stuff including the -X.. parameters. I could see that the parameters I had specified were not being picked up because they were specified higher up! Another face palm moment.

There was a lot more interesting stuff in the file, but this was not relevant to my little problems.

Once z/OSMF was active I took a dump using the f izusvr1,javacore command and looked at the information on the shared classes cache

1SCLTEXTCMST Cache Memory Status
1SCLTEXTCNTD Cache Name      Cache path
2SCLTEXTCMDT sharedcc_IZUSVR /tmp/javasharedresources/..._IZUSVR_G37
2SCLTEXTCPF Cache is 85% full
2SCLTEXTCSZ Cache size = 104857040
2SCLTEXTSMB Softmx bytes = 104857040
2SCLTEXTFRB Free bytes = 14936416

This is where I found the shared cache was not what I was expecting! I originally spotted that the cache was too small – and made it bigger.

Afterwards…

Remember to delete the javacore files.

I removed the -Xdump:java:events=vmstart statement, because I found it more convenient to use the f izusvr1,javacore command to take a dump when needed.

Looking at the performance of Liberty products and what you can do with the information

The Liberty Web Server on z/OS can produce information on the performance of “transactions”. It may not be as you expect, and it may not be worth collecting it. I started looking into this to see why certain transactions were slow on z/OSMF.

This blog post covers

  • How does the web server work?
  • How can I see what is going on?
  • Access logs
  • SMF 120
  • WLM
  • Do I want to use this in production?

How does it work?

A typical Web Server transaction might use a database to debit your account, and to credit my account. I think of this as a fat transaction because the application does a lot of work,

z/OSMF runs on top of Liberty, and allows you to run SDSF and ISPF within a web brower, and has tools to help automate work. You can also use REST services, so you can send a self contained HTTP request to z/OSMF, for example request information about data sets, or jobs running on the system. Both of these send a request to z/OSMF, which might send a request to a TSO userid, get the response back and pass the response back to the requester. I think of this as a thin transaction because the application running on the web server is mainly acting as a router or broker. What the end user sees as a “transaction” may be many micro services – each of which is a REST requests.

How can I see what is going on?

You can see what “transactions” or requests have been issued from

  • the access log – a flat file of requests
  • SMF 120 records
  • WLM reports.

Access logs

In the <httpEndpoint … accessLoggingRef=”accessLogging”…>, the accessLogging is a reference to an <httpAccessLogging… statement. This creates a flat files containing a record of all inbound HTTP client requests. If you have multiple httpEndpoint statements, you can have multiple accessLogging files. You can control the number and size of the files.

This has information with fields like

  • 10.1.0.2 – the IP address the request came in from
  • COLIN – the userid
  • {16/May/2021:14:47:40 +0000} – the date, time and time zone of the event
  • PUT /zosmf/tsoApp/tso/ping/COLIN-69-aabuaaac – the request
  • HTTP/1.1 – the type of HTTP request
  • 200 – the HTTP return code
  • 78 – the number of bytes sent back.

You can have multiple httpEndpoint definitions, for example specifing different IP address and port combinations. These definitions can point to a shared or indivdual httpAccessLogging statement, and so can share (or not) the flat files. This allows you to specify that for one port you will use the accessLogging, and another port does not have access logging.

The SMF 120 records.

The request logging writes a record to SMF for each request.

<server>
<featureManager>
<feature>zosRequestLogging-1.0</feature>
</featureManager>
</server

I have not been able to find a formatter for these records from IBM, so I have written my own, it is available on github.

It produces output like

Server version      :2                                                              
System              :S0W1                                                           
Syplex              :ADCDPL                                                         
Server job id       :STC02771                                                       
Server job name     :IZUSVR1                                                        
config dir          :/var/zosmf/configuration/servers/zosmfServer/                  
codelevel           :19.0.0.3  
                                                     
Start time          :2021/05/16 14:42:33.955288                                     
Stop  time          :2021/05/16 14:42:34.040698                                     
Duration in secs    : 0.085410                                                      
WLMTRan             :ZCI4                                                           
Enclave CPU time    : 0.042570                                                      
Enclave ZIIP time   : 0.042570                                                      
WLM Resp time ratio :10.000000                                                      
userid long         :COLIN                                                          
URI                 :/zosmf/IzuUICommon/externalfiles/sdsf/index.html               
CPU Used Total      : 0.040111                                                      
CPU Used on CP      : 0.000000                                                      
CPU Delta Z**P      : 0.040111 
                                                     
WLM Classify type   :URI                                                            
WLM Classify by     :/zosmf/IzuUICommon/externalfiles/sdsf/index.html
WLM Classify type   :Target Host                            
WLM Classify by     :10.1.1.2                               
WLM Classify type   :Target Port                            
WLM Classify by     :10443   
                               
Response bytes      :7003                                   

Origin port         :57706                                  
Origin              :10.1.0.2                                              

This has information about

  • The Liberty instance, where it is running and configuration information
  • Information about the transaction, start time, stop time, duration and CPU time used
  • WLM information about the request. It was classified as
    • URI:…index.html
    • Target Host:10.1.1.2
    • Target port:10443
  • 7003 bytes were sent to the requester
  • the request came from 10.1.0.2 port 57706

The SMF formatter program also summarises the records and this shows there are records for

  • /zosmf/IzuUICommon/externalfiles/sdsf/js/ui/theme/images/zosXcfsCommand_enabled_30x30.png
  • /zosmf/IzuUICommon/externalfiles/sdsf/js/ui/theme/sdsf.css
  • /zosmf/IzuUICommon/externalfiles/sdsf/sdsf.js
  • /zosmf/IzuUICommon/externalfiles/sdsf/IzugDojoCommon.js
  • /zosmf/IzuUICommon/persistence/user/com.ibm.zos.sdsf/JOBS/JOBS

This shows there is a record for each part of the web page, icons, java scripts and style sheets.

Starting up SDSF within z/OSMF created 150 SMF records! Refreshing the data just created 1 SMF record. The overhead of creating all the SMF records for one “business transaction” may be too high for production use.

As far as I can tell this configuration is server wide. You cannot enable it for a specific IP address and port combination.

WLM reports

Much of the data produced in the records above can be passed to WLM. This can be used to give threads appropriate priorities, and can produce reports.

You enable WLM support using

  • <featureManager>
    • <feature>zosWlm-1.0 </feature>
  • </featureManager>
  • <zosWorkloadManager collectionName=”MOPZCET”/>
  • <wlmClassification>
    • <httpClassification transactionClass=”ZCI6″ resource=”/zosmf/desktop/”/>
    • <httpClassification transactionClass=”ZCI1″ resource=”/**/sdsf/**/*”/>
    • <httpClassification transactionClass=”ZCI3″ resource=”/zosmf/webispf//”/>
    • <httpClassification transactionClass=”ZCI4″ resource=”/**/*”/>
    • <httpClassification transactionClass=”ZCI2″ resource=”IZUGHTTP” />
    • <httpClassification transactionClass=”ZCI5″ port=”10443″/>
  • </wlmClassification>

Where the httpClassification maps a z/OSMF resource to an 8 character transaction class. THe records are process from the top until there is a match. For example port=10443 would not be used because of the generic resource=/**/* definition.

These need to be mapped into the WLM configuration…

WLM configuration

You can configure WLM through the WLM configuration panels.

option 6. Classification rules.

option 3 to modify CB(Component broker)

          -------Qualifier--------                 -------Class--------  
 Action   Type      Name     Start                  Service     Report   
                                          DEFAULTS: STCMDM      TCI2
  ____  1 CN        MOPZCET  ___                    ________    THRU
  ____  2   TC        ZCI1     ___                  SCI1        RCI1
  ____  2   TC        ZCI2     ___                  SCI2        RCI2
  ____  2   TC        ZCI3     ___                  SCI3        RCI3
  ____  2   TC        ZCI4     ___                  SCI4        RCI4
  ____  2   TC        ZCI5     ___                  SCI5        RCI5
  ____  2   TC        ZCI6     ___                  SCI6        RCI6
  ____  2   TC        THRU     ___                  THRU        THRU

For the Type=CN, Name=MOPZCET, this value ties up with the <zosWorkloadManager collectionName=”MOPZCET” above. Type=CN is for CollectionName.
For the subtype 2 TC Name ZCI4, This is for TransactionClass which ties up with a httpClassification transactionClass statement.

The service class SCI* controls the priority of the work, the Report class RCI* allow you to produce a report by this name.

If you make a change to the WLM configuration you can save it from the front page of the WLM configuration panels, Utilities-> 1. Install definition, and activate it using Utilities-> 3. Activate service policy.

If you change the statements in Liberty or z/OSMF I believe you have to restart the server.

How to capture the data

The data is written out to SMF records on a timer, or on the SMF end-of-interval broadcast. If you change the interval, SMF sends an end-of-interval broadcast and writes the records to SMF. For example on my dedicate test system I use the operator command setsmf intval(10) to change the interval to 10 minutes. After the next test, I use setsmf intval(15) etc..

The data is kept in SMF buffers, and you may have to wait for a time, before the data is written out to external storage. It SMF data is being produced on a regular basis, it will be flushed out.

How to report the data

I copy the SMF data to a temporary data

//IBMURMF  JOB 1,MSGCLASS=H RESTART=POST 
//* DUMP THE SMF DATASETS 
// SET SMFPDS=SYS1.S0W1.MAN1 
// SET SMFSDS=SYS1.S0W1.MAN2 
//* 
//SMFDUMP  EXEC PGM=IFASMFDP 
//DUMPINA  DD   DSN=&SMFPDS,DISP=SHR,AMP=('BUFSP=65536') 
//DUMPINB  DD   DSN=&SMFSDS,DISP=SHR,AMP=('BUFSP=65536') 
//* MPOUT  DD   DISP=(NEW,PASS),DSN=&RMF,SPACE=(CYL,(1,1)) 
//DUMPOUT  DD   DISP=SHR,DSN=IBMUSER.RMF SPACE=(CYL,(1,1)) 
//SYSPRINT DD   SYSOUT=* 
//SYSIN  DD * 
  INDD(DUMPINA,OPTIONS(DUMP)) 
  INDD(DUMPINB,OPTIONS(DUMP)) 
  OUTDD(DUMPOUT,TYPE(70:79)) 
  DATE(2020316,2021284) 
  START(1539) 
  END(2359) 
/* 

and display the report classes

//POST EXEC PGM=ERBRMFPP 
//* PINPUT DD DISP=SHR,DSN=*.SMFDUMP.DUMPOUT 
//MFPINPUT DD DISP=SHR,DSN=IBMUSER.RMF 
//SYSIN DD * 
SYSRPTS(WLMGL(RCPER)) 
/* 

The output in //PPXSRPTS was

REPORT CLASS=RCI4                                                      
-TRANSACTIONS--  TRANS-TIME HHH.MM.SS.FFFFFF  
AVG        0.01  ACTUAL                69449  
MPL        0.01  EXECUTION             68780  
ENDED       160  QUEUED                  668  
END/S      0.13  R/S AFFIN                 0  
#SWAPS        0  INELIGIBLE                0  
EXCTD         0  CONVERSION                0  
                 STD DEV              270428  
                                              
----SERVICE----   SERVICE TIME  ---APPL %---  
IOC           0   CPU    2.977  CP      0.05  
CPU        2551   SRB    0.000  IIPCP   0.05  
MSO           0   RCT    0.000  IIP     0.19  
SRB           0   IIT    0.000  AAPCP   0.00  
TOT        2551   HST    0.000  AAP      N/A  
/SEC          2   IIP    2.338                
ABSRPTN     232   AAP      N/A                
TRX SERV    232                               
                                              

There were 160 “transactions” within the time period or 0.13 per second, The average response time was 69449 microseconds, with a standard deviation of 270428. This is a very wide standard deviation, so there was a mixture of long, and short response times.

The total CPU for this report class was 2.977 seconds of CPU, or 0.019 seconds per “transaction”.

Do I want to use this in production?

I think that the amount of data produced is managable for a low usage system. For a production environment where there is a lot of activity then the amount of data produced, and the cost of producing the data may be excessive. This could be an example of the cost of collecting the data is much larger than the cost of running the workload.

As z/OSMF acts as a broker, passing requests between end points you may wish just to use your existing reporting structures.

I used z/OSMF to display SDSF data, and set up and ISPF session within the web browse. This created two TSO tasks for my userid. If you include my traditional TSO session and the two from z/OSMF this is three TSO sessions in total running on the LPAR.

Double whoops – I’ve been ransomware-d and … it gets worse

This story comes from a friend of a friend, so I cannot tell how much it true, but it another good example of showing some things are “obvious” only when you understand it.

I heard that some one’s office systems had been compromised by a ransomware attack, where their files had been encrypted and they money was demanded to decrypt them. The first whoops. While someone else was sorting the problem for the office, the person was pleased that he kept backups of all his key files, on a separate portable hard disk drive attached via a USB cable. The person realised that his backup files had been encrypted as well, so he was unable to restore his own backups of his key files. The second whoops. Looking back this was an obvious consequence of having the backups connected connected to the machine.

I would have expected that any decent backup package would make the files read only, but then if the ransomware was able to get into administrator mode, it would be able to change these “protected” files as well.

I think the only answer is to take backups off your machine – for example over a network, and hope the ransomware is smart enough not to corrupt files across a network. You could also backup your key files to a CD which is write once, and then becomes read only.

As I wrote this I remembered that I had been meaning to backup some family photographs and documents that only exist on my machine (and backups). I had sent a copy to my brother, but when he got a new machine he did not copy the files across!

I was also reminded of the University that diligently backed up the system every week. Which was fine until the building with the computer, and cupboard full of backups was destroyed in a fire.

Understanding display authrec match(…) on midrange.

I found the documentation for the display authrec match(…) command hard to understand. There are ambiguous backward references (the profile … which profile?), too many ‘and’s, and I think some ‘or’s are missing. Below is how I interpret it.

  • Match exact: Select the record where the specified profile name specified is an exact match for a setmqauth record
  • Match profile: Display the setmqaut records which would be used to compute permissions, for the specified profile, and (specified userid or specified group).
  • Match membership:
    • For the specified userid do Match: profile() for the userid
    • For each group the specified userid is in, do Match: profile() for that group

I see the processing is in two stages

  1. Stage 1 extract the autrecs for the specified profile name
  2. Stage 2 filter the list using the specified userid or group.

Stage 1: extract the authrecs matching the specified profile name.

For match exact

Select the record where the specified profile name specified is an exact match for a setmqauth record. Profile(‘CP.**.99’) will match only ‘CP.**.99’.

For match profile and membership

If the specified profile has a generic then treat this as match(exact).

If the specified profile has no generics then extract all records which would apply when checking this profile.

For example for queue CP.AA.BB.99 might return

  • profile CP.AA.BB.99 (entity colinpaice) – this userid created the queue
  • profile CP.AA.BB.99 (entity mqm) – this entry is create when the queue is created
  • profile CP.**.99 (entity testuser) – this was done via setmqaut – not selected because less specific generic profile
  • profile CP.*.BB.99 (entity testuser) – this was done via setmqaut.

Stage 2. Filter the records depending on the specified userid or group.

Take the set of records from stage 1 and filter them. You can specify the principal (userid) or group. Note: If your qm.ini has SecurityPolicy=group then even if you have specified you setmqauth with a userid, it will use a group instead. This may mean that displaying a userid may give no results.

Match exact

Compare the specified entity and entity type with those in the the records. If they match display the record.

Match profile

Compare the specified entity, and entity type with those in the the records. If they match display the record.

Match membership

Compare the specified entity, and entity type with those in the the records. If they match display the record.

If the entity type in the record is group, and the specified userid is a member of the group then display it.

Examples

Match(profile) principal()

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   principal('testuser')
AMQ8459I: Not found.    

There is no profile defined for the userid ‘testuser’

Match(profile) group()

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   group('testuser')
   PROFILE(CP.*.99)                        ENTITY(testuser)  

There was a setmqauth -m -n “CP.*.99” -t queue -p testuser +get . Because of the qm.ini setting, and userid authorisations were converted to group authorisations. On some Unix systems, when a userid is created, it creates a group with the same name, and connects the userid to the group.

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   group('test') 
AMQ8864I: Display authority record details.
    PROFILE(**) ENTITY(test) AUTHLIST(DSP,INQ) 

All members of group test, have Display and Inquire permissions on any queue.

Match(membership) principal()

dis authrec profile(CP.AA.99) objtype(queue) match(membership) principal('testuser')
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)
                         
AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)  

The display match(membership) combines all of the above. Any specific records, plus records for any group testuser is in.

Match(membership) group()

dis authrec profile(CP.AA.99) objtype(queue) match(membership) group('testuser')
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)

Because the group is specified, then this acts the same as match(profile) group(‘testuser’).

Match(profile) no group nor principal

dis authrec profile(CP.AA.99) objtype(queue) match(profile)   
AMQ8864I: Display authority record details.
   PROFILE(CP.AA.99)                       ENTITY(colinpaice)
 
AMQ8864I: Display authority record details.
   PROFILE(CP.AA.99)                       ENTITY(mqm)
 
AMQ8864I: Display authority record details.
   PROFILE(CP.*.99)                        ENTITY(testuser)
                
AMQ8864I: Display authority record details.
   PROFILE(CP.**.99)                       ENTITY(testuser)
                 
AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)

No userids or groups were specified, so all relevant autrecs for the profile CP.AA.99 are displayed.

Observation: There is a profile for PROFILE(CP.**.99) ENTITY(testuser) which does not show up when dis authrec profile(CP.AA.99) objtype(queue) match(membership) principal(‘testuser’) is used.

This is because with generic profiles, only the most specific generic profile is used, see Profile Priorities.

MQ Context on z/OS

Having struggled to get MQ Context working on mid range MQ, I thought I would try the same on z/OS.

If you want to allow applications to set Putdate, Putime, PutApplName etc. The application needs access to MQ Context. MQ MCA channels use this when putting a message from a remote queue manager, to keep the original values.

Which profiles are used?

You can disable context checking by defining a profile ‘qmgr.NO.CONTEXT.CHECKS’. If you want to enable context checking remove this profile if it exists.

You can display it using

RLIST MQADMIN CSQ9.NO.CONTEXT.CHECKS

You configure queue context using the profile qmrg.context.queue

for example

RLIST MQADMIN CSQ9.CONTEXT.CP0000 all
CLASS NAME
----- ----
MQADMIN CSQ9.CONTEXT.** (G)
...
LEVEL  OWNER      UNIVERSAL ACCESS  YOUR ACCESS  WARNING
-----  --------   ----------------  -----------  -------
 00    IBMUSER          NONE             ALTER    NO
...
USER ACCESS
---- ------
IBMUSER ALTER

This says that for the queue CP0000, display the profile CSQ9.CONTEXT.CP0000. It returned

  • MQADMIN CSQ9.CONTEXT.** this is the profile used
  • IBMUSER ALTER the only user authorised to this resource – with ALTER access it IBMUSER
  • The default access is NONE.

When a userid tried to open the queue – with set context options, the open got return code 2035 and a message on the console.

ICH408I USER(COLIN ) GROUP(SYS1 ) NAME(COLIN PAICE )
CSQ9.CONTEXT.CP0000 CL(MQADMIN )
INSUFFICIENT ACCESS AUTHORITY
FROM CSQ9.CONTEXT.** (G)
ACCESS INTENT(CONTROL) ACCESS ALLOWED(NONE )

This shows the resource used CSQ9.CONTEXT.CP0000. The RACF profile used was CSQ9.CONTEXT.**. The userid had NONE access, and wanted CONTROL access.

You could define a more specific profile for example CSQ9.CONTEXT.CP*, and that would be used in preference to the CSQ9.CONTEXT.** profile.

The z/OS documentation Determining RACF protection says

Although multiple generic profiles can match a general resource name, only the most specific profile
actually protects it. For example, AB.CD, AB.CD.* and AB.**.CD all match the general resource name AB.CD, but AB.CD.* protects the resource.

With Midrange MQ on Unix, the permission is taken from all of the groups the userid is in- if one of the userid’s groups has get authority, the userid has get authority. With z/OS just one profile is used.

Changing a profile – don’t forget to refresh.

When changing a profile you need to remember to refresh the RACF in memory profiles, and tell MQ to pick up the changes.

I changed a profile

ralter MQADMIN CSQ9.CONTEXT.** UACC(CONTROL)

Refreshed the RACF in-memory profiles

setropts racflist(MQADMIN) refresh

And told MQ to refresh its profiles

%csq9 refresh security

How easy is it to display security information for MQ on z/OS?

I asked this question for midrange, and here are the answers for z/OS.

Key question are

Displaying security information

The RACF commands are

RLIST to display profile information, and who has access to the profile

SEARCH This allows you to search for profiles matching a parameter.

RLIST display profile information

An example command and output

RLIST MQADMIN CSQ9.CONTEXT.CP0000 all
CLASS NAME
----- ----
MQADMIN CSQ9.CONTEXT.** (G)
...
LEVEL  OWNER      UNIVERSAL ACCESS  YOUR ACCESS  WARNING
-----  --------   ----------------  -----------  -------
 00    IBMUSER          NONE             ALTER    NO
...
USER ACCESS
---- ------
IBMUSER ALTER

This says display the profile CSQ9.CONTEXT.CP0000. It returned

  • MQADMIN CSQ9.CONTEXT.** this is the profile used by RACF to determine the permissions
  • IBMUSER ALTER the only user authorised to this resource is IBMUSER, with ALTER access
  • The default access for any userid not covered is NONE.

SEARCH for a profile

An example command to list all MQQueue profiles for queue manager CSQ9.

SEARCH CLASS(MQQUEUE) FILTER(CSQ9.*)
CSQ9.AMSQ
CSQ9.NONE
CSQ9.ZZZZ
CSQ9.** (G)

Is a user authorised to use this queue?

Use RLIST to tell you the profile used for checking

  • Check the Universal Access
  • Check to see if the userid in the list
  • Check the groups in the list and see if the userid is a member of the group.

Which profile gave what access to the queue

Use the RLIST MQQUEUE qmgr.queueName.

Who is authorised to this queue

Use the rlist command as described above. You may have to write a script to post process the data, and replace the group name with the member of the group. I used the Rexx interface IRRXUTIL and wrote about 100 lines of code to do this. Please contact me if you are interested in this.

Can I audit the list of people and their access to queues beginning with CP?

Not easily.

The command

SEARCH CLASS(MQQUEUE) FILTER(CSQ9.CP*)

gives ICH31005I NO ENTRIES MEET SEARCH CRITERIA

The command

SEARCH CLASS(MQQUEUE) FILTER(CSQ9.A*)

Gives one queue (CSQ9.AMSQ). It does not list the default CSQ9.** for any other queues

You would have to issue the MQ command to get a list of queues, the parse the list, and pass the queue name to the RLIST command, and collect the set of userids and groups. Finally, change any groups to the list of members of the group.

I used the Rexx interface IRRXUTIL and wrote about 100 lines of code to do this. Please contact me if you are interested in this.

Thoughts on planning for security on midrange.

As I was working on some other blog posts relating to security profiles on midrange MQ, I thought about the best way to set up MQ security. Below are some things you need to consider, outside of the well documented topics.

Give people the authority they need to do their job – but no more

One of the rules of security is to give people the authority they need to do their job – but no more. People may need access to one set of resources (eg queues) but do not need access to other sets of resources (other queues). This means you need more than one security profile – one for each. Having one big security group covering both sets of resources is a no-no.

Use groups to give permissions rather than to userids

It is better to give groups permissions, than to give individual’s userid permissions. To add a new member to the team you add their userid to the group(s) – rather than to all the queues. To remove someone’s access – you delete them from the group(s). These changes are easy to do and easy to reverse. If you did it at the userid level, there could be many records you would needs to process.

If you want to give a userid permission to do something, then other people are likely to want the same permission. Give permission to one group, and all userids in that group will get the access.

Removing authority

If you want to remove someone’s authority to do something

  • “Just remove” them them from the mqm group.
  • Use dmpmqaut .. -p userid -e to see which groups have the permission. Either
    • “Just remove” the permission from the group. This means other people in the group will also lose permission
    • or “just remove” the userid from the group, which may mean the userid may lose access to other facilities.

“Just remove” could be a lot of work, as you want to remove access to one queue, but retain their access to other queues. You may need to set up other groups to cover this.

Review your setup

You may want to review your groups, permissions, and userids within groups for your MQ estate and develop a scheme where you have a queue, and groups to define the permissions to the queue. If you have queues PAY_IN…. and PAY_OUT… you will need two AUTHREC records, one for each.

Generic profiles

If you have multiple generic profiles which apply to a queue, only the most specific profile will be used. Profile AA.BB.*.DD would be used in preference to AA.*.CC.DD

If a generic profile exists with the same name, but different groups, then all applicable instances of the most specific profile will be used. Profile AA.BB.*.DD group test, and profile AA.BB.*.DD group test2, would both be used if the userid is in both groups.

Separation of responsibilities

You may not want people who can administer queues, from processing messages on queues. Userids in the mqm group, or in the default group of the userid that created the queue, have all permissions for the queue. You may want to have a tool for deploying administrative changes, and use MQAdmin groups to control who can administer the object.

It took me a while to partially understand setmqaut and dspmqaut – and your security may be a mess.

On Linux, I was looking into MQ context security , where you can prevent a user from putting a fake information into the MQMD, and to my surprise it kept on working! It took me a while to work out what was going on.

Some of the things I learned include

  • If a userid is in the mqm group, it overrides any setmqaut statements. You may want to have a group called MQADMIN, and have your administrators in that group rather than the mqm group; so they can create or alter queues , but not put messages to sensitive queues.
  • Your overall access permission depends on the settings for the policy for the userid and any groups it is in. If one of your groups has permission, you have permission. If no group has permission, your userid gets no permission.
  • If you add or remove userids from groups, you have to use the runmqsc command refresh security to pick up the changes.
  • If you are using a userid in the mqm group to test your application – you may be testing with a different set of permissions to your target user.
  • It is better to define security profiles using groups rather than userids. If someone joins the team, you just put them in the group. If security is done using userids, you have to replicate all of the definitions of an existing user. Similarly if someone leaves the team.
  • The difference between group and principal (userid) is fuzzy. You define authority on a userid – but it appears as a group. Thanks to Joshm who pointed out that on Unix the qm.ini SecurityPolicy=user|group|UserExternal|default controls whether security is done with userids or groups. I need to explore this, but it explains some of my confusion
  • Some of the commands, for example dmpmqaut do not work as documented, but you can work round it.
  • setmqauth and dspmqauth parameters are read asis – with display authrec mixed case fields need to be surrounded with single quotes.
  • If you remove permission, then check, as you may get permission from more than one group.

Controlling the ability to change the context of a message.

This facility allows an application to specify the MQMD fields PutApplType, PutApplName, PutDate and ApplOriginData, rather than have them filled in by the queue manager. An MCA channel would do this when putting a message, to keep the original information. See origin context.

The key security parmeter is “setall” which allows these fields to be set.

Being a good tester, I displayed the permissions before changing them.

dspmqaut -m QMA -n CP0000 -t queue -p colinpaice

This says display the attributes for Queue CP0000 for user colinpaice.

I also defined authorities for another userid

setmqaut -m QMA -n CP0000 -t queue -p testuser +allmqi

I could not see how to use dspmqaut to display the authorities for all users of a queue, however you can use the runqmqsc DIS AUTHREC command, dmpmqaut and dmpmqcfg -m … -x authrec to dump out the permissions for users of an object.

The documentation is a little unclear – it talks about entity without actually saying what an entity was. I assumed the entity was the queue name, it actually means userid or group – but this is not strictly true.

runmqsc dis authrec displayed

dis authrec profile(CP0000)
     3 : dis authrec profile(CP0000)

AMQ8864I: Display authority record details.
   PROFILE(CP0000)                         ENTITY(mqm)
   ENTTYPE(GROUP)                          OBJTYPE(QUEUE) 
   AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,
   PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
   PROFILE(CP0000)                         ENTITY(testuser)
   ENTTYPE(GROUP)                          OBJTYPE(QUEUE)
   AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)
   ENTTYPE(GROUP)                          OBJTYPE(TOPIC)
   AUTHLIST(DSP)
...                        

This needs a bit of explaining.

Display information about a group’s access to a profile

AMQ8864I: Display authority record details.
   PROFILE(CP0000)                         ENTITY(mqm)
   ENTTYPE(GROUP)                          OBJTYPE(QUEUE)
   AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID) 

For Object type of Queue, there is a profile for object CP0000. The profile applies to group mqm. It has lots of access. That was pretty self explanatory.

Display information about a userid’s access to a profile

AMQ8864I: Display authority record details.
   PROFILE(CP0000)                         ENTITY(testuser)
   ENTTYPE(GROUP)                          OBJTYPE(QUEUE)
   AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

This is just like the first section, except I had defined it for principal (userid) testuser – not a group.

Using DIS AUTHREC gave

dis authrec profile(CP0000) objtype(queue) group('testuser')
AMQ8864I: Display authority record details.
PROFILE(CP0000) ENTITY(testuser)
ENTTYPE(GROUP) OBJTYPE(QUEUE)
AUTHLIST(BROWSE,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

dis authrec profile(CP0000) objtype(queue) principal('testuser')
4 : dis authrec profile(CP0000) objtype(queue) principal('testuser')
AMQ8459I: Not found.

However if I display information about the userid, it displays

uid=1002(testuser) gid=1004(testuser) groups=1004(testuser),1005(test)

so there is a group for the userid as well as the userid itself.

All in all – I found this a little confusing.

Other profiles.

AMQ8864I: Display authority record details.
   PROFILE(**)                             ENTITY(test)
   ENTTYPE(GROUP)                          OBJTYPE(TOPIC)
   AUTHLIST(DSP)

There is a generic profile ** for object type topic. If there was a topic CP0000 then userid/group test would have DSP access.

Displaying what you want

You can limit the object type displayed

dis authrec profile(CP0000) objtype(queue)

this just displayed the queue records for profile CP0000 (and so did not display the topic record above), so you can be selective.

You can use

dis authrec profile(CP0000) objtype(queue) group(‘testuser’)

or principal(‘userid’) to display records for just a group or userid.

However

dis authrec profile(‘CP*’) objtype(Queue)

gives

AMQ8459I: Not found.

Because there is no profile with name CP*. It does not display all profiles beginning with CP.

Using setmqaut

I wanted to test out context security, and did not want my application to be able to set certain fields in the MQMD.

If the userid had permissions, put and setall, my application was able to set the application name.

I turned off the permission with

setmqaut -m QMA -n CP0000 -p testuser -t queue -setall

and got rc=2035 MQRC_NOT_AUTHORIZED as expected.

Note: The userid also needs the “global level” switch

setmqaut -m QMA –t qmgr -p testuser +setall

enabled at the queue manager level. The application needs both +setall permissions to be able to set these context fields in the MQMD at put time.

This is not the end of the story…

With my normal administrator userid colinpaice, I used

setmqaut -m QMA -n CP0000 -t queue -p colinpaice -setall

to remove the setall permission; and this had no effect! I was still able to set the certain fields in the MQMD.

Hidden away in the documentation is a line

If a principal is a member of more than one user group, the principal effectively has the combined authorities of all those user groups

After I issued the setmqauth command, the dspmqauth command

dspmqaut -m QMA -n CP0000 -t queue -p colinpaice

gave

get
put
setall

which shows the userid still has setall access.

From the dis authrec command, it gave

2 : dis authrec profile(CP0000) objtype(QUEUE) 
AMQ8864I: Display authority record details.
PROFILE(CP0000) ENTITY(mqm)
ENTTYPE(GROUP) OBJTYPE(QUEUE)
AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

so the group mqm has SETALL access to the queue.

It gets more confusing. I used the command

setmqaut -m QMA -n CP0000 -t queue -p colinpaice -setall

to turn off setall permissions

dspmquth with -p colinpaice showed setall

dspmquth with -g colinpaice did not have setall!

This has me stumped for a few hours!

Being in the mqm group trumps the setmqaut definitions.

If a userid is in the mqm group, then it looks like there are no other checks. An implications of this is if your development team are in the mqm group, and they checking “error paths”, the MQ call will work, when it would fail for non group mqm users.

I used

sudo gpasswd –d colinpaice mqm

to remove my userid from the group. I had to use the runmqsc command refresh security command to get the queue manager to pick up the change.

Once I had done this then the setmqaut commands with +setall and -setall worked as expected. I used

sudo gpasswd –add colinpaice mqm

and the runmqsc refresh security command, for the setmqaut commands to be ignored.

The command

dspmqaut -m QMA -n CP0000 -t queue -p colinpaice

was accurate, in that when the userid was in the group, the command shows what access the userid has.

When the userid was not in the group, I remove access

setmqaut -m QMA -n CP0000 -t queue -p colinpaice -remove

the display command gave

Entity colinpaice has the following authorizations for object CP0000:
crt

When the userid was in group mqm the dspmqaut command gave

dspmqaut -m QMA -n CP0000 -t queue -p colinpaice
Entity colinpaice has the following authorizations for object CP0000:
get
browse
put
inq
set
crt
dlt
chg
dsp
passid
passall
setid
setall
clr

How easy is it to display security information on midrange.

If you thought it was easy, I would disagree with you.

While investigating a security problem (why my code worked, when it should have got a not authorised return code) I struggled to answer questions like

There are links to my solutions for some of the problems.

Morag contacted me and said:

I wrote a little blog post about using DISPLAY ENTAUTH – hope it helps. See here:

Morag

What commands are there?

There are four MQ commands providing similar functions

Definition of terms

Profile name

This can be the name of a queue, or a string which is used for comparisons and can include single * or double **. For example

  • PAYROLL for a particular queue,
  • PAYROLL.*.INPUT, which would apply to objects PAYROLL.A.INPUT, PAYROLL.B.INPUT, but not PAYROLL.A.B.INPUT, nor PAYROLL.INPUT.
  • PAYROLL.**.INPUT which would apply to PAYROLL.A.INPUT, PAYROLL.B.INPUT, PAYROLL.A.B.INPUT, but not PAYROLL.INPUT.
  • ** used as a fall back if there are no other definitions.

Example profiles

  • setmqaut -m QMA -n PAYROLL -t queue -g testuser +put
  • setmqaut -m QMA -n ‘PAYROLL.*.INPUT’ -t queue -g testuser +put
  • setmqaut -m QMA -n ‘PAYROLL.**.INPUT’ -t queue -g testuser +put
  • setmqaut -m QMA -n ‘**’ -t queue -g MQAdmin +chg +dlt

On Unix you need the single quotes around any profile which includes *, to prevent the shell replacing it with a list of file names.

You need to be careful about quoting strings with mixed case data. dspmqaut and setmqaut do not need quoting. For runmqsc display|set authrec the mixed case fields need quoting.

A profile is created

  • by the setmqaut command,
  • the runmqsc set authrec command,
  • when the object is created. In which case, the queue manager creates an entry for the mqm group, and for the principal group of the user that created the object. Userid ibmsys1 created a queue, and there were entries for group mqm, and group zpdt. These definitions have full access. (The userid that created a queue can put and get messages from the queue – and you cannot separate administration of the queue, and usage of the queue.)

Userid and group

Unix has the concept of User Private Groups (UPG). A userid can have a group with the same name as the userid, which makes the security just a touch more confusing.

You can use the id command to list which groups a userid is in

id ibmsys1
uid=1001(ibmsys1) gid=1006(zpdt) groups=1006(zpdt),27(sudo),1007(MQAdmin)

Userid ibmsys1 with numeric id 1001 is a member of 3 groups: zpdt, sudo and MQAdmin.

You can specify the authority to be given to a userid – but it looks like the userid is not used, but the default group is used! See Explicit giving a userid access – does not give the userid access. This could just be a Linux feature.

Displaying information.

dspmqaut -m QMA -n ‘PAYROLL.**.INPUT’ -t queue -g testuser

This lists the information for the group testuser and the profile PAYROLL.**.INPUT’. It reports “put” access.

What access does the group ibmsys1 have?

dspmqaut -m QMA -n ‘PAYROLL’ -t queue -g ibmsys1

Entity ibmsys1 has the following authorizations for object PAYROLL:

 

 

Shows the group has no access defined.

What access does the userid (principal) ibmsys1 have?

dspmqaut -m QMA -n ‘PAYROLL’ -t queue -p ibmsys1

Entity ibmsys1 has the following authorizations for object PAYROLL:
dlt
chg

This is because userid ibmsys1 has MQAdmin as one of its group.

Adding -e option to the dmpmqaut command (Display all profiles used to calculate the cumulative authority that the entity has to the object specified in -n Profile) shows which profiles will be used.

dmpmqaut -m QMA -n ‘PAYROLL’ -t queue -p ibmsys1 -e
profile: **
object type: queue
entity: MQAdmin
entity type: group
authority: dlt chg

We can see that the profile used to determine the access permissions came from the “**” profile statement for group(entity) MQAdmin.

Explicit giving a userid access – does not give the userid access.

On my Linux Ubuntu machine, using the following command to explicitly give a userid access; gives it to the userid’s principal group.

setmqaut -m QMA -n PAYROLL -t queue -p ibmsys1 +put

dmpmqaut -m QMA -n PAYROLL -t queue -p ibmsys1 -e
profile: PAYROLL
object type: queue
entity: zpdt
entity type: group
authority: put

profile: **
object type: queue
entity: MQAdmin
entity type: group
authority: dlt chg

This shows the group zpdt got the access – not the userid.

For this userid we can see that there was a specific profile “PAYROLL” for the zpdt group giving put permission, and the generic profile for the MQAdmin group giving inquire and display permissions.

The display authrec command

If you use dis authrec profile(PAYROLL) it will display all possible profiles including the profiles for queues, process etc. Ensure all mixed case values are surrounded by single quotes.

dis authrec profile(PAYROLL) objtype(queue)

gave

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(colinpaice)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(mqm)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(BROWSE,CHG,CLR,DLT,DSP,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

AMQ8864I: Display authority record details.
  PROFILE(PAYROLL) ENTITY(testuser)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
 AUTHLIST(PUT)

AMQ8864I: Display authority record details.
  PROFILE(**) ENTITY(test) ENTTYPE(GROUP) OBJTYPE(QUEUE) AUTHLIST(DSP,INQ) 

AMQ8864I: Display authority   record details. 
  PROFILE(**) ENTITY(MQAdmin)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(CHG,DLT)

but

dis authrec profile(PAYROLL) objtype(queue) group(MQAdmin)

gives not found however, using single quotes,

dis authrec profile(PAYROLL) objtype(queue) group(‘MQAdmin’)

gives

AMQ8864I: Display authority record details.
  PROFILE(**) ENTITY(MQAdmin)
  ENTTYPE(GROUP) OBJTYPE(QUEUE)
  AUTHLIST(CHG,DLT)

List all objects

You can use

runmqsc dis authrec type(queue)

to display all queue records with (up to three lines per entry) see above for an example.

You can use the dmpmqaut -l option Dump only the profile name and type. Use this option to generate a terse list of all defined profile names and types.

dmpmqaut -m QMA -l -t queue

Which gives output like

profile: PAYROLL, object type: queue
profile: PAYROLL.*.INPUT, object type: queue
profile: PAYROLL.**.INPUT, object type: queue

Unfortunately this is output to stderr, so you cannot immediately pipe this into grep or other commands.

You can use

dmpmqaut -m QMA -l -t queue 3>&1- 1>&2 2>&3|grep PAYROLL

The dmpmqaut command says you can use generics and wild cards in the name described Using OAM generic profiles on AIX®, Linux®, and Windows systems. I could not get this to work, for example using PAYROLL.**

The runmqsc DIS AUTHREC command says profile can have The name of the object or generic profile for which to display the authority records.

I think the documentation means “the string used in the profile definition, not a generic search argument.”


For example I wanted to use a qualifier ‘*’, but the command

dmpmqaut -m QMA -n ‘CP.*’

gives

No matching authority records.

runmqsc dis entauth – display the authorizations an entity has to a specified object

This very useful command was pointed out to me by Morag. It has been in MQ since MQ 7.5.

It allows you to ask what permissions a userid or group has to a specific resource. For example

A typical display

dis entauth principal(‘testuser’) objtype(queue) objname(CP0000)
AMQ8866I: Display entity authority details.
OBJNAME(CP0000) ENTITY(testuser)
ENTTYPE(PRINCIPAL) OBJTYPE(QUEUE)
AUTHLIST(BROWSE,CRT,GET,INQ,PUT,PASSALL,PASSID,SET,SETALL,SETID)

Queue does not exist

dis entauth principal(‘testuser’) objtype(queue) objname(CP000ZZZZZZ)
AMQ8147E: IBM MQ object CP000ZZZZZZ not found.

Silly me – I missed the quotes off from around the userid


dis entauth principal(testuser) objtype(queue) objname(CP0000)
3 : dis entauth principal(testuser) objtype(queue) objname(CP0000)
AMQ8871E: Entity, principal or group not known.

Dump mq configuration

You can use the dmpmqcfg command to dump out the authrec records.

dmpmqcfg -m QMA -x authrec -o 1line

gives output like (where there is one line for each SET… statement).

SET AUTHREC PROFILE(‘CP.*.00’) GROUP(‘colinpaice’) OBJTYPE(QUEUE) AUTHADD(GET,PUT)
SET AUTHREC PROFILE(‘PAYROLL.**.INPUT’) GROUP(‘testuser’) OBJTYPE(QUEUE) AUTHADD(PUT)
SET AUTHREC PROFILE(‘@class’) GROUP(‘colinpaice’) OBJTYPE(QUEUE) AUTHADD(CRT)

The -o 1line option creates one line of output for each record. You can use platform tools to process the data, for example use grep to filter lines, and parse the parts between ‘ ‘. See an example python script at the bottom.

The documentation says -n [*|ObjectName] Filter the definitions produced by object or profile name, the object/profile name can contain a single asterisk. But it didnt work for me.

My basic queries

Is this user authorised to use this queue?

You can use

dspmqaut -m QMA -n PAYROLL.A -t queue -p ibmsys1

If the queue does not exist you will get AMQ7085E: Object PAYROLL.A, type queue not found.

Or you can use the runmqsc command

dis entauth principal(‘testuser’) objtype(queue) objname(CP0000)

What security profiles are involved with this user and that queue.

Or which profiles gave a user access to the queue?

Display the userids id and groups. If userid is in group mqm, it will get all access regardless of the setmqaut settings.

Display all the profiles involved with a userid -e … or group -g …

dmpmqaut -m QMA -n PAYROLL.A -t queue -e

Display all the permission a user has with a userid -p … or group -g …

dspmqaut -m QMA -n ‘CP.ZZ.00’ -t queue -p testuser
Entity testuser has the following authorizations for object CP.ZZ.00:
inq
crt
dsp

Display the profiles which determine the permissions for a user

dmpmqaut -m QMA -n ‘CP.ZZ.00’ -t queue -e -p testuser
profile: **
object type: queue
entity: test
entity type: group
authority: inq dsp

profile: @class
object type: queue
entity: test
entity type: group
authority: crt

The inq and dsp permissions came from the “**” profile for the test group.

Who is authorised to this queue?

This is a tricky one. I could find two hard ways of

dmpmqaut -m QMA -n ‘CP.ZZ.00’ -t queue 2> file

dmpmqcfg -m QMA -x authrec -o 1line > file

Then post-processing the output to extract group name and permissions, then merging the records.

For example to see the members of a group.

getent group |grep mqm

gives
mqm:x:1003:colinpaice,ibmsys1

I want to audit the access to the queues beginning with CP.

Another tricky one.

  • dmpmqcfg -m QMA -n ‘CP*’ -t queue -o 1line -x object > queuelist
  • This gives a list of all queues starting with CP. Process this list to select each queue in turn
  • dmpmqaut -m QMA -n $queuename -t queue -e > accesslist
  • This accesslist file has the profile name, the group and the permissions.
  • Take each group name, and extract the members of the group.
  • For each member of the list use
  • dspmqaut -m QMA -n $queue -t queue -p $userid
  • Take this list extract the permissions, and produce a pretty report.

If there is a better way I would love to hear it.

A partial solution is to take the output of

dmpmqaut -m QMA -l -t queue 2>list

dmpmqcfg -m QMA -x authrec -o 1line > file

Then use tools like grep to extract the records of interest, then use other tools to issue dmpmqaut -m QMA -n ‘….’ -t queue, capture the output and process it.

But this means you have to interpret generic profiles like ‘CP.**.INPUT’ yourself.

Sample code for post processing dmpmqcfg output

Below is a small Python program that takes the output from dmpmqcfg and processes it to replace groups with users in the groups, and gives one line per userid,permission,profile, source_group

#
# Copyright (c) 2020 Stromness Software Solutions.
# 
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Eclipse Public License v1.0
# which accompanies this distribution, and is available at
# http://www.eclipse.org/legal/epl-v10.html
# 
# * Contributors:
# *   Colin Paice - Initial Contribution
# 
from sys import stdin
import grp

for line in stdin:
  sline= line.strip()
  s = sline.split(' ')
  if s[0] != "SET":
     continue
  p = s[2][9:-2] # get the profile
  g = s[3][7:-2] # get the group 
  try:
    ginfo = grp.getgrnam(g)
    ids = ginfo.gr_mem 
    if len(ids) == 0:  # group can be empty
       ids = [g]  
  except KeyError:
    ids = [g] # group not found - must be userid
  t = s[4][8:-1] #type

  a = s[5][8:-1]  # get the authorities
  aa = a.split(",")
# Output userid authority profile type group_it_came_from
  for aaa in aa:
    for i in ids:
       print(i,aaa,p,t.g) 

Invoke it as

cat data | python3 ~/python/sec.py |sort -k1,3|uniq |grep -v mqm 

grep -v mqm removes all accesses due to group mqm membership

Example output

colinpaice SET SYSTEM.SELECTION.VALIDATION.QUEUE QUEUE 
colinpaice SET TEMP QUEUE colinpaice
colinpaice SET XQMB QUEUE colinpaice
colinpaice SUB SYSTEM.ADMIN.TOPIC TOPIC colinpaice
testuser PUT COLIN_* QUEUE test
testuser PUT CP0000 QUEUE testuser