You need a SAF Angel for Liberty Web Server on z/OS

I spent a week trying to get MQWEB on z/OS to work using digital certificates using RACF as a repository, and had lots of frustrations, and some successes. I found that to get security working you need an Angel process.

Some parts are well document, some other parts are not, so I’ll try to fill some of the gaps.

How does MQWEB do security?

The Liberty server has a couple of ways of Authentication and Authorisation.

  • The basic repository where you define users, passwords and their role (MQWebAdmin, MQWebAdminRO or MQWebUser) within the xml configuration file.   This provides the most basic levels of protection – for example the MQ administrator has to create and maintain the passwords.  It is easy to set up, but not very secure.
  • Using the System Authorization Facility (SAF) interface.  This is an interface which applications can use to get to the security back end.  There is a choice of back-ends, for example RACF from IBM, and Top Secret from CA.  This can be configured to be very secure, but has more administrator work to set up.

How does SAF work?

There are RACF profiles which control access to restricted facilities.  If you have access to the profile you can perform the function.  For example when you logon to the a server, the server thread has to run as your userid, to allow your userid access to resources.  You set up a profile, and explicitly give the server access to the profile permitting the server to “run on behalf of another userid”.

There is a C function on z/OS that allows my userid to check to see if your userid is authorised to a resource.    If my userid has access to the correct profile, my userid can get the information about your userid.

Some “system-like” functions assume the requester is authorised, and bypass some of the checks.  For example the RACROUTE FASTAUTH check request.   These have another level of control in that they need to be in a restricted, Supervisor (kernel) state, to be able to issue the requests.

What is the angel process?

Instead of giving Java programs access to this restricted Supervisor mode, there is an Angel address space which can execute the restricted requests, and the Java program can call the code running in the Angel address space.  (Under the covers there is a Program Call to execute the code, in the same way that MQ and DB2 requests do.)   Of course, there is a RACF profile to control which address spaces can access the Angel, and other profiles to configure what restricted functions the Angel process can execute.

If the Angel process was not running, I could not logon to MQWEB.   There were messages saying my userid did not have access to MQWebAdmin, MQWebAdminRO or MQWebUser.   The checking is done within the LIberty and is very simple, and restrictive – and does not work because it looks for group names  MQWebAdmin etc… but MQWebAdmin is too long for a z/OS group name which can have up to 8 characters in it.

Setting up the Angel process.

  • The Angel process can be shared by all of the web servers on the LPAR.
  • In a highly available environment you need more than one Angel on an LPAR.
  • MQ ships Angel code, but it may be at a different level to other Liberty servers’s Angel code.  You should run on the latest level of code, but I expect if they are reasonably current( within 1 year) they should all work ok.
  • This gives a good overview of the Angel.
  • This also gives a good overview and instructions
  • See here for configuration instructions.
  • I think the only Angel interface used by MQWEB  is SAFCRED  (SAF Credentials) and PRODMGR, but it does no harm having access to services you do not use.
  • You cannot stop an Angel process if it has servers “connected” to it.
  • If you cancel the Angel, the Web Server stops working,  it may get  CEE3250C The system or user abend S0D6 R=00000027 was issued  in the message.log file,  and abend.
  • For availability you may want two queue managers on an LPAR, so you need two Angel process.
  • There is no configuration you do to an Angel, so the only reason why you may want to shut down the Angel during normal running is to apply fixes.
  • You need to ensure the Angel process is started before the Web Server is started, as the Web Server only connects at start up, see below.
  • Each Angel running in an LPAR needs a unique name.  You can have a default, unnamed Angel, or give your Angel a name.

I called my angel process ANGEL.   From WAS and z/OS Connect they talk about their Angels with names like BBZANGL, BAQZANGL, which are not very memorable, hard to remember, and easy to mistype.

You start an angel process (once the JCL has been configured) using a command like

  • S ANGEL
  • S ANGEL,NAME=ANGEL1
  • S ANGEL.ANGEL1,NAME=ANGEL1
  • S ANGEL.ANGEL2,NAME=ANGEL2

The first command starts the default, unnamed, Angel.

Using the S ANGEL.name command is useful if you have more than one angel as it means you can use the STOP name command to that particular instance.

Configure your web server to end if the angel is not available

You can configure your Liberty server to end if the Angel process is not running.    Without it, the Web Server would start, and requests to use it would fail, because the unauthorised interface would be used. Using this option means you find out Sunday night and not when your business starts at 0900 on Monday morning!

You configure it by editing the server jvm.options file by adding

-Dcom.ibm.ws.zos.core.angelRequired=true

If the Angel is not available, this statement prevents the Web Server from starting

You can specify the angel name using

-Dcom.ibm.ws.zos.core.angelName=MYANGEL

If you are using the default angle name, comment this out

# -Dcom.ibm.ws.zos.core.angelName=

When your Web Server starts you should get messages in the message.log  like

CWWKB0103I: Authorized service group SAFCRED is available.

Tracing violations.

You can start an Angel using

S ANGEL,SAFLOG=YES

I gave my MWEB userid no access to SAFCRED, and restarted MQWEB.  With SAFLOG=YES, I got the following on the joblog

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
BBG.AUTHMOD.BBGZSAFM.SAFCRED CL(SERVER )
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )

and in the MQWEB directory/logs/message.log file

CWWKB0104I: Authorized service group SAFCRED is not available.

With SAFLOG=NO, there was no message on the joblog, but the same message in  MQWEB directory/log/message.log file

CWWKB0104I: Authorized service group SAFCRED is not available.

From this we can see that you should always specify SAFLOG=YES.

Angel commands

You can use

F ANGEL,DISPLAY,SERVERS

This gives a message like

CWWKB0052I ACTIVE SERVER ASID 3f JOBNAME CSQ9WEB

You can use

F ANGEL,TRACE=Y

This writes a trace to //STDOUT in the Angel job.

The output is like

Trace: 2020/07/25 15:38:07.877974 t=8D5E88 key=S2 (04002008)
Description: write_to_operator_location, entry
message_p: CWWKB0057I WEBSPHERE FOR Z/OS ANGEL PROCESS ENDED NORMALLY

Which is not very helpful, as it traces what the Angel process is doing – rather than the Web Servers using it.

How to use RACF callable services from C

Trying to use the RACF callable services was like trying to find treasure with an incomplete map.  I found it hard to create a C program to query a repository for ID information.   This post is mainly about using the RACF callable services.

I was trying to understand the mapping of a digital certificate to a z/OS userid, but with little success.  I found a RACF callable service which appeared to do what I wanted – but it did not give the answers – because,  like many treasure maps, I was looking in the wrong place.

RACF has two repositories for mapping identities to userid.

  • RACDCERT MAP which was the original way of mapping names.  As far as I can tell, the only way of getting the certificate to userid mapping programmatically, is to use the certificate to logon, and then find the userid!   This is used by Liberty Web Server.
  • RACMAP MAP which is part of Enterprise wide identification.   It maps identity strings, as you may get from LDAP,  to a userid. You can use the r_usermap callable service to get this information.

It took me some time to realise that these are different entities, and explains why there was no documentation on getting Liberty to work with RACMAP to handle certificates.  I found out RACMAP does not map certificate, after I got my program working.

The r_usermap service documentation is accurate – but incomplete, so I’ll document some of the things I learned in getting this to work.

The callable service to extract the userid  from identity information is documented here.  In essence you call the assembler routine r_usermap or IRRSIM00.

Building it

When you compile it, you need to provide the stub IRRSIM00 at bind time.  I used JCL

//S1 JCLLIB ORDER=CBC.SCCNPRC 
//DOCLG EXEC PROC=EDCCBG,INFILE='ADCD.C.SOURCE(C)', 
// CPARM='OPTF(DD:COPTS)' 
//COMPILE.COPTS DD * 
LIST,SSCOMM,SOURCE,LANGLVL(EXTENDED) 
TEST 
/* 
//COMPILE.SYSIN DD * 
  Source program goes here
//BIND.CSS DD DISP=SHR,DSN=SYS1.CSSLIB 
//BIND.SYSIN DD * 
INCLUDE CSS(IRRSIM00) 
/*

You need code like

#pragma linkage(IRRSIM00, OS) 
int main(){...
...
char * workarea ; 
workarea = (char *) malloc(1024)   ; 
long ALET1= 0; 
...
long SAF_RC,RACF_RC,RACF_RS; 
...
rc=  IRRSIM00(workarea, // WORKAREA 
             &ALET1  , // ALET 
             &SAF_RC, // SAF RC 
...

Some fields are in UTF-8.

To covert from EBCDIC to UTF-8, (it looks like ASCII )  I used

cd = iconv_open("UTF-8", "IBM-1047"); 
...
struct 
{ 
  short length ; // length of string following or 0 if ommitted 
  char value[248]; 
} DN; 
char * sDN= "CN=COLIN.C=GB"; 
size_t llinput= strlen(sDN); 
size_t lloutput= sizeof(DN.value); 
char * pOutValue= &DN.value[0]; 
rc = iconv(cd,        // from  iconv_open
           &sDN,      // input string
           &llinput,  // length of input 
           &pOutValue,// output 
           &lloutput);// length of output 
if (rc == -1) // problem
{ 
  perror("iconv"); 
  exit(99); 
} 
DN.length  =sizeof(DN.value) - ll2; // calculate true length

What access do I need?

You need

permit IRR.RUSERMAP class(FACILITY) access(READ) ID(....)
SETROPTS RACLIST(facility ) REFRESH

Output

Once I had got the program to compile and bind, and got the authorisation it worked great.

It only works with the RACFMAP …  command, not the RACFDCERT command, obvious now I know!  To get the information from the RACDCERT MAP, you need to use initACEE.

What no one tells you about setting up your RACF groups – and how to do it for MQ.

Introduction

The RACF documentation has a lot of excellent reference materials describing the syntax of the commands, but I could not find much useful information on how to set up RACF specifically for products like MQ, CICS, Liberty etc.

It is bit like saying programming has the following commands, load, store, branch; but fails to tell you that you can do wonderful things like draw Mandelbrot pictures using these instructions.

You need to plan your group structure before you try to implement security, as it is hard to change once it is in place.

The big picture

You can set up a hierarchy of groups so the site RACF person can set up a group called MQ, and give the MQ team manager authority to this group.

The manager can

  • define groups within it
  • connect users to the group
  • give other people authority to manage the group.

We can set up the following group structure

  • MQM
    • MQOPS – for the MQ operators
      • MQOPSR for operators who are allow to issue only Read (display) commands
      • MQOPSW for operators who can issue all command, display and update
    • MQADMS – for the MQ administrators
      • MQADMR – for MQ administrators who can only use display commands
      • MQADMW – for MQ administrators who can use all commands
    • MQWEB….

You should place an operator’s userid in only one group MQOPSR or MQOPSW as these are used to control access.  MQM, MQOPS, MQADMS, MQWEB are just used for administration.

You permit groups MQOPSR and MQOPSW to issue a display command, but only permit group MQOPSW to issue the SET command.

Setting up groups to make it easy to administer

A group needs an owner which administers the group.  The owner can be a userid or a group.

A group has been set up called MQM, and my manager has been made the owner of it.

My manager has connected my userid PAICE to the MQM group with group special.

CONNECT PAICE GROUP(MQM) SPECIAL

I can define a new group MQOPS for example

ADDGROUP MQOPS SUPGROUP(MQM) OWNER(MQM) DATA(‘MQ operators’)

The SUPGROUP says it is part of the hierarchy under MQM.  I can create the group under MQM because I am authorised,  If I try to create a group with SUPGROUP(SYS1) this will fail because I am not authorised to SYS1.

The OWNER(MQM) says people in the group MQM with group special can administer this new group.

Because my userid (PAICE) has group special for MQM, I can now connect users to the new group, for example

CONNECT ADCDB GROUP(MQMD ) AUTHORITY(USE ).

I can create another group under MQMD called MQMX, and connect a userid to it.

ADDGROUP MQMX  SUPGROUP(MQMD) OWNER(MQMD) DATA(‘MQ Bottom group’)
CONNECT ADCDE GROUP(MQMX ) AUTHORITY(USE )

My userid PAICE can administer this because of the OWNER() inheritance up to GROUP(MQM)

If I list the groups I get

LISTGRP MQM 
INFORMATION FOR GROUP MQM 
    SUPERIOR GROUP=SYS1 OWNER=IBMUSER 
    SUBGROUP(S)= MQM2 MQMD  
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS= 
       PAICE    JOIN        000000              NONE 
         CONNECT ATTRIBUTES=SPECIAL 

LISTGRP MQMD 
INFORMATION FOR GROUP MQMD 
    SUPERIOR GROUP=MQM OWNER=MQM 
       SUBGROUP(S)= MQMX 
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS=  
       ADCDB USE 000000 NONE CONNECT ATTRIBUTES=NONE
 
LISTGRP MQMX 
    INFORMATION FOR GROUP MQMX 
    SUPERIOR GROUP=MQMD OWNER=MQMD 
    NO SUBGROUPS 
    USER(S)= ACCESS= ACCESS COUNT= UNIVERSAL ACCESS= 
       ADCDE     USE        000000              NONE 
         CONNECT ATTRIBUTES=NONE

All SUPGROUP() does is to define the hierarchy as we can see from the LISTGRP.    We can display the groups  and draw up a picture of the hierarchy.   You can use the LISTGRP command repeatedly,  or use the DSMON program(EXEC PGM=ICHDSM00) and use option
USEROPT RACGRP to get a picture like

 LEVEL GROUP 
1 SYS1 (IBMUSER ) 
2 | MQM (IBMUSER ) 
3 | | MQMD 
4 | | | MQMX 
3 | | MQM2 (IBMUSER )

Using OWNER(group) instead of OWNER(userid)

  • If you have OWNER(groupname) it is easy to administer the groups.  When someone joins or leaves the department, you add or remove the userid from groupname.  One change.
  • If you have OWNER(userid), then you have to explicitly connect the userid to each group with group special.  When there is a new person you have to add the userid to each group individually.  When someone leaves the team you have to remove the persons userid from all of the groups. This could be a lot of work.

Delegation.

You could define an operator MQOP1 and give the userid group-special for group MQOPS.   This userid (MQOP1) can be used to add or remove userids in the MQOPSR and MQOPSW groups.

Looking at the MQOPS groups we could have groups and connected userids

  • MQM with MQ security userids PAICE, BOB having group-special
    • MQOPS with the operations manager and deputy MQOP1, MQOP2 having group special
      • MQOPSR with STUDENT1, STUDENT2 who are only allowed to issue display commands
      • MQOPSW with PAICE, TONYH, CHARLIE
    • MQADMS….

and similarly for the MQ administration eam.

Userid PAICE can connect userids to all groups.  MOP1 can only connect userids to the MQOPSR and MQOPSW, and not connect to the MQ ADMIN groups.

You use groups MQOPSR and MQOPSW for accessing resources. Groups MQM and MQOPS have no authority to access a resource, they are just to make the administration easier.

You may also want to consider having a group for application development.  The group called PAYRDEVT is under MQM, is owned by the manager of the payroll development team.

When the annual userid validation check is done, the development manager does the checks, and tells the security department it has been done.

Permissions

There is no inheritance of permissions.  If a userid needs functions available to groups MQMD and MQMX, the user needs to be connected to both groups.

You only connect userids to groups, you cannot have groups within groups.  There may be many groups of userids which are allowed to issue an MQ display command, but only one group who can issue the SET command.

 

Suggested MQ groups

You need to consider

  • production and test environments
  • resources shared by queue managers, queue managers with the same configurations in a sysplex which can share definitions
  • queue managers as part of a Queue Sharing Group
  • queue manages that need isolation and so may have common operations groups, but different administration and programming groups.

You might define

  • Group MQPA for the queue manager super group. (MQ, Production system, A)
  • Groups for MQWEB. The Web server roles are described here.
  • Groups for controlling MQ, operations and administrations, read only or update
  • Groups for who can connect via batch, CICS etc
  • Groups for application usage, who can use which queues

Groups for MQWEB

For MQWEB the MQ documentation describes 4 roles: MQWebAdmin, MQWebAdminRO, MQWebUser, MFTWebAdmin; and there is console and REST access.

Each role should have its own group.  The requests from “Admin” and “Read Only” run with the userid of the MQWEB started task.   The request from “User” run with the signed on user’s authority.

You might set up groups

  • MQPAWCO MQPA – MQWebAdminRO Console Read Only.
  • MQPAWCU MQPA – MQWebUser  Console User only.  The request operates under the signed on userid authority.
  • MQPAWCA MQPA – MQWebAdmin Console Admin.
  • MQPAWRO MQPA – MQWebAdminRO REST Read Only.
  • MQPAWRU MQPA – MQWebUser  REST User only.   The request operates under the signed on userid authority.
  • MQPAWRA MQPA – MQWebAdmin REST Admin Only.
  • MQPAWFA MQPA – MFTWebAdmin MFT REST Admin. 
  • MQPAWFO MQPA-  MFTWebAdmin MFT REST Read Only.

I would expect most people to be in

  • MQPAWCU MQPA – MQWebUser  Console User only.  The request operates under the signed on userid authority.
  • MQPAWRU MQPA – MQWebUser  REST User only.   The request operates under the signed on userid authority.

so you can control who does what, and get reports on any violations etc.  If people use the MQWEB ADMIN you do not know who tried to issue a command.

Groups for operations

The operations team may be managing multiple queue managers, so you may need groups

  • PMQOPS for Production
    • PMQOPSR
    • PMQOPSW
  • TMQOPS for Test
    • PMQOPSR
    • PMQOPSW

If some operators are permitted to manage only a subset of the queue managers you will need a group structure that can handle this, so have a special group XMQOPS for this.

  • XMQOPS for  the special queue manager
    • XMQOPSR
    • XMQOPSW

Groups for administration.

This will be similar to operations.

Groups for end users.

This is for people running work using MQ.

Usually there are checks to make sure a userid can connect to the queue manager, using the MQCONN resource.  Some customers have a loose security set up, and rely on the CICS to check to see if the userid is allowed to use a CICS transaction, rather than if the userid is allowed to access a queue.

Linux where’s my window? Use the keyboard to bring favourite window to the front

Using z/OS from my Linux laptop, I use X3270 to define the 3270 screens.  I use multiple screens ( master console, ISPF with main userid, ISPF with test userid).  It can be hard to find the right session.  I also try to use the keyboard rather than then mouse – how do I do it?
There is a good program wmctrl which can help you manage your all you windows, but I use it mainly for x3270 sessions.

For example wmctrl -a colin@ brings a window with “colin@” in the title, and gives it focus.  You then associate this with a keyboard action and the problem is solved.

wmctrl can do many things including

  • switch desktops
  • list desktops and windows managed by wmctrl
  • hide windows or make them full screen

To associate a keyboard action and a command,

  • display the settings (for example from the down arrow at top of the screen)
  • Keyboard – this lists your keyboard shortcuts.
  • go to the bottom of the list and click on +
  • enter
    • name:my name
    • command:wmctrl -a colin@
  • and click on Set Shortcut…
  • a small window pops up – type the key combination for example Ctrl 1

I have Ctrl+1 for the master console, Ctrl+2 for my main ISPF session, and Ctrl+3 for my “end user” ISPF session, and Ctrl+4 for Firefox

You can create an x3270 session using a command like

x3270 -model 5 colin@localhost:3270

which creates a session to port 3270 on my localhost, and puts -model 5 colin@localhost:3270 in the title bar. You could create another session x3270 -model 5 super@localhost:3270 so so use “super@” to identify it in the wmctrl command.

 

x3270 – where’s the money key?

On z/OS the default escape character for OMVS is the cent symbol ¢ (Unicode 00a2) which I do not have on my keyboard.

I set up in ~/.x3270pro

! Use the 'mine' keymap, defined below
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
   <Key>Escape: Clear()\n\
   <Key>End: FieldEnd()\n\
   Ctrl<Key>Delete: EraseEOF()\n\
   Ctrl<Key>Right: NextWord()\n\
   Ctrl<Key>Left: PreviousWord()\n\
   Ctrl<Key>Up: Home()\n\
   <Key>Control_L: Reset()\n\
   <Key>Control_R: Reset()\n\
   <Key>Prior: PF(7)\n\
   <Key>Next: PF(8)\n\
   <Btn3Down>: PA(1)\n\
   Ctrl<Key>backslash: Key(U+00a2)\n\
   Alt<KeyPress>4: Key(U+00a2)

So now I can use Alt 4 or Ctrl \ to enter the value.

 

Making x3270 green screens blue or red, or yellow with green bits.

I have several ISPF screens on z/OS, and I found it confusing knowing which one I was on.   I solved it  by making the screen with the all powerful userid displayed in red, and the “normal” screen in multi colours, and the screen with no authority in green.

With x3270 you can use Options -> Colour Scheme -> Default 3279 |Green, to use a predefined colour scheme.

I created my own colour scheme by editing  ~./x3270pro and adding

! define the list of colour schemes
x3270.schemeList: Default 3279: default\n\
    Bright: bright\n\
    Reverse: reverse\n\
    Green Screen: greenScreen\n\
    Colin: colin 
! now define my list
x3270.colorScheme.colin: \
    black deepSkyBlue pink red \
    green turquoise yellow white \
    black blue3 orange purple \
    paleGreen paleTurquoise2 grey white \
    white black dimGray \
    4 2 1 15

 

This gave a good description, and a good list of options is  here.

You can now use the command to start a session with this scheme.

x3270 -model 5 -scheme colin tso@localhost:3270

Playing twister with Liberty and falling over

Twister is a game where you have you put your left foot here, your left hand there, your right foot here, and in trying to put your right hand over there you fall over.  This is how I felt when I was trying to understand the SSL definitions in MQWEB.  In the end I printed off the definitions and used coloured pens to mark the relevant data.

Let’s start with the easy bit.

  • When mqweb starts, it reads configuration information from a file server.xml
  • This includes two files the “IBM” stuff in  /usr/lpp/mqm/V9R1M1/web/mq/etc/mqweb.xml and the user stuff in /u/mqweb/servers/mqweb/mqwebusr.xml .
  • SSL parameters are defined with and <ssl… id=”thisSSLConfig”   keyStoreRef=”defaultKeyStore” .. /> tag.    This points to the keystore to use.
  • The keystore has <keyStore id=”defaultKeyStore”  ….>.   There is a simple link from SSL to the keystore.   You could have multiple keystores,  if so you just change the  keyStoreRef= to point to a different one.
  • You can have more than one <ssl…/> definition.  You might have one, and there is one in the “IBM stuff”, so you need <sslDefault sslRef=”thisSSLConfig”/>  to point to the ssl statement to use.

That should all be clear, and make sense.   A bit like saying you have a right foot, a left foot and two hands.

The zos_saf_registry.xml used when you want to use the SAF interface on z/OS has some SSL definitions.  I was trying to understand them.   This one here(put a finger on it) points to that one, (put a finger on it), which points to this other one (put a finger on it), which has an end-comment.  Whoops that didn’t work.    As I said a bit like playing Twister.

<sslDefault sslRef=”mqDefaultSSLConfig”/> in the user mqwebuser.xml  points to content in the “IBM stuff”.  By the various levels of indirection this points to <keyStore id=”defaultKeyStore” location=”key.jks” type=”JKS” password=”password”/> .  This keystore has a self signed certificate provided by IBM. If you find your browser complains about using a self signed certificate, this may well be the cause.

In the zos_saf_registry.xml are commented statements

  • <keyStore id=”defaultKeyStore” location=”safkeyring://userId/keyring” …/>
  • <ssl id=”thisSSLConfig”  keyStoreRef=”defaultKeyStore” …/>
  • <sslDefault sslRef=”thisSSLConfig”/>

To me these have been defined upside down, sslDefault should come first.

As these are after the sslDefault sslRef=”mqDefaultSSLConfig statment, if you uncomment them, they will be picked up and the “IBM stuff” will not be processed.

You can uncomment these statements and use them to add your definitions.

My definitions are

<sslDefault sslRef="defaultSSLConfig"/> 
<ssl id="defaultSSLConfig" keyStoreRef="racfKeyStore" 
   sslProtocol="TLSv1.2" 
   clientAuthenticationSupported="true" 
   clientAuthentication="true" 
   serverKeyAlias="LABELMQWEBHSCEKE"/> 

<keyStore filebased="false" id="racfKeyStore" 
   location="safkeyring://START1/MQRING" 
   password="password" 
   readOnly="true" 
   type="JCERACFKS"/> 

<webAppSecurity allowFailOverToBasicAuth="false"/>

ADCD Personalisation of z/OS for first time users

People like to customise how they work.  This blog is the z/OS personalisation I tend to do.  If you have other suggestions please let me know.

Running zPDT on Ubuntu

The recommendation is to run zPDT under a different userid ibmsys1.   To be able to switch between userids

  • Switch user from pull down, select ibmsys1
  • You can switch back to the your normal userid using  Ctrl-Alt-F2, and use Ctrl-Alt-F3 to switch to the ibmsys1 userid
  • I could not see how to cut and paste between userids so I have a file I write to from one userid, and read the file from  “the other side”
  • Post on virtual terminals. and how to switch without F keys.

Ive been running on my normal userid without any problems

Set time zone

Edit /etc/profile and set the TZ.  It defaults to EST5EDT.  I used BST for British Summer Time ( or Europe/London).

Screen size  using bigger screens.

  • Use x3270 -model 5  to get 132*27 screen size.  It works for the console and ISPF terminal .The -oversize 133×60 parameter should work.
  • Logon and use ISPF =0  ro set defaults.  Scroll down
    • To have command line at the top / Command line at bottom  remove the /
    •  Scroll down.  Screen format 3 1. Data 2. Std 3. Max 4. Part
    • Terminal Type 4 1. 3277 2. 3277A 3. 3278 4. 3278A

Other ISPF personalisation

  • Options
    _ Command line at bottom  remove the / to have the command line at the top
    / Tab to point-and-shoot fields so you can tap to column headers, press enter and sort by the columne
  • Member list options
    / Scroll member list
    / Allow empty member list
    / Allow empty member list (nomatch)
    / Empty member list for edit only
  • pfshow off remove the PFKEYS at the bottom
  • ISPF  keys set PF12 to retrieve not cancel
  • ISPF set scroll to CSR not PAGE in all applications
  • ISPF 3.4 use reflists list of the last 30 data sets used, or your own list
  • Setting the ISPF main panel.
    • Copy ADCD.Z24A.ISPPLIB(ISR@PRIM)  to USER.Z24A.ISPPLIB(MYMAIN).
    • Add extra content and comparisons at the bottom for example ISMF,’PGM(DGTFMD01) NEWAPPL(DGT)’ .
    • The following are already defined
      •  RACF,’PANEL(ICHP00)’
      • ISMF,’PGM(DGTFMD01) NEWAPPL(DGT)’
      • SMPE,’PGM(GIMSTART) PARM(&ZCMD) NOCHECK’
      • WLM,’CMD(%IWMARIN0)’
    • When you use the TSO Logon panel specify Command ===> ispf panel(MYMAIN)  

OMVS customising

Escape key to break into long running commands use the escape key.  Default is the cent key ¢

You can use x3270 to set a key to this value, in .x3270pro

x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
  <Key>Escape: Clear()\n\
  <Key>End: FieldEnd()\n\
  Ctrl<Key>Delete: EraseEOF()\n\
  Ctrl<Key>Right: NextWord()\n\
  Ctrl<Key>Left: PreviousWord()\n\
  Ctrl<Key>Up: Home()\n\
  <Key>Control_L: Reset()\n\
  <Key>Control_R: Reset()\n\
  <Key>Prior: PF(7)\n\
  <Key>Next: PF(8)\n\
  <Btn3Down>: PA(1)\n\
  Ctrl<Key>1: PA(1)\n\ 
! the next define Alt 4 and Ctrl \ as &cent.
  Alt<Key>4: String("\\x00a2")\n\
  Ctrl<Key>backslash: String("\\x00a2")

Note

  • the \\ escape characters.
  • lines end in \n\   which is new line + continuation
  • this also defines Ctrl 1 as PA1
  • String(“\\x00a2\n”) would type the cent symbol and press enter

Set userids OMVS information

create directory /u/adcd then use RACF command

ALTUSER ADCDA OMVS(HOME('/u/adcd') PROGRAM('/bin/sh'))
ALTUSER START1 OMVS( PROGRAM('/bin/sh'))

z/OS customisation

  • Set clock time zone.  Copy  ADCC.Z24A.PARMLIB(CLOCK00) to USER.Z24A.PARMLIB(CLOCK00) and edit it.  Note:  FEU.Z24A.PARMLIB(CLOCK00) is used by default.
    •  SET TIMEZONE=E.01.00 command
  • Change TZ in /etc/profile.  It defaults to EST5EDT.  I use BST-1
  • RMF not a good idea – sometimes abends (S0C6) other times locks up z/OS

The following come from FEU.Z24A.PARMLIB

IEASYM00
IEASYS00
IEASYSWS
CLOCK00
AUTORDT
IEACMD00
MPFLST00

GTF

Create user.z24a.proclib(GTF).   The text in bold is new text compared to SYS1.PROCLIB(GTF).  It deletes the data set before reallocating it.  It allows a userid to be specified.

//GTFNEW PROC M=GTFPARM,ID=SYS1 
//DELETE EXEC PGM=IEFBR14 
//IEFRDER DD DSNAME=&ID..TRACE,UNIT=SYSDA,SPACE=(TRK,20), *
// DISP=(MOD,DELETE) 
//IEFPROC EXEC PGM=AHLGTF,PARM='MODE=EXT,DEBUG=NO,TIME=YES', *
// TIME=1440,REGION=2880K 
//IEFRDER DD DSNAME=&ID..TRACE,UNIT=SYSDA,SPACE=(TRK,20), *
// DISP=(NEW,KEEP) 
//SYSLIB DD DSNAME=USER.Z24A.PROCLIB(&M),DISP=SHR

Create USER.Z24A.PROCLIB(GTFPARM)

TRACE=SYSM,USR,TRC,DSP,PCI,SRM

and any others you need for example  USER.Z24A.PROCLIB(GTFRACF)

TRACE=USRP
USR=(F44)
END

ADCD. Backups – why, when, how – whoops.

Having got my own personal z/OS running on my laptop, I now need to look after it.  When I worked for IBM there was a team of people who looked after the z/OS systems, including backups, security and applying fixes. Suddenly with my personal z/OS,  there are a lot of things I need to think about.  Today’s topic is backups.

On my Linux  machine I have backups being taken daily to an external hard drive.  I have a Linux on a USB in case I have problems with my main machine.  How do I do backups on z/OS?

What do I want to backup? Is the wrong question.

The real question should be What do I want to restore?  For example I can get a copy of the operating system from my original download files – or from IBM, but I need to be able to restore the files particular to me.  It is better to restore the total system rather than rebuild it, because of all the additional configuration you had to do (which you may not have record of).  The JCL I have written, the data in the database or MQ queue files, security profiles.

What situations do need to restore from?

It can range from

  • I messed up – I edited a file, and now it does not work.  I cannot undo the changes.  I deleted a file.  I want to go back to last week’s copy.
  • By accident you had two copies of a program updating a file – and corrupting it.
  • The database change you made cannot be undone – you added a new field, and now the record length is longer than the 4KB buffers.
  • There has been an I/O error on the disk (though this is rare).
  • I had my laptop stolen.
  • My 3 year old child used my hard drive as a toy and found it does not float on water.

You also need to ask how long do I have to recover?  If the answer is a week, then you can order a new hard drive, and wait a week for it to be delivered.  If you need it back within hours, you’ll have a spare disk just in case (or you did a make copies to this disk – so all you need to do is use it).

Setting up z/OS

As a rule, with ADCD you should not use any of the ADCD volume for your own data.  Create your own volumes and put your data on that.   Create a user catalog, and use alias’s from the master catalog for this user catalog.  If you have a new ADCD system you need to import the user catalog, and redefined the aliases.

Backup the USER.* data sets.   Do not change the ADCD.* or SYS1.* data sets.

Some of the subsystems, DB2, CICS and MQ have data files on the A4PRD* volumes.   This means you need to backup the volumes – and will be a challenge during migration.

When can I backup?

You should backup when the files are not being used.

  • You can edit a file, use tso xmit to make a copy of the PDS, then save the file you were editing.   That is OK. Using TSO XMIT while the file is being saved could cause a consistency problem.
  • You need to backup some files as logical files, so for example backup the MQ.PAGESET.   If this data set was spread across two disks, and you do an image copy of the first disk, followed by the image copy of the second disk, the data is likely to be inconsistent (if you restore you may not find out for a week after the restore!)  MQ  (and DB2) have logic to be able to recover when a logical dataset is restored.  Some systems have a quiesce capability which stop activity to the file, without stopping the subsystem.
  • Doing full volume backups should be done when the volume is not in use, either the z/OS is down, or the volume has been varied offline and removed from zPDT.  Shutting down may be better, so all the volumes are consistent together.  Sometimes there is data in buffers which has not been written to disk (lazy write), so you have to be careful.

You might try to backup only what has changed. This could be difficult.  Unless the disks/files are read only, there is a chance that a file has changed, or a file has been put on a disk.

How do I backup files?
PDS and sequential files.

You can use the TSO XMIT (TRANSMIT) command to take a file or library and create a file which is easy to transport.

To restore it you use TSO RECEIVE indsn(…) newname(abc…) so can have the current and restored versions with different names.   This allows you to process just one, or as many members as you want.

Files in USS

The file behind the filesystem is a VSAM file.

You could use unix commands like tar or pax to package up a file or directory.   The output can be a file in the file system or into a z/OS dataset.

You could use ADRDSSU to backup the whole file system – see the next topic.

Other files

Traditionally these files are backed up use the ADRDSSU or AMATERSE (or both) utilities which can backup the file, and any indexes etc that go with it.  The output can be a z/OS dataset, or DUMPed to tape.

Full volume backups

Shutdown z/OS  down cleanly, stop zPDT (to ensure buffers in Linux are flushed), and backup the linux files.   Restart z/OS.

Where do I backup to?

To recover from operator errors on “user files”, having the backup on z/OS may be enough.

To be able to recover from system problems, or disk problems, put the backups on a different file systems.  If my z/OS system is on the SSD on my laptop, have the files go to an external file system.  Some people will have their hard drives copied to another disk system, or even “off site”.

Getting backups out of z/OS

You can use FTP into TSO or USS to copy the files.  If you use pax output to a TSO file, you can ftp into TSO.  If you pax output into an unix file, FTP into USS.

You can also virtual tape, so ADRDSSU writes to a tape which maps to a file on the Linux file.

Having backed up the files what then?  Plan for a whoops.

  • It is worth checking that your backups restore, for example restore to a spare HDD, and try to boot from it.
  • It is also worth checking that you are backing up what you think you are backing up.  I know of one customer who was backing up the MQ pagesets, but did not change the backup job when they added more page sets.  I have been guilty or repeating a line and not changing the data set name, so data set A was backed up twice, and data set B was not backed up.
  • Determine how long it will take to restore disks, restart, and recover the file(s) of interest.  If this duration is too long – review your backup and restore procedures.

What next?

I asked about backup on the zPDT group forum and had lots of great comments.  Below is a summary of the comments.

  • Use of Clonezilla. This is a partition and disk imaging/cloning program similar to True Image® or Norton Ghost®. It helps you to do system deployment, bare metal backup and recovery.
  • Use ADRDSSU DUMP followed by AMATERSE to make the z/OS backups smaller.
  • Use of a Synology Network Addressed Storage for your backups.  Synology has comments like “Good for home users and small businesses”.
  • Use ADRDSSU to dump to a volume.  Vary volume offline, then backup the volume.
  • Do not use any of the AD-CD supplied volumes for your data. Create your own volume(s) and simply add them to the devmap for new releases. You need to have a usercatalog on your volume(s) and import it to subsequent releases. You can try to make ALIAS definitions carry forward; I usually just recatalog my datasets for each new release.
  • Use LVM snapshots. With the snapshot Linux grsync with an external drive
  • Use of Borg. The main goal of Borg is to provide an efficient and secure way to backup data. Borg cuts all data into chunks, builds a hash and if the hash is not yet known, the chunk is compressed and stored in a repository. Otherwise only a pointer is set for the chunk in the current archive. This saves a lot of time and disk space (after the initial backup) because only the changed parts of the z-disk images are compressed and stored into the archive.

How long will it take?

This depends on the media you are using, and how much data.  On my laptop copying an 8GB volume from HDD to SSD took about 4 minutes or about 30 MB/second. Compressing it may speed this up.

Some good JCL examples.

Thanks to James Alexander from Hostbridge for the following examples.

The user submits a tape job with an extra "mount" tape step:
//EXP       EXPORT SYMLIST=(DSNAME,UNIT,HLQ,VOL)                  
//*                                                              
//          SET HLQ=MYHLQ                                        
//          SET DSNAME=BACKUP.D999999.DFDSS                
//          SET UNIT=591                                          
//          SET VOL=J00001                                        
//*                                                              
//MOUNT     EXEC MOUNT,UNIT=&UNIT,DSNAME=&DSNAME,VOL=&VOL        
//*                                                              
//*  What follows is a standard DFDSS backup to tape. We compress
//*  it here so less disk space is used.                          
//*                                                              
//BACKUP    EXEC  PGM=ADRDSSU,REGION=0K                          
//SYSPRINT  DD  SYSOUT=*                                          
//TAPE1    DD  UNIT=&UNIT,VOL=SER=(&VOL),                        
//         DISP=NEW,DSN=&DSNAME,LABEL=(1,SL)                      
//SYSIN     DD    *,SYMBOLS=JCLONLY,DLM=$                        
 DUMP DATASET(                              -                    
         INCLUDE(&HLQ..**           )       -                    
      )                                     -                    
      OUTDDNAME(TAPE1)                      -                    
      TOLERATE(ENQFAILURE)                  -                    
      OPTIMIZE(4)                           -                    
      COMPRESS                                                    
$                                                                
//

The mount step executes AWSCMDX that runs a Linux script.   If the “DSNAME” tape file exists it mounts it;  if not it copies a tape template file and then mounts it.  A Linux job fires once an hour and syncs all of the files in the tape directory to AWS S3.  With this any user can run a tape job and get offsite backups,   Using the same methodology they can also do their own restores.

Here is the mount proc:

//MOUNT     PROC UNIT=590,DSNAME=BAD.DATASET.NAME,VOL=T00001
//* 
//X         EXPORT SYMLIST=(DSNAME,UNIT,VOL)   
//S         SET UNIT=&UNIT,DSNAME=&DSNAME,VOL=&VOL 
//* 
//M         EXEC PGM=AWSCMDX,PARMDD=MYPARMS
//SYSPRINT  DD   SYSOUT=*
//TAPE      DD   UNIT=(580,,DEFER),LABEL=(1,BLP),VOL=SER=123456,DSN=X
//MYPARMS   DD *,SYMBOLS=JCLONLY
./mountfile &UNIT &DSNAME &VOL 
/*

And here is the mountfile script in Linux:

#!/bin/bash
Unit=$1
Filename='/z/backup/tapes/'$2
Template='/z/backup/TapeTemplate'
echo 'Checking to see if the tape file exists'
if ! [ -e "$Filename" ]
then
    echo 'File does not exist copying template'
    cp $Template $Filename
fi 

echo 'Mounting '$Filename' on unit '$Unit
awsmount $Unit -m $Filename

Why is MQWEB not accepting my certificate ? An end to head banging

I found there were many reasons why a browser’s or curl application’s digital certificate did not work with MQWEB, from an option missing, to unsupported handshake option.  Often there the messages were the vague “A problem has occurred”.

I tried to cause as many problems as possible, and blogged what you get, and the resolution; but event then I found there were even more ways of it failing.

 

I’ve written some java programs called checkTLS which act as a client or a server.

  • You can use your web browser into the server application and see information about what is being used, and if it can detect any problems (such as expired CA)
  • You can extract your certificates from the browser, and then talk to MQWEB, and see what happens in the handshake

This is alpha code.   I would be interested in any comments

  • Is this useful?
  • Does it work for you?
  • Is it too verbose?