Category: Z/OS

ADCD. Backups – why, when, how – whoops.

Having got my own personal z/OS running on my laptop, I now need to look after it.  When I worked for IBM there was a team of people who looked after the z/OS systems, including backups, security and applying fixes. Suddenly with my personal z/OS,  there are a lot of things I need to think about.  Today’s topic is backups.

On my Linux  machine I have backups being taken daily to an external hard drive.  I have a Linux on a USB in case I have problems with my main machine.  How do I do backups on z/OS?

What do I want to backup? Is the wrong question.

The real question should be What do I want to restore?  For example I can get a copy of the operating system from my original download files – or from IBM, but I need to be able to restore the files particular to me.  It is better to restore the total system rather than rebuild it, because of all the additional configuration you had to do (which you may not have record of).  The JCL I have written, the data in the database or MQ queue files, security profiles.

What situations do need to restore from?

It can range from

  • I messed up – I edited a file, and now it does not work.  I cannot undo the changes.  I deleted a file.  I want to go back to last week’s copy.
  • By accident you had two copies of a program updating a file – and corrupting it.
  • The database change you made cannot be undone – you added a new field, and now the record length is longer than the 4KB buffers.
  • There has been an I/O error on the disk (though this is rare).
  • I had my laptop stolen.
  • My 3 year old child used my hard drive as a toy and found it does not float on water.

You also need to ask how long do I have to recover?  If the answer is a week, then you can order a new hard drive, and wait a week for it to be delivered.  If you need it back within hours, you’ll have a spare disk just in case (or you did a make copies to this disk – so all you need to do is use it).

Setting up z/OS

As a rule, with ADCD you should not use any of the ADCD volume for your own data.  Create your own volumes and put your data on that.   Create a user catalog, and use alias’s from the master catalog for this user catalog.  If you have a new ADCD system you need to import the user catalog, and redefined the aliases.

Backup the USER.* data sets.   Do not change the ADCD.* or SYS1.* data sets.

Some of the subsystems, DB2, CICS and MQ have data files on the A4PRD* volumes.   This means you need to backup the volumes – and will be a challenge during migration.

When can I backup?

You should backup when the files are not being used.

  • You can edit a file, use tso xmit to make a copy of the PDS, then save the file you were editing.   That is OK. Using TSO XMIT while the file is being saved could cause a consistency problem.
  • You need to backup some files as logical files, so for example backup the MQ.PAGESET.   If this data set was spread across two disks, and you do an image copy of the first disk, followed by the image copy of the second disk, the data is likely to be inconsistent (if you restore you may not find out for a week after the restore!)  MQ  (and DB2) have logic to be able to recover when a logical dataset is restored.  Some systems have a quiesce capability which stop activity to the file, without stopping the subsystem.
  • Doing full volume backups should be done when the volume is not in use, either the z/OS is down, or the volume has been varied offline and removed from zPDT.  Shutting down may be better, so all the volumes are consistent together.  Sometimes there is data in buffers which has not been written to disk (lazy write), so you have to be careful.

You might try to backup only what has changed. This could be difficult.  Unless the disks/files are read only, there is a chance that a file has changed, or a file has been put on a disk.

How do I backup files?
PDS and sequential files.

You can use the TSO XMIT (TRANSMIT) command to take a file or library and create a file which is easy to transport.

To restore it you use TSO RECEIVE indsn(…) newname(abc…) so can have the current and restored versions with different names.   This allows you to process just one, or as many members as you want.

Files in USS

The file behind the filesystem is a VSAM file.

You could use unix commands like tar or pax to package up a file or directory.   The output can be a file in the file system or into a z/OS dataset.

You could use ADRDSSU to backup the whole file system – see the next topic.

Other files

Traditionally these files are backed up use the ADRDSSU or AMATERSE (or both) utilities which can backup the file, and any indexes etc that go with it.  The output can be a z/OS dataset, or DUMPed to tape.

Full volume backups

Shutdown z/OS  down cleanly, stop zPDT (to ensure buffers in Linux are flushed), and backup the linux files.   Restart z/OS.

Where do I backup to?

To recover from operator errors on “user files”, having the backup on z/OS may be enough.

To be able to recover from system problems, or disk problems, put the backups on a different file systems.  If my z/OS system is on the SSD on my laptop, have the files go to an external file system.  Some people will have their hard drives copied to another disk system, or even “off site”.

Getting backups out of z/OS

You can use FTP into TSO or USS to copy the files.  If you use pax output to a TSO file, you can ftp into TSO.  If you pax output into an unix file, FTP into USS.

You can also virtual tape, so ADRDSSU writes to a tape which maps to a file on the Linux file.

Having backed up the files what then?  Plan for a whoops.

  • It is worth checking that your backups restore, for example restore to a spare HDD, and try to boot from it.
  • It is also worth checking that you are backing up what you think you are backing up.  I know of one customer who was backing up the MQ pagesets, but did not change the backup job when they added more page sets.  I have been guilty or repeating a line and not changing the data set name, so data set A was backed up twice, and data set B was not backed up.
  • Determine how long it will take to restore disks, restart, and recover the file(s) of interest.  If this duration is too long – review your backup and restore procedures.

What next?

I asked about backup on the zPDT group forum and had lots of great comments.  Below is a summary of the comments.

  • Use of Clonezilla. This is a partition and disk imaging/cloning program similar to True Image® or Norton Ghost®. It helps you to do system deployment, bare metal backup and recovery.
  • Use ADRDSSU DUMP followed by AMATERSE to make the z/OS backups smaller.
  • Use of a Synology Network Addressed Storage for your backups.  Synology has comments like “Good for home users and small businesses”.
  • Use ADRDSSU to dump to a volume.  Vary volume offline, then backup the volume.
  • Do not use any of the AD-CD supplied volumes for your data. Create your own volume(s) and simply add them to the devmap for new releases. You need to have a usercatalog on your volume(s) and import it to subsequent releases. You can try to make ALIAS definitions carry forward; I usually just recatalog my datasets for each new release.
  • Use LVM snapshots. With the snapshot Linux grsync with an external drive
  • Use of Borg. The main goal of Borg is to provide an efficient and secure way to backup data. Borg cuts all data into chunks, builds a hash and if the hash is not yet known, the chunk is compressed and stored in a repository. Otherwise only a pointer is set for the chunk in the current archive. This saves a lot of time and disk space (after the initial backup) because only the changed parts of the z-disk images are compressed and stored into the archive.

How long will it take?

This depends on the media you are using, and how much data.  On my laptop copying an 8GB volume from HDD to SSD took about 4 minutes or about 30 MB/second. Compressing it may speed this up.

Some good JCL examples.

Thanks to James Alexander from Hostbridge for the following examples.

The user submits a tape job with an extra "mount" tape step:
//EXP       EXPORT SYMLIST=(DSNAME,UNIT,HLQ,VOL)                  
//*                                                              
//          SET HLQ=MYHLQ                                        
//          SET DSNAME=BACKUP.D999999.DFDSS                
//          SET UNIT=591                                          
//          SET VOL=J00001                                        
//*                                                              
//MOUNT     EXEC MOUNT,UNIT=&UNIT,DSNAME=&DSNAME,VOL=&VOL        
//*                                                              
//*  What follows is a standard DFDSS backup to tape. We compress
//*  it here so less disk space is used.                          
//*                                                              
//BACKUP    EXEC  PGM=ADRDSSU,REGION=0K                          
//SYSPRINT  DD  SYSOUT=*                                          
//TAPE1    DD  UNIT=&UNIT,VOL=SER=(&VOL),                        
//         DISP=NEW,DSN=&DSNAME,LABEL=(1,SL)                      
//SYSIN     DD    *,SYMBOLS=JCLONLY,DLM=$                        
 DUMP DATASET(                              -                    
         INCLUDE(&HLQ..**           )       -                    
      )                                     -                    
      OUTDDNAME(TAPE1)                      -                    
      TOLERATE(ENQFAILURE)                  -                    
      OPTIMIZE(4)                           -                    
      COMPRESS                                                    
$                                                                
//

The mount step executes AWSCMDX that runs a Linux script.   If the “DSNAME” tape file exists it mounts it;  if not it copies a tape template file and then mounts it.  A Linux job fires once an hour and syncs all of the files in the tape directory to AWS S3.  With this any user can run a tape job and get offsite backups,   Using the same methodology they can also do their own restores.

Here is the mount proc:

//MOUNT     PROC UNIT=590,DSNAME=BAD.DATASET.NAME,VOL=T00001
//* 
//X         EXPORT SYMLIST=(DSNAME,UNIT,VOL)   
//S         SET UNIT=&UNIT,DSNAME=&DSNAME,VOL=&VOL 
//* 
//M         EXEC PGM=AWSCMDX,PARMDD=MYPARMS
//SYSPRINT  DD   SYSOUT=*
//TAPE      DD   UNIT=(580,,DEFER),LABEL=(1,BLP),VOL=SER=123456,DSN=X
//MYPARMS   DD *,SYMBOLS=JCLONLY
./mountfile &UNIT &DSNAME &VOL 
/*

And here is the mountfile script in Linux:

#!/bin/bash
Unit=$1
Filename='/z/backup/tapes/'$2
Template='/z/backup/TapeTemplate'
echo 'Checking to see if the tape file exists'
if ! [ -e "$Filename" ]
then
    echo 'File does not exist copying template'
    cp $Template $Filename
fi 

echo 'Mounting '$Filename' on unit '$Unit
awsmount $Unit -m $Filename

Getting z/OS running on my Ubuntu laptop with zD&T and zPDT

Having downloaded and installed the zD&T, ZPDT and z/OS products on my laptop.  I was keen to start using it.   Here are part of my journey.

The  IBM ZPDT Guide and Reference redbook is excellent.

I used zD&T to install the products,  but it is missing a few things.

Check the environment

I used /usr/z1090/bin/z1090instcheck in section 4.1.84 of the red book to check the configuration.

I had

 UBUNTU kernel.core_pattern is |/usr/share/apport/apport which is BAD

I left it as it was, it is about what happens if there is a coredump.

Section 4.1.11 talks about checking /etc/sysctl.conf. You can use

sudo  /usr/z1090/bin/aws_sysctl

to set all of the parameters to value values.   It prompts before it changes any value, so is safe.

Creating the devmap

I used the perl script to create my devmap file.

Number of processors

You may want to set the number if processors less than the number of cores in your machine, so you can continue to run if zPDT is 100% busy.
Use the top command.  At the top it shows “load average: 5.3 3.7 4.1” this shows 3 numbers – this is 3 cores.

Set the number of cores using the

processors 1 # number of processors

statement.  Event with 1 processor, sometimes the laptop locked up, the cursor did not move,  or was about 5 seconds behind moving the mouse.

Storage usage

z/OS requires a minimum of 2GB to start (otherwise you get a message at IPL)

IAR057D LESS THAN 2GB OF REAL STORAGE IMPACTS SYSTEM AVAILABILITY

Using the top command gave

 

Use the free -g command to display the storage in your machine, so you do not over allocate the storage.

      total used free shared buff/cache available
Mem:     7     3    0      3          3         0
Swap:    0     0    0

The devmap had

memory 5984m  # define storage size for virtual host

When I stared zD&T  I got

AWSSTA146I Starting independent 1090 instance 'ibmsys1'
AWSEMI001T Insufficient Memory for 1090 to start.

I changed the memory to

memory 2G  # define storage size for virtual host

And I got into the IPL.

First IPL

I did a cold start, as per instructions.

I saw a message “waiting for vtam”.  So I reipled.

I used the command k s,del=n to prevent messages rolling off the top of the screen, and saw

CSV028I ABEND306-0C ISTCFCR2 and VTAM won’t start

I issued the commands

  • SETPROG APF,ADD,DSNAME=NET621.SCNMLNK1,VOLUME=A4PRD3
  • S VTAM

and the system started.

Edit ADCD.Z24A.PARMLIB(PROGAB)  and add  NET621.SCNMLNK1  A4PRD3, and do the same the the same data set on SARES1

Basic checks

The system worked as expected.

I could use x3270 localhost:3270 to get into z/OS using a 3274 “hardware controller” or x3270 10.1.1.2 via TCPIP.  This showed my IP network was basically working.

To get FTP working

Use the MVS command S  FTPD this takes a short time to start.

I installed vsftp on Ubuntu and started it

  • sudo apt install vsftpd
  • sudo service vsftpd start

I could FTP 10.1.1.2 and enter the userid and password.  When I issued any command such as ls, it hung.

I used this article on the Ubuntu firewall. The command  sudo ufw status verbose gave

Status: active

I disabled the firewall in linux

sudo ufw disable

and ftp worked.

The log showed me the activity sudo grep -i ufw /var/log/syslog |grep 10.1.1

Either of the commands give access through the firewall

  • sudo ufw allow from 10.1.1.2
  • sudo iptables -A INPUT -p tcp -s 10.1.1.2 -j ACCEPT

TCP error messages

I got

11.45.32 *EZZ9308E UNRESPONSIVE NAME SERVER DETECTED AT IP ADDRESS 9.26.4.6
11.45.32 EZZ9310I NAME SERVER 9.26.4.6
TOTAL NUMBER OF QUERIES SENT 2
TOTAL NUMBER OF FAILURES 2
PERCENTAGE 100%

This is because it is trying to use the DNS server .  The default is to use DNS then local. To fix this edit  ADCD.Z24A.TCPPARMS(TCPDATA)  insert the bold line

; LOOKUP statement 
; ================ 
; LOOKUP indicates the order of name and address resolution. DNS means 
; use the DNSs listed on the NSINTERADDR and NAMESERVER statements. 
; LOCAL means use the local host tables as appropriate for the 
; environment being used (UNIX System Services or Native MVS). 
; 
; LOOKUP DNS LOCAL  the default
LOOKUP LOCAL

Specifying LOCAL uses /etc/hosts which has

10.1.1.2 S0W1.CANLAB.IBM.COM S0W1 
127.0.0.1__ localhost localhost.localdomain localhost4 
___________ localhost4.localdomain4__________________ 
::1________ localhost localhost.localdomain localhost6 
___________ localhost6.localdomain6________________

Where the _ is really hex ’41’ !

I went into OMVS and issued

mv /etc/resolv.conf /etc/resolv.conf.old
touch /etc/resolv.conf
chmod 755 /etc/resolv.conf

ISPF primary menu

The initial ISPF menu is in

ADCD.Z24A.ISPPLIB(ISR@PRIM)

You can copy it to USER.Z24A.ISPPLIB and tailor it.  If you create a new member name – COLIN, use the command at logon ispf panel(COLIN)

The supplied ISR@PRIM has options which are not listed on the display RACF, ISMF, SMPE, WLM S (for SDSF).  This allows you to issue =S or =ISMF.

When you logon to TSO there is an ISPFLITE procedure with no optional products in the list, in case you have problems.

Unexpected messages

I got the following message many times at IPL  once for each disk.

IEC816I xxx VARY ONLINE – CU AUTHORIZATION FAILED SER=IBM-01024

A VARY ONLINE command attempted to validate the use of advanced features for the device.
The authorization failed.

I think this can be ignored.

RACF set up.

RACMAP command gets Abend System 684 rc 004

Copy ADCD.Z24A.PARMLIB(IKJTSO00) to USER.Z24A.PARMLIB(IKJTSO00)

Add RACMAP

Use SET IKJTSO=00 to activate it.

 

Getting z/OS installed on my Ubuntu laptop

Some people retire and buy an open top sports car or big motorbike.  Up here in Orkney  the weather can change every day, so instead of buying a fast car with an open top, when I retired, I got z/OS running on my laptop for the similar sort of price!   This means I can continue “playing” with z/OS and MQ, and helping the next generation to use z/OS.  At the end of this process I had the software installed on my laptop, many unwanted DVDs, and a lot of unnecessary cardboard.

I’ll cover my journey in getting the product and installing it, so anyone following in my footsteps will know what to expect and the time frame.  The process works, but could be slicker.

What options are there?

There are two emulators

Hercules – Hercules is an open source software implementation of the mainframe System/370 and ESA/390 architectures, in addition to the new 64-bit z/Architecture. Hercules runs under Linux, Windows (98, NT, 2000, and XP), Solaris, FreeBSD, and Mac OS X (10.3 and later).

zPDT from IBM.  IBM System z® Personal Development Tool (IBM zPDT®), which produces a small System z environment suitable for application development. zPDT is a PC Linux application. When zPDT is installed (on Linux), normal System z operating systems (such as IBM z/OS®) can be run on it. zPDT provides the basic System z architecture and emulated IBM 3390 disk drives, 3270 interfaces, OSA interfaces, and so on.  It needs a USB dongle or a license server to run.

What software can be used?

  • Products like z/OS, z/VM and z/VSE are licensed to run on only zPDT software.
  • Using z/VM to provide a coupling facility allows z/OS sysplex functions to be run.  Your USB dongle needs to have the support for this.
  • Public domain or “copyrighted software provided without charge”  software like OS/360,  DOS/VS, MVS, VM/370 were in the field a long time ago and can be installed on Hercules without a license.     You can also get MTS (which I used when I was at University).

What software is available to me?

I will not cover Hercules, as it is not licensed, and will only cover the IBM solution.

  1. For (big) business partners who are developing software to run on z/OS, you need to get approval to become a z ISV.  You can then  get the hardware dongles and the software from one part of IBM
  2. For other people (like me) who want to use z/OS running on a laptop for non production work there is zDevelopment and Test (zD&T). This comes in 3 flavours.
    1. ZD&T Personal Edition enables a single user to run an IBM® Z distribution on a personal computer. For more information about Personal Edition, see Personal Edition.
      • See here for the price.  You can pay for a 1 year subscription, get support from IBM,  download the code and any updates, and a hardware dongle which has the license to use/decrypt the code
      • You can pay for a perpetual license where you get the 1 year subscription as above, but can use it for ever (no support or updates after 1 year).  You can renew the license for the dongle at no charge.
    2. ZD&T Enterprise Edition enables enterprises to host an IBM Z distribution on low-cost Intel-based x86 machines. Enterprise Edition provides a web-based interface. You can extract, deploy, and manage the application images from an existing Z or ADCD packages. For more information about ZD&T Enterprise Edition, see Enterprise Edition.
      • I could not find the price for this. The license set-up looks very complex.  It looks like you need multiple machines to implement it  With a flexible licensing method, ZD&T Enterprise Edition can be used on cloud, VMs, or in-housed physical 8086 hardware. The Enterprise Edition also comes with a single user license that is known as Authorized User (AU) license, or with a multi-user license that is known as the Resource value Unit (RVU) license.
    3. ZD&T Parallel Sysplex can be used to enable a Sysplex environment that is running within z/VM®. For more information about ZD&TParallel Sysplex, see Parallel Sysplex.
      • I could not find the price for this. It looks like you have to use  a separate machine running as a license server.  The Software-based License Server and ZD&T Parallel Sysplex cannot be installed on the same machine.   I got confused between hardware dongle, software License Server and Rational tokens (which look like they need a different machine).

How do I purchase it?

See the IBM website.  That is the for the UK.    Or use

  • HTTP://IBM.COM
  • sign on
  • search for Development test environment
  • Select IBM Z Development and Test Environment in the right hand side of the window.

I’ve paid my money – now what?

This page says Software available for immediate download after online purchase.  This was not true for me.

  • You need an IBM id – this takes seconds to obtain.
  • To be able to download software and get the hardware key, you need access to Passport Advantage.
  • To get access to Passport Advantage you need a site number
  • To download software you need an IBM customer number, and an entry in a database saying what you are entitled to download.
  • To get a customer number takes a few days.

You should plan on two weeks from ordering the package to be able to run it.

The sequence of events before I could download the software and order the USB key…

  • I paid my money on the day 1
  • I quickly received an an email from IBM saying “thank you for your order”.   I was expecting a slick process like Amazon, saying “Your dongle has been dispatched – expect it in 3 days, you can download the software now” – but no.
  • I quickly received an email from IBM saying “Welcome to IBM Rational License Key Center.  Here is your License Key Center account ID:123456789.  Here are the instructions for downloading your license key”.  Great – the first question it asks  is “what is the serial number of your hardware key”.  I had not received it yet, so could not download the license.
  • I created a Passport Advantage account using the “License Key Center account ID” as my “site”.  This worked, and I got an email saying “IBM Welcomes you to Passport Advantage Online”
  • I logged on and tried downloading software – there was non available to me.  I could not order a hardware dongle as I needed a customer number.
  • I had an email from IBM Philippines asking “please confirm if this is for personal or commercial use”.  I said this was for me using as part of my company.    As a result it was flagged as “personal use”.
  • Day 2. I got an electronic PDF invoice, which told me my site number was as above.
  • I received my “Proof of entitlement” giving me my customer number and site number.
  • Later that night I got an email “IBM Electronic Support: Welcome to IBM Electronic Support“
  • Day 3. I could order my hardware dongle and there was software for me to download!
  • The money was taken from my account
  • The hardware USB key arrived on day 11 – but the courier notified my it was coming on day 15.

So overall allow for a couple of days before you can access the software, and 2 weeks for the box of dongles.

Downloading the software

If you use the DownloadDirector, check that directory is empty before you start the download – you can move files to a sub directory, or select a different directory for the downloads.  The default directory is ~/DownloadDirector.

You need about 46GB just to download the files,and 270 GB when the files are unpacked.  There are 31 unpacked files of 8GB and one file of 15 GB for z/OS and its disks.  If you are going to allocate additional disks, plan for 8GB for each.

At first glance, the download looked pretty simple. It is, unless you want to put the files on a different drive to the default.

  • From the IBM Passport Advantage site click on Download software.
  • It had IBM Z Development and Test Environment Personal Edition displayed.  I clicked on it.
  • It popped up a window saying
    • IBM Z Development and Test Environment Personal Edition
    • Please do not select an operating system and language for all Engineering and Rational software.   I don’t know what this means.  I ignored it
    • Operating system had a pull down list 1) All operating systems 2) Redhat.   I use Ubuntu so I chose 1) All operating systems.
    • Language had a pull down list of languages.  I selected English and clicked Go.
  • The download page said Required: 38 files (45319MB)  which was the first time I had been told about the amount of space needed.
  • I could select all files or individual files.   This page  gives a list of packages to volids.  A package may have more than one file in it – despite what the description is.
  • I clicked on Estimate download time for selected files.   it gave me T1 Download Director 951 minutes, HTTP 4758 minutes.
  • I clicked on  Download, which displayed a terms and conditions page.  Click “HTTP” or “Download Director”, click “I agree” and click “download now” .
  • It displayed “Your browser might ask to open/save a JNLP file in order to launch Download Director. Open the JNLP file with Java WebStart (javaws).”  Click OK
  • It popped up a window “Opening IBM_DownloadDirector.jnlp” Select “Open with “Oracle Java 8 Web Start (default)”   click OK
  • There is a pop up ” Do you want to run this application?  IBM Download Director?” Click on Run
    • The first time I ran it, it tried to put all of the downloads in ~/DownloadDirector.  I clicked cancel, and Setup, and specified the download location to my external hard drive, and clicked “Always ask for Download location”. When I reran it, I think it ignored the location and  put the files in the ~/DownloadDirector path.   The second time I came through this process, it prompted me for the Download Location as expected.  It would be nice if the first time through it prompted for download location rather than take the default.
  • Check where the files are being downloaded to, and restart the download if they are going to the wrong place.
  • The documentation says Verify the integrity of downloaded ADCD packages by using the MD5SUM that is in the adcd.md5 and pe.md5 files.   You can use the command  md5sum -c nov2019_adcd_md5.txt  to do a checksum on the downloads.

Ordering the hardware dongle.

In Passport Advantage, select “Software download & media access“.  Then select “Request Media”.   It should have the hardware dongle items you need.  I can’t remember what I ordered, but I overachieved and ordered stuff I didn’t want.   I cannot remember what the request page looked like, but once the order had been fulfilled it I could see I had ordered

  • IBM Z Development and Test Environment Version 9.1 Hardware Key Multiplatform Multilingual DVD Media Pack (BT0MIML)
  • IBM Z Development and Test Environment Version 9.1 Multiplatform Multilingual DVD Media Pack (BT0MJML)
  • IBM Z Development and Test Environment Hardware Key Version 9.5 Multiplatform Multilingual DVD Media Pack (BT0NUML)
  • IBM Z Development and Test Environment Personal Edition Version 10.0 Multilingual Hardware Key Media Pack (BT0P6ML)
  • IBM Z Development and Test Environment Personal Edition V12.0 Multilingual Hardware Key Media Pack (BT0PFML)

I do not know which of these I should have ordered, the names all look similar.   I was reminded of the phrase from the original game of Adventure “you are in a maze of twisty little passages , packages all alike“

What came in the dongle box?

I  was notified that the package would be delivered on day 15, but it arrived on day 11.  I was expecting a small jiffy bag. I got a box 30 cm * 24 cm * 20 cm, with lots of Russian Doll type boxes – it was like Christmas!

In the box I had

  • A box labelled RD&T for System z V9.1 Hardware key media pack containing
    • a box containing
      • a bag containing
        • a plastic wallet containing
          • the USB
    • a CD labelled  IBM Rational Developer and test environment for system z V9.1
    • some instructions
    • other paper work
  • A box labelled RD&T for System z V9.5 Hardware key media pack containing
    • a box containing a bag,containing a plastic wallet, containing the USB
    • a CD labelled  IBM Rational Developer and test environment for system z V9.5
    • some instructions
    • other paper work
  • A box labelled IBM z Systems Development and Test Environment Personal Edition  V10 containing
    • a box containing a bag, containing a plastic wallet, containing the USB
    • a CD labelled IBM z Systems Development and Test Environment Personal Edition  V10
    • some instructions
    • other paper work
  • A box labelled IBM z Systems Development and Test Environment Personal Edition  V12 containing
    • a box containing a bag, containing a plastic wallet, containing the USB
    • a CD labelled IBM z Systems Development and Test Environment Personal Edition  V12
    • some instructions
    • other paper work
  • A bigger box containing CDs
    • IBM Rational Integration Tester Platform Pack 1 CD
    • IBM Rational License Key Server 1 CD
    • IBM Rational Developer and test environment  for System z v9.1 Software distribution for z/OS 1.1.3 12 CDs
    • IBM Rational Developer and test environment  for System z v9.1 Software distribution for z/OS 2.1 15 CDs

Overall I got 4 USB keys (It  could be a challenge to use all 4 ,as my laptop has only one spare USB slot), lots of software and a lot of cardboard.  As I have already downloaded the images, I have two lots of software CDs I do not need. I think the provisions system needs to be looked at, so I get just what I need (one dongle) instead of a lot of waste cardboard and plastic.

Getting the license for the keys

The documentation has a topic Obtaining an update file from Rational License Key Center  which worked.

  • Follow the link and logon
  • It showed me IBM Rational Developer for System z Unit Test.
  • The serial number of one of my USBs was like 02-00222.
  • Number of Server Instances:1
  • Number of Licenses:1
  • Click generate, wait for 10 second and a window is displayed
  • Click download – and save it
  • Follow the instructions in the documentation.  I cannot use “su” so I used sudo ./Z1091_token_update…  which worked.  note the upper case Z in Z1091
  • The status command “sudo ./Z1091_token_update -status” gave me
    • Info: Processing Status request.
    • Info: Found both ADCD-1 License and ADCD-2 License.
    • Info: Command completed with 0 error(s).

As I was only entitled to one license – I had 3 spare dongles but with no license for them.

Installing zD&T.

The instructions are here.

  1. Check the instructions are for the level of zD&T you are using.   Google found me the version for 12.04; I was using 12.05
  2. The license is displayed.
    1. The license is displayed using the more command, so you can use ‘f’ and ‘b’ to go forward and back.  (Until you get to the last page when you cannot go back,  you have to decline license, and go through the install again.
    2. I found the license was not clear.   It looks like big chunks are repeated with only minor variations, which were too subtle for me.
    3. Some software you are allowed to install but not use: Fault Analyzer for z/OS and File Manager for z/OS.  (File manager:  IBM® File Manager for z/OS® (base component) provides comprehensive, user-friendly tools for working with Websphere MQ data, HFS files and QSAM, VSAM and IAM data sets. These tools include the familiar view, edit, copy and print utilities found in ISPF, enhanced to meet the needs of application developers)
    4. I think the words about Authorized User Single Session apply to zD&T personal Edition, and the words about  Resource Value Unit (RVU) apply to the enterprise edition.
    5. I don’t understand When determining the number of entitlements required for Licensee’s installation or use of the Program, Licensee is allowed to define up to two log-in identifiers for use by system programmers (i.e. system administrators or database administrators) to support Licensee development and test activities, which are not used to determine the number of entitlements required for the Program.  I do not understand this (what entitlements?, what is not used ?).  At least two userids are defined, for example IBMUSER, so additional  system programming userids may not be needed.  I do not know if I am allowed to define more userids for doing MQ application development, and defining MQ resources.
    6. The license refers to “Program”.   The text has Program Name : IBM Z Development and Test Environment Enterprise Edition Version 12.0.5, so I think the term “Program” to mean the whole package, so CICS, MQ and z/OS are individual Programs.
      1. This sounds a bit recursive, The license Programs, have programs (load modules) which have programs (compilable source code), the compiled programs have z architecture  instructions which deep down have programs in zPDT, which have instructions which have microcode programs which run on the chip.
    7. I could not find how to get a copy of the license, so I cancelled the installation and found the license in ~/DownloadDirector/zdtpefolder/license/Lic_en.txt
    8. Text like L/N: L-JWOG-BKVNF6 are the license number.  I think it is IBM internal use only as I googled it but could not find it.
    9. So overall written by lawyers and hard to understand.

Optional questions.

It asked Do you want to install Network Configuration for IBM® ZD&T Personal Edition ?(y/N):

I don’t think it installs anything.   I think it configures the network. See installation  and network.  The configuration scripts for example in /opt/ConfigGuideSample/zdt_config_network10.sh enables packet forwarding,  uses iptables to set up routing, Network Address Translation(NAT) with destination of 10.1.1.2.

I could IPL and logon without doing this optional step.  To get FTP working, I had to make one network configuration action.

It uses iptables-save  to make a copy of the IP configuration.  You may want to issue sudo iptables-save > myipconfig.txt  to save your current configuration before using this install to change it.

It makes changes to the running system, then saves them, and changes /etc/rc.local so the commands are executed at Linux boot time.

It asked Optional: Enter y to install all needed dependencies or enter n to decline.

I thought I had checked these, so I replied n.  I don’t know if it checks them and warns of any missing ones.

The installation complete message is not as documented.   The documentation says “If the package is installed successfully, the following output is displayed”

z1091-1-10.55.04.x86_64

I got

ii z1091 1.10.55.04 amd64 z1091, version 1.10.55.04, build date - 02/19/20 for Linux on Ubuntu 64bit

which is close enough.

Unzipping the files

Files  ending in .gz can be unzipped with gunzip.  The other files have to be decrypted and you need the dongle to do that.

The command

gunzip file.gz

takes the file, unzips it to file and deletes file.gz.  The option -k says do not delete the input file.

You can use the command

gunzip -c file.gz > /directory/file

the -c option says write to sysout and do not delete the input file.

To unzip all the files (this took a couple of hours to execute).

  • you could issue gunzip *.gz  . You may want to check that the directory has only the zP&T files in it before executing the command.  Remember it will delete the .gz file afterwards.  (This helps save space).
  • if you want to have the unzipped files on a different drive you could copy the .gz files to the drive then issue  gunzip *.gz in the directory.

Unzip and decrypting the RES files.

When I tried to unzip the *RES files using my colinpaice userid I got.  /usr/z1090/bin/Z1091_ADCD_install: error while loading shared libraries: libawsDiskItf.so: cannot open shared object file: No such file or directory.

I logged switched to the ibmsys1 userid, and followed the instruction in the documentation. I got

  • /usr/z1090/bin/Z1091_ADCD_install ./A4RES1.ZPD ./A4RES1
  • LIC hasp: * Communication error between API and local Sentinel License Manager : code=33

It took just over 4 minutes to unzip the SARES1 disk

Overall all

The overall process worked.  It took longer than I expected to get entitlement, and the hardware dongles.  I also have 3 dongles I cannot use, a large pile of cardboard, and two piles of CDs I don’t think I need.

My favourite Z/OS commands

I’m putting together some education materials for new z/OS users, and would like a list of common or favourite commands (the commands your fingers know without having to think about them).  If you have any more useful commands, please tell me and I’ll add them.

If there is enough interest I can make the commands into html links to the online manuals – if so please let me know.

Useful links

Console commands

Managing the 3270 console

  • k a,ref shows areas on operator console
  • k a,none – removes “out of line area” from console.  You may need to do k e,d to remove the area first
  • k a,14 makes the area 14 lines deep
  • k a,6,6 makes two areas A and B, can then use D A,L,L=B for to display the command  D A,L in window B

Some commands,  eg display commands comes up in a “out of line” frame at bottom of the screen

  • k d,f scroll forward, sometimes pf8 does this. Check with d pfk
  • k e,d remove the bottom “out of line display” area

Screen

  • k q clear the backlog of queues messages
  • k e clear the scrolling messages
  • k s,ref display the current settings
  • k s,del=rd have message scroll off the top automatically
  • k s,del=n stop messages scrolling off the top
  • k e,n  remove nth message from the top of the screen – useful of the messages do not scroll.
  • d pfk
  • k n,PFk=(001,CMD=’d a,l;d ts,l’) sets the pf key to the command strings separated by ;
  • PA1 retrieve previous command

If you lose the console

Useful MVS commands

OMVS

  • chmount -w /usr … do your work … chmoount -r /usr

VTAM

Subsystem eg DB2, MQ

TCP

JES2

The command prefix is £ or $ depending on your system and keyboard

Commands useful when the spool has filled up

  • $DSPOOL how full is the spool
  • $D JOBDEF display the number of JOES ( Job Output elements = numbers of files) JOENUM can be increased dynamically using $T OUTDEF,JOENUM=xxxx.  Don’t forget to change it in SYS1.PARMLIB(JESPRMxx) in order to make the change permanent.
  • $DJQ,SPL=(%>1) which jobs are using more than 1% of the spool
  • $DS,SPL=(%>1) which STCs are using more than 1% of the  spool
  • $dt,spl=(%>1) which TSO are using more than 1% of the spool
  • $DJQ,DAYS>2 which jobs are more than 2 days old
  • $DO JQ,JM=CCP* display all output for jobs beginning with CCP also $DO S and $do S
  • $PO JQ,JM=CCP* Purge all output for jobs beginning with CCP also $PO S and $po S
  • $DO JQ,AGE>4 Which output data sets are more than 4 days old
  • $PO JQ,AGE>4  Purge output data sets are more than 4 days old also $PO S and $po S
  • $do jq,jm=cp7*,a>7 Which output data sets belong to cp7* and are a week old also $DO S and $do S
  • $po jq,jm=im7*,a>7 Purge output data sets belong to im7* and are a week old also $PO S and $po S

ZFS

TSO commands

  • ddlist or isrddn what are my TSO allocations – useful for looking for ISPPLIB concatentation etc
    • Line commands
      • B Browse the first sixteen data sets or a single data set.
      • E Edit the first sixteen data sets or
      • a single data set.
      • V View the first sixteen data sets or a single data set.
      • M Show an enhanced member list for the first sixteen data sets or a single data set.
      • F Free the entire DDNAME.
      • C Compress a PDS using the existing allocation.
      • I Provide additional data set information.
      • Q Display list of users or jobs using a data set.
    • Primary commands
      • Apf Browse Con CList COUnt CUstom
      • DUPlicates Enq EXclude Find Locate LOAD
      • LONg LPa Member MList  Only Parmlib
      • Reset Select SHort
    • Member name (ddstring) Scan allocations for a particular member.
    • Select modname  Search for a loaded module without searching any allocated data sets.
    • CUstom Show the values in ISPTCM and some ISPF configurations.
  • mount filesystem(‘PAICE.ABC.ZFS’) mountpoint(‘/u/paice/abc’) type(ZFS) mode(read)
  • unmount filesystem(‘PAICE.ABC.ZFS’) normal

RACF commands

What RACF data sets do I have? you can use the TSO RVARY command to display them or the operator command #RVARY, where # is the subsystem recognition character defined in the IEFSSN member for RACF.

SDSF

  • OWNER myuserid to see jobs that I own, regardless of prefix.
  • “I use ST command almost exclusively in stead of DA, O etc..”
  • SYSNAME  allows you to see jobs on another connected system
  • PREFIX abc displays jobs beginning with abc
  • filter ? allows you to select on multiple criteria owner eq paice, jobname eq MQ*
  • Sort … sort on column of data sort cpu%
  • ARRange CPU% A REAL arranges the columns so the CPU% column is displayed after the REAL column
  • Prefix commands
    • S display the spool
    • SE display in edit mode
    • C issues JES command to cancel the job
    • ? displays the different output files within the job
    • SJ allows you to edit the JCL and resubmit it
  • log s display the system message log for your MVS system.
  • log o displays the merged, sysplex-wide system message log
  • ulog displays the output from commands you have issued

MQ using CSQUTIL or equivilant command

SMF

SMS – ISMF

  • Display all options on the ISMF panel 0 ISMF Profile.0 User Mode Selection  set to 2 For a Storage Administrator (SA)
  • To tailor columns displayed, used view command, specify columns and save it
  • Display disks and information about disks  2 Volume.1 Dasd ,
    • Acquire Physical Data Y,
    • Storage Group Name ” “
    • CDS Name ” “
    • Press enter
  • Display storage group name , 6 Storage Group, 1. List
    • and disks in the storage group 1. List,  then line operator listvol

TASID

TASID is a “monitor” tool developed internally within IBM to monitor activity on a z/OS system. It is displayed in ISPF on TSO.     It displays IPL info, storage usage, address space usage etc.   Think of unix “top” function.

Note that some options may not  operate correctly on all z/OS systems. Download it from IBM here.

USS Unix services

  • ls -T file display file tag, ASCII, EBCDIC etc ISO8859-1 is ASCII, code page, codepage
  • chtag -p display/change file tag info
  • chtag -t -c ISO8859-1 filename  change the filename to be oeditable
  • chtag -r filename remove any tag information from the file.
  • whence name like which name, tells you where a command came from eg whence oedit is /bin/oedit
  • tar  like zip
  • cp copy files
  • find
    • find . -name *.sh all files of type .sh in the current directory
    • find . -mtime +1 all files in the current directory modified older than one day ago
    • find . -mtime -1 all files in the current directory modified less than one day ago
    • find . -mtime 4 all files in the current directory modified exactly 4 days ago
    • find . -ctime 3 all files with attributes change exactly 3 day ago
    • find . -atime -1 all files in the current directory accessed less than one day ago. Though I do not see how this works in a R/O file system across IPLs
    • find . -mtime -1 -a -type f files modified less than 1 day ago, and it is a regular file. -type d for dictionary
  • mkdir 
  • mkdir -p and the intermediate directories
  • pax another way of packing files to make them portable. It keeps meta data.
    • pax -W “seqparms=’space=(cyl,(30,30))'” -wzvf “//’COLIN.PAX.CONSOLE'” -x os390 *.c *.h *.s *.py to create a dataset with the contents.
    • pax -vf “//’COLIN.PAX.CONSOLE'” *.s to list the dataset, matching *.s
    • pax -rvf “//’COLIN.PAX.CONSOLE'” *.s to read the dataset matching *.s
    • tar -tf myfile.pax To display contents of a pax file.  This can be used on Linux.
    • tar -xf myfile.pax  name.type To extract name.type from a pax file. It keeps the directory structure from the pax file.
  • TSO OMVS command for example the ESCape NOPFSHOW
  • du -ska . display summary of the size of each file below ‘.’ directory in 1KB blocks
  • du -ka . display the size of each file below ‘.’ directory in 1KB blocks.
  • du -ka /u/paice gives the space on the directory
  • du -ka /u/paice/* gives the space on the files within the directory
  • du -ka . | sort display the size of each file below ‘.’ directory in 1KB blocks sorted in ascending size.
  • cksum file  gives a checksum of a file, and number of bytes in the file.  There is the same command on Linux.  See here for commands on other platforms.
  • od -t cx1 colin.conf > ab display a file in text and hex. x1 says one byte hex number. x4 is as an int.
  • umask set or display the default permissions mask for when creating a unix file.

WAS, Liberty,

Angel processes

Liberty

WLM

D WLM.

V WLM,POLICY=….

SLIP – to take an action when something happens

  • D SLIP
  • D SLIP=0001
  • SLIP SET,MSGID=BPXM023I,ACTION=SVCD,END
  • SLIP MOD,ENABLE ,ID=trapid
  • SLIP MOD,DISABLE,ID=trapid
  • SLIP SET,COMP=122,ACTION=SVCD   for system abend code 122
  • SLIP SET,IF,RANGE=(2060825A,2060825C),JOBNAME=PYT, ACTION=SVCD take a dump when in the range. Jobname can be started task name.

OMVS

IPCS

  • LIST 20608252. ASID(X’0047′) LENGTH(X’1000′) INSTRUCTION display the data in instruction
  • LIST 20608252. ASID(X’0047′) LENGTH(4096) str give the value with character representation

What’s in ABCs of z/OS System Programming?

The ABC of z/OS Systems Programming is a great collection of books on z/OS, but I could find no easy documentation to tell me what was in each pdf file.   So here it is.

Volume 1, Intro to z, Systems Programmer, TSO, ISPF, JCL,  SDSF, Storage concepts
Volume 2,  Parmlib, Subsystems, LPA, SMP/E, LE
Volume 3, SMS, Datasets, Catalogs
Volume 4, VTAM, Comms Server, IP
Volume 5, Base and Parallel Sysplex, GRS, RRS, ARM, sysplex failure management System Logger, z/OS system operationGDPS, zSeries availability
Volume 6, Security on z/OS, RACF and SAF, Cryptography
Volume 7,  Infoprint Server, IP PrintWay, NetSpool, Infoprint Server, Transform, Infoprint Central
Volume 8,  Diagnosis fundamentals, IPCS, Dump analysis, problem diagnosis, Diagnostic procedures
Volume 9,  z/OS UNIX, TCP/IP installation, zSeries file system, z/OS UNIX security, Shell and programming tools
Volume 10,  z Architecture, PRISM, z14
Volume 11 Capacity planning, Performance management, RMF, SMF
Volume 12,  Workload Manager, WLM policy, WLM goal management, WLM functions, WLM ISPF application
Volume 13,  JES3

 

ZPDT redbook  reference and guide

Storage management

Summary of all storage management books
Using ADRDSSU program to backup and restore files pdf     Knowledge centre   Copy Dataset

 

Not for humans but for search engine

MQRC_EPH_ERROR 2420 (0974) (RC2420)

  • You have specified a channel in MQCONNX and this is not in the CCDT, so if you have a channel called QMACLIENT, and use use “QM” or “QM*” both will give MQRC_HOST_NOT_AVAILABLE.
  • You had a network problem, for example the application gets MQRC_CONNECTION_BROKEN. If the next MQ verb the application issues is MQCONN or MQCONNX this will fail with MQRC_HOST_NOT_AVAILABLE. You need to issue MQDISC, or retry the MQCONN(X) a second time.
  • You specified a connection address like 127.0.0.1:1414 when it was expecting 127.0.0.1(1414).

MQRC_UNKNOWN_OBJECT_QMGR: 2086 (0826) (RC2086) with a client application

This can be caused when using a client connection and specifying a queue manager name of the format “*name” (for availability) . The application takes this queue manager name, and uses it in the MQOD.
If the first character of the Queue Manager Name is “*” then MQINQ should be used to retrieve the actual queue manager name, or do not use the “*name”.

MQRC_NOT_AUTHORIZED: 2035 (07F3) (RC2035) with MQCONNX

Trying to use MQCONNX to connect to a queue manger. The info from the Knowledge centre and the AMQ message say a blank userid or password was given. I also found the following can cause the same return code

  • mqcno.SecurityParmsPtr = 0;
  • csp.CSPPasswordLength = 0;
  • sp.CSPUserIdLength = 0;
  • csp.CSPPasswordPtr= 0;
  • csp.CSPUserIdPtr = 0;
  • csp.AuthenticationType != MQCSP_AUTH_USER_ID_AND_PWD;

MQRC_ENVIRONMENT_ERROR: 2012 (07DC) (RC2012) with MQCONNX

Trying to use MQCONNX with MQCNO_RECONNECT_Q_MGR or MQCNO_RECONNECT;

  • Not using threaded application. My C program was built with -lmqic instead of -lmqic_r -lpthread
  • SHRCONV = 0 on the channel definitions

MQRC_Q_MGR_NAME_ERROR: 2058 (080A) (RC2058)

  • export MQCHLLIB not pointing to correct location
  • export MQCHLTAB pointing to the wrong name, or not set and AMQCLCHL.TAB not found in the location pointed to by MQCHLLIB
  • remember to update your .profile so this does not happen again
  • you are using a CCDT and passed in a QMNAME of XXXX, for all channels with QMNAME XXXX none could connect to the queue manager in the conname.
  • You think you were using a mqclient.ini file … but are now in a different directory
  • You are using the correct mqclient.ini file.  It has a ChannelDefinitionFile=… file.   This ccdt file is missing entries for the queue manager.  use the runmqsc command DIS CHL(*) where chltype(eq,svrconn) to display the valid channels on the server.
  • You tried to connect with the queue manager name, and need to connect to the QM group name.
  • You forgot the * in front of the queue manager name when using groups.

MQRC_KEY_REPOSITORY_ERROR: 2381 (094D) (RC2381)

  • MQSSLKEYR not set to the keystore path and file name
  • you specified …/key.kdb instead of /key without the .kdb
  • remember to update your .profile so this does not happen again

 

MQRC_OPTIONS_ERROR:2046 (07FE) (RC2046)

During MQCONNX: mqcno.Options = MQCNO_CD_FOR_OUTPUT_ONLY + MQCNO_USE_CD_SELECTION;

Solved it using

  • mqcno.Options = MQCNO_CD_FOR_OUTPUT_ONLY + MQCNO_USE_CD_SELECTION
  • or
  • mqcno.Options = MQCNO_CD_FOR_OUTPUT_ONLY
  • but not both

MQRC_CD_ERROR2277 (08E5) (RC2277)

I received message in the /var/mqm/error/*.LOG saying

AMQ9498E: The MQCD structure supplied was not valid.

EXPLANATION: The value of the ‘ChannelName’ field has the value ‘0’. This value is invalid for the operation requested.

This is only partially true. If you specify mqcno.Options=MQCNO_CD_FOR_OUTPUT_ONLY, this returns the name of the channel to you. In this case specifying a blank channel name is valid. If this options value is not specified, then a channel name is required.

AMQ9202E: Remote host not available, retry later.

EXPLANATION:
The attempt to allocate a conversation using TCP/IP to host ” for channel
QMZZZ was not successful. However the error may be a transitory one and it may be possible to successfully allocate a TCP/IP conversation later.

This is not strictly accurate.

In my MQCONNX I specified a channel name of QMZZZ which did not exist in the Client Channel Definition Table (CCDT).

  • Check the channel name in ClientConn.ChannelName
  • Specify mqcno.Options = MQCNO_CD_FOR_OUTPUT_ONLY so it ignores what is in the channel, and picks one from the entries in the CCDT.

AMQ9498E: The MQCD structure supplied was not valid.

EXPLANATION:
The value of the ‘ChannelName’ field has the value ‘0’. This value is invalid for the operation requested.
ACTION:
Change the parameter and retry the operation.

  • I got this when I specified a blank (not ‘0’ ) in the ChannelName field. If I specified mqcno.Options = MQCNO_CD_FOR_OUTPUT_ONLY I did not get this error message, as the specified channelname value is ignored. I fixed the problem by changing the MQCNO, not the MQCD

PCF: MQRCCF_MSG_LENGTH_ERROR: 3016 (0BC8) (RC3016)

I got this when using PCF and got my lengths mixed up, for example StrucLength was longer than the structure.

PCF: MQRCCF_CFST_PARM_ID_ERROR: 3015 (0BC7) (RC3015)

I got this when I issued INQUIRE_Q and passed in a channel name PCF:MQRC_UNEXPECTED_ERROR 2195 (0893) RC2195

I also got back section MQIACF_ERROR_IDENTIFIER (1013) with a value of 2031619. I cant find what this means.
My problem was I had specified an optional section – but not a required one.

PCF:MQRCCF_CFST_PARM_ID_ERROR 3015 (0BC7) RC3015

I got this when using MQCMD_INQUIRE_Q, and I had specified MQCACF_Q_NAMES instead of MQCACF_Q_NAME ( no ‘s’).

MQWEB on z/OS

SRVE0279E: Error occured while processing global listeners for the application com.ibm.mq.rest:
java.lang.NoClassDefFoundError: com.ibm.mq.mft.rest.v1.resource.MFTCommonResource (initialization failure)

SRVE0279E: Error occured while processing global listeners for the application com.ibm.mq.console: java.lang.NoClassDefFoundError: com.ibm.mq.ui.api.ras.RasDescriptor (initialization failure)

SRVE0321E: The [SecurityFilter] filter did not load during start up.
SRVE0321E: The [JSONFilter] filter did not load during start up.
SRVE0321E : The [MQConsoleSecurityFilter] filter did not load during start up.

I got this because the MQ JMS libraries had not been installed. I had /colin3/mq923/web, but was missing/colin3/mq923/java .

Liberty

CWPKI0024E: The certificate alias BPECC specified by the property com.ibm.ssl.keyStoreServerAlias is not found in KeyStore ://IZUSVR/KEY

The RACF command RACDCERT LISTRING(KEY ) ID(IZUSVR) <check the case>

gives

Certificate Label Name Cert Owner USAGE DEFAULT
-------------------------------- ------------ -------- -------
BPECC ID(START1) PERSONAL YES

So it is in the key store.

You need to check there is profile for the keyring, and as the requester needs access to the private key, has update access to it.

The userid issuing the command may not have access to the keyring. The private key was needed, so needs update access to the keyring.

RLIST rdatalib START1.KEY.LST authuser
RDEFINE RDATALIB IZUSVR.KEY.LST UACC(NONE) 
PERMIT IZUSVR.KEY.LST CLASS(RDATALIB) ID(IZUSVR) ACCESS(UPDATE)
SETROPTS RACLIST(RDATALIB) REFRESH 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh

Note: The SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh is not strictly needed but it is worth doing it in case there were updates to the certificates and the refresh command was not done.

Other options

  • The certificate was not in the keyring
  • It was NOTRUST
  • It had expired
  • The CA for the certificate was not in the keyring,
  • The userid did not have update access to the keyring when there are private certificates from other userids. See here

CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

This can be caused by

  • the requester not having access to the private key in the keyring.
  • no valid certificate in the ring.

CWWKB0117W: The IZUANG1 angel process is not available. No authorized
services will be loaded. The reason code is 4,104.
CWWKB0115I: This server is not authorized to load module bbgzsafm.
No authorized services will be loaded.

You need to define profiles and give the userid access to them

RDEF SERVER BBG.ANGEL UACC(NONE)                                    
RDEF SERVER BBG.ANGEL.ANGEL  UACC(NONE)                             
RDEF SERVER BBG.AUTHMOD.BBGZSAFM UACC(NONE)                         
RDEF SERVER BBG.AUTHMOD.BBGZSCFM UACC(NONE)                         
RDEF SERVER BBG.AUTHMOD.BBGZSAFM.ZOSWLM UACC(NONE)                  
RDEF SERVER BBG.AUTHMOD.BBGZSAFM.TXRRS  UACC(NONE)                  
RDEF SERVER BBG.AUTHMOD.BBGZSAFM.PRODMGR UACC(NONE)                 

PERMIT BBG.AUTHMOD.BBGZSAFM.SAFCRED  CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM          CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.LOCALCOM CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.PRODMGR  CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.SAFCRED  CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.TXRRRS   CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.TXRRS    CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.WOLA     CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.ZOSAIO   CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.ZOSDUMP  CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSAFM.ZOSWLM   CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSCFM CLASS(SERVER) -                       
  ID(START1) ACCESS(READ)                                         
PERMIT BBG.AUTHMOD.BBGZSCFM.WOLA     CLASS(SERVER) -              
  ID(START1) ACCESS(READ)                                         
SETROPTS RACLIST(SERVER) REFRESH

Z/OSMF

ERROR   ] CWPKI0022E: SSL HANDSHAKE FAILURE:  … PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid certification path to requested target.

With message
The signer might need to be added to local trust store … , located in SSL configuration alias izuSSLConfig.  The extended error message from the SSL handshake exception is: PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid certification path to requested target.

Action: A client has sent a certificate and Liberty is trying to validate it

  1. The certificate from the client  is self signed and not in the keyring (or trust keyring if this is used)
  2. The CA or intermeditate CAs are not in the keyring
  3. The CA’s are in the keyring, but not trusted
  4. There are CAs with the same name, but not the same content in the keyring. Check dates and other attributes

It may be that the Server’s certificate is being used to validate, so check the certificate being used by z/OSMF or Liberty.

Firefox is getting Error code: SEC_ERROR_UNKNOWN_ISSUER

Check your certificates.   You need the CA and any intermediate CAs in the “Authorities” section of certificates.  They may need to be trusted.

They are not automatically imported when you import a certificate.

IZUG476E: The HTTP request to the secondary z/OSMF instance “S0W1” failed with error type “HttpConnectionFailed” and response code “0”

I got this when trying to submit a job in the workflow topic.   You should get some ffdcs generated.

I had

  • java.net.UnknownHostException: s0w1.dal-ebis.ihost.com 
  • WorkflowException: IZUWF9999E: The request cannot be completed because an error occurred.  The following error data is returned: “IZUG476E:The HTTP request to the secondary z/OSMF instance “S0W1” failed with error type “HttpConnectionFailed” and response code “0” .”

Ping s0w1.dal-ebis.ihost.com and nslookup s0w1.dal-ebis.ihost.com did not return any data.

I edited /etc/hosts/

10.1.1.2 S0W1.CANLAB.IBM.COM S0W1 
10.1.1.2 s0w1.dal-ebis.ihost.com

and tso ping s0w1.dal-ebis.ihost.com worked.

I had to restart z/OSMF for it to pick up the change.

Server reports Certificate errors – certificate_unknown

  • unable to find valid certification path to requested target
  • Rethrowing javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
  • certificate_unknown

This was caused by the trust store at the client end did not have the CA certificate for the certificate sent from the server.  It may have had it, but it may have expired.

You may also get sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target because the trust store did not have the CA certificate, or the certificate was not valid – for example not trusted, or expired.

java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.

Check in the trace and ffdc.  I got errors

FFDC1015I: An FFDC Incident has been created: “java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=TEMP4Certification Authorit2, OU=TEST, O=TEMP is not trusted; internal cause is: java.security.cert.CertPathValidatorException: Certificate chaining error
com.ibm.ws.ssl.core.WSX509TrustManager checkServerTrusted” 

CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN (the cerificate used by the server)  was sent from the target host. The signer might need to be added to local trust store safkeyring://my/TRUST, located in SSL configuration alias defaultSSLSettings.

The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl  could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued
by (my ca)  is not trusted; internal cause is:  java.security.cert.CertPathValidatorException: Certificate chaining error

 IZUWF9999E: The request cannot be completed because an error occurred. The following error data is returned:  “java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.”

Action: Add the CA for the server’s certificate to the trust store.   I had to restart z/OSMF to pick it up

CWPKI0033E: The keystore located at safkeyringhybrid://START1/KEY did not load because of the following error: Invalid keystore format

Change

location=”safkeyringhybrid://USERID/Keyring to location=”safkeyring://USERID/Keyring to

BPXF024I

You get this message if the syslogd program is not running.

BPXP015I HFS PROGRAM /usr/lpp/zosmf/lib/libIzuCommandJni.so IS NOT MARKED PROGRAM CONTROLLED.   BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR DAEMON (BPX.DAEMON) PROCESSING.

Use the command extattr /usr/lpp/zosmf/lib/libIzuCommandJni.so to check the Program Controlled attribute is set. Use the extattr +p…. to set it if required.

I had the wrong SAF_PREFIX(‘IZUDFLT‘) in USER.Z24A.PARMLIB(IZUPRMCP).   IZUDFLT was correct.

I had other problems like invalid password when I logged onto the web browser.

Fix the problem and regenerate.

IZUG807E  An error occurred while attempting to load a required program library. Error: “require is not defined”

With an FFDC saying SRVE0190E: File not found: /IzuUICommon/1_5/zosmf/util/ui/resources/common.css

Action: close the browser and restart it

BPXO042I with D OMVS,PFS

I was expecing D OMVS,PFS or D OMVS,P to give me BPXO068I and a list of Physical File Systems.

it gives BPXO042I when the command failed.

This was due to having an HFS definition in my z/OS 3.1 system. HFS is not supported on 3.1 . I removed the definition and it worked.

FSUM2378 The start of the session was not recorded. The slot (in /etc/utmpx) for this terminal could not be updated, or a new slot for the terminal could not be created.

Function = pututxline(), terminal name = ‘/dev/ttyp0001’, program name = ‘/bin/fomtlinc’, errno = 141 (X’0000008D’), reason code =
5620060, message = ‘EDC5141I Read-only file system.’

I got this when the /etc/utmpx file was not access readwrite. My /etc was mounted read only, and the permissions of the file were 744. When I made permissions 777 it worked.

With IBMUSER with id(0) this worked with no problems.

RACF certificates

IRRSDL00 R_datalib RC 8 RS 44 (0x2c)

I got this when the job userid did not have update access to the keyring for accessing private certificate information. Eg RDATALIB profile START1.CCPKeyring.IZUDFLT.LST, and z/OSMF userid IZUSVR. The profile may not exist.

IRRD103I An error was encountered processing the specified input data set.

I got this when using RACDCERT CHECKCERT(‘COLIN.CARSA.PEM’).

The error was caused by having the file open read write. If I exited from the file, the command worked.

I also got this when using RACDCERT ADD(dsn) and dsn was not variable blocked.

IRRD104I The input data set does not contain a valid certificate.

The certificate did not have a subject DN in it.

EZD1287I TTLS Error RC: 435 Initial Handshake

435 Certification authority is unknown.

I got this having replaced the CA certificate. Deleting a certificate removes it from any keyring. When you recreate the CA, you need to add it to every keyring it was in. Before deleting a certificate it is worth listing it to see where it is used. I added it to my keyring and it worked!

IRRD109I The certificate cannot be added. Profile…. is already defined.

Action use RACDCERT LIST ID(…) to list all the certificate belonging to a user. Search for the CN value Due to a mistake, a certificate had been created using the label LABEL00000006.

I then used RACDCERT ID(START1) DELETE(LABEL(‘LABEL00000006’)) to delete it

IRRD140I The filter value does not begin with a valid prefix.

Ensure you are using upper case sod

IDNFILTER(‘CN=SSCA256.OU=CA.O=DOC.C=GB’)

instead of

IDNFILTER(‘cn=SSCA256.ou=CA.o=DOC.c=GB’)

TLS trace

java.security.cert.CertPathValidatorException: Could not determine revocation status

This is displayed when a self signed certificate is processed. It could be a self signed certificate, or the top of the hierarchy of a chain of signers.

Java java.security.NoSuchAlgorithmException: TLSv1.3 SSLContext not available

z/OS does not support TLS v1.3 yet, and this is thrown. It was announced in April 2020.

CWWKS4000E: A configuration exception has occurred. The
requested TokenService instance of type Ltpa2 could not be
found.

I found I could no longer authenticate to z/OSMF and there were CWWKS4000E messages in the z/OSMF logs. In my /global/zosmf/data/logs/zosmfServer/logs/message…. I had near the top of the file

CWWKS4106E: LTPA configuration error. Unable to create or read LTPA key file:
/global/zosmf/configuration/servers/zosmfServer/resources/security/ltpa.keys

I renamed /global/zosmf/configuration/servers/zosmfServer/resources/securityldpa.keys to keys.saved, and restarted z/OSMF.

On restart it recreated the file, and I could logon successfully.

CWWKS1100A: Authentication did not succeed for user ID COLIN. An invalid user ID or password was specified.

Also check the stderr log

[ERROR ] CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user COLIN has insufficient authority to access APPL-ID IZUDFLT.

SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

CONNECT user_id GROUP(group_id)

or


Permit IZUDFLT class(APPL) id(userid) Access(read)
setropts raclist(Appl) refresh

IKJ56251I USER NOT AUTHORIZED FOR SUBMIT YOUR TSO ADMINISTRATOR MUST AUTHORIZE USE OF THIS COMMAND

You need to give the userid access to the TSOAUTH resource

//TSO3 EXEC PGM=IKJEFT01
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
PERMIT CONSOLE CLASS(TSOAUTH) ID(COLIN) ACCESS(READ)
PERMIT JCL CLASS(TSOAUTH) ID(COLIN) ACCESS(READ)
PERMIT PARMLIB CLASS(TSOAUTH) ID(COLIN) ACCESS(READ)
SETROPTS RACLIST(TSOAUTH)

I’ve been told the following is no longer needed (but used to be needed)

PERMIT SUBMIT CLASS(TSOAUTH) ID(COLIN) ACCESS(READ)


IKJ56702I INVALID GROUP, PKIGRP3

I got this with

DELGROUP PKIGRP3

The message is totally wrong.

I could not delete the group because it had users connected to it. When I removed the userids it worked OK.

ICSF

IEC143I 213-85, … RC=X’00000008′,RSN=X’0000271C’

You may need to refresh the in memory copy of the PKDS.

IEC614I … RC 192, DIAGNOSTIC INFORMATION IS (040343C9)

You need to know to look up the last 4 digits 43c9 in DFSMS Diagnostic aids. The code means SMS-managed volumes specified for non-SMS request.

You can use the operator command D SMS,VOL(USER00), to list one volid.  If this is SMS managed, it gives the storage group name. If it is not SMS managed it gives: IGD005I COMMAND REJECTED VOLUME …… IS NOT AN SMS MANAGED DASD VOLUME .

Note: The command d sms,sg(All),listvol lists all volumes defined to SMS – even though they may not exit on the z/OS IMAGE.

MQ applications

IEW2456E SYMBOL CSQB1CON UNRESOLVED.
IEW2456E SYMBOL CSQB1DSC UNRESOLVED.

Was using cc to compile in Unix Services, and had Binder option dll. The compiler did not have this option, and so gave this message.

I used

cc -c -o c.o -Wc,SO,LIST(lst),SHOWINC,SSCOM,DLL,LSEARCH(‘COLIN.MQ924.SCSQC370′) -I //’COLIN.MQ924.SCSQC370’ c.c
cc -o mqsamp -V -Wl,LIST,MAP,INFO,DYNAM=DLL,AMODE=31 //’COLIN.MQ924.SCSQDEFS.OBJ(CSQBMQ1)’ c.o

Note I had to create the COLIN.MQ924.SCSQDEFS.OBJ, when using the xlc compiler.

IOEZ00312I Dynamic growth of aggregate ZFS.USERS in progress,
IOEZ00329I Attempting to extend ZFS.USERS by a secondary extent.
IEF196I IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS,
IEF196I IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS, 588
IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IOEZ00445E Error extending ZFS.USERS. DFSMS return code = 104, PDF code = 204.

MSG IEC070I 104-204 data set would exceed 4 gig if extended.

z/OS

CCN0629(U) DD:SYSLIN has invalid attributes.
CCN0703(I) An error was encountered in a call to fopen() while processing DD:SYSLIN.

We got these when compiling a C program, and using SYSLIN.
The problem is that the procedures such as EDCCBG, have

//SYSLIN .. DCB=(RECFM=FB,LRECL=80,BLKSIZE=3200)

when the data set had a blksize of more than 3200 (eg 27920). It sees and reports the mismatch.

Unix services

FSUM7332 syntax error: got Word, expecting )

I was trying to use a Python virtual environment and used the command

. env/bin/activate

The problem was the code page of the file.

I needed

export _BPXK_AUTOCVT=ON

I put this in my .profile file/

openssl

I got the following using x3270.

Use export SSL_VERBOSE_ERRORS=”1″ to get more info

Error: SSL: Private key file load ("...") failed:
error:0909006C:PEM routines:get_name:no start line

Using

openssl s_client -connect 10.1.1.2:2023 -cert … -certform PEM

gave more info

unable to load client certificate private key file
error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY

I needed certificate and key, for example

x3270 -port 2023 -trace -tracefile x3270.trace -certfile ~/ssl/ssl2/colinpaice.pem –keyfile /home/colinpaice/ssl/ssl2/colinpaice.key.pem 10.1.1.2

Z/OS

IEE535I … INVALID PARAMETER

I had

TRACE CT,WTRSTART=CTWTR
IEE535I TRACE INVALID PARAMETER

TRACE CT,WTRSTART=CTWTR,WRAP
ITT038I … WERE SUCCESSFULLY EXECUTED.

The first command was copied from a document. It had a trailing non blank space (x41). Remove it and the command works.

Try pasting the command into an ISPF edit session and using hex on to display the command.

EDC5164I SAF/RACF error. errno2 rs 199754829 0be8044d 0x0b8044d

I got this when I was trying to authentic using pthread_security_applid_np, and the certificate sent up did not have a subject DN.

The lack of subject DN was caused by commonName = supplied being missing in openssl when doing openssl ca … -policy signing_policy.

BPX1SOC TTLS_INIT_CONNECTION rv -1 rc ECONNRESET(1121) rs 2007593789 (0x77a9733d) 77a9733d EDC8121I Connection reset

The bpxmtext 77a9733d gives

TCPIP
JrTtlsHandshakeFailed: AT-TLS was unable to successfully negotiate a secure
TCP connection with the remote end.
Action: Review message EZD1286I for more information about the error.

On syslog was

EZD1287I TTLS Error RC: 403 Initial Handshake

Where 403 is The required certificate was not received from the communication partner.

The Wireshark output had a Certificate flow from the client to the server. This had no certificate in it.

The reason for this was,

  • the client had an RSA certificate
  • the Signature Hash Algorithms sent from the server did not include RSA.

The client was thus unable to send a certificate matching the SHA.

If I specified RSA only signature pairs, I could only use an RSA certificate. An Elliptic Curve certificate (ECDSA) had the same message and error code.

BPX1BND rv -1 rc EADDRINUSE(1115) rs 1951167047 (0x744c7247) EDC8115I Address already in use.

Because a program may not know that the “FIN” (end of conversation) has got to the other end, a socket enters a TIMEWAIT state. The IBM documentation says

If the server cannot wait for one to four minutes, you can use the setsockopt() call in the server to specify SO_REUSEADDR before it issues the bind() call. In that case, the server will be able to bind its socket to the same port number it was using before, even if the TIMEWAIT period has not elapsed. However, the TCP protocol layer still prevents it from establishing a connection to the same partner socket address. As clients normally initiate connections and clients use ephemeral port numbers, the likelihood of this is low.

BPX1SND rv -1 rc EOPNOTSUPP(1112) rs 1977578120 (0x75df7288) EDC8112I Operation not supported on socket.

I got this trying to issue bpx1snd() when there was data in the receive buffer. I used bpx1rcv to read the data, and the problem went away.

I peeked at the data before getting it, so I knew the length of the data to get, and so avoided waiting for data.

char buf[4000];
int lbuff = sizeof(buf); 
int alet = 0; 
int flags = MSG_PEEK; 
BPX1RCV( &sd,   // socket desciptor 
        &lbuff, 
        &buf, 
        &alet, 
        &flags, 
        &rv, // -1 or number of bytes 
        &rc, 
        &rs); 
 
printf("BPX1RCV Peek bytes %d data... \n",rv   ); 

lbuff = rv; // the number of bytes in the buffer 
flags = 0       ; 
BPX1RCV( &sd,   // socket descriptor 
        &lbuff, 
        &buf, 
        &alet, 
        &flags, 
        &rv, // -1 or number of bytes 
        &rc, 
        &rs); 

printf("BPX1RCV bytes %d data... \n",rv   );

BPXF135E RETURN CODE 00000079, REASON CODE 055B005C

I got this using the command

MOUNT FILESYSTEM(‘COLIN.ZFS2’) TYPE(ZFS) MOUNTPOINT(‘/u/ibmuser/temp’ )

code 79 is invalid. The 005b005c means already in use. Either

  • COLIN.ZFS2 is already mounted
  • there is something else mounted on /u/ibmuser/temp

You can use the D OMVS,F command to display the file system and where they are mounted.

BPXF135E RETURN CODE 00000081, REASON CODE 053B006C

May because the file system is mounted READ and it needs to be RDWR.

BPXMTEXT 053B006C -> JRFileNotThere: The requested file does not exist.

Problem 1

I had MOUNTPOINT(‘/u/ibmuser/test’ ) (which did not exit) not the correct MOUNTPOINT(‘/u/ibmuser/temp’ )

Problem 2

I was trying to mount it at /my. I had to go into Unix and issue mkdir /my only then could I mount the file system.

BPXF137E RETURN CODE 00000079, REASON CODE 0588002E.

THE UNMOUNT FAILED FOR FILE SYSTEM …

002E is JRFilesysNotThere. Check the file system is mounted

BPXF137E RETURN CODE 00000072, REASON CODE 058800AA

BPXF137E RETURN CODE 00000072 (the resource is busy) , REASON CODE 058800AA JRFsParentFs The file system has file systems mounted on it.

I was trying to unmount a ZFS file systemm, and got the above messages. It means you cannot unmount it, because you have other file systems attached to it. On the z/OS console it had 

BPXF271I FILE SYSTEM ZFS.USERS                             
FAILED TO UNMOUNT BECAUSE IT CONTAINS MOUNTPOINT DIRECTORIES FOR  
ONE OR MORE OTHER FILE SYSTEMS WHICH MUST BE UNMOUNTED FIRST,     
INCLUDING FILE SYSTEM COLIN.ZFS2

I used

unmount filesystem('COLIN.ZFS2') Immediate

and got message on the console

IOEZ00048I Detaching aggregate COLIN.ZFS2

RACF

ICH409I 500-002 ABEND

RACF abended S 500 00000002. I had problems with the RACF database. I carefully reallocated it using 

//IBMCOPYR  JOB    1,MSGCLASS=H 
//STEP      EXEC   PGM=IEFBR14 
//SYSUT1    DD     SPACE=(CYL,(10,10),RLSE), 
//             DCB=(LRECL=4096,RECFM=F),DISP=(MOD,DELETE), 
//             DSN=SYS1.COLIN.RACFDB.Z31B 
//STEP      EXEC   PGM=IRRUT200 PARM=ACTIVATE 
//SYSRACF   DD     DSN=COLIN.RACFDB.NEW,DISP=SHR 
//SYSUT1    DD     SPACE=(CYL,(70),,CONTIG), 
//             DCB=(DSORG=PS),DISP=(NEW,CATLG), 
//             DSN=SYS1.COLIN.RACFDB.Z31B 
/* 
//SYSUT2    DD     SYSOUT=A 
//SYSPRINT  DD     SYSOUT=A 
//SYSIN     DD     * 
    INDEX 
    MAP 
    END 
/*

and it worked.

ICH15004I BACKUP DATASET CAN NOT BE SWITCHED; dsname IGNORED

#rvary list gave me

ACTIVE USE  NUM VOLUME   DATASET                    
------ ---  --- ------   -------                    
 YES   PRIM   1 B3CFG1   SYS1.RACFDS                
 YES   BACK   1 B3USR1   SYS1.COLIN.RACFDB.Z31B

I used rvary switch,dataset(SYS1.COLIN.RACFDB.Z31B) and got the above message.

I should have just used rvary switch.

ICH21053I Unexpected return code=00000004 and reason code=00000000 from IBM MFA while processing user …

I stopped and restarted the AZF server AZF#in00 and the problem went away.

ICH408I USER(…) GROUP(…) NAME(ADCDA )NOT AUTHORIZED TO ADMINISTER DIGITAL CERTIFICATES OR CERTIFICATE REQUESTS. READ DENIED

and

IKYI002I SAF Service IRRSPX00 Returned SAF RC = 8 RACF RC = 8 RACF RSN = 8 Request denied, not authorized.

This can be caused a user not having access, or by the wrong userid being used:

User not having access

The user issuing the request was not authorised to IRR.RPKISERV.PKIADMIN CLASS(FACILITY).

Note what the message says

  • READ DENIED
  • UPDATE DENIED

Use

tso rlist facility irr.RPKISERV.PKIADmin auth

and connect the userid ( if required) to a group or give the required access with

PERMIT IRR.RPKISERV.PKIADMIN CLASS(FACILITY)
ID(ADCDA ) ACCESS(read )

setropts raclist(FACILITY) refresh

The wrong userid being used

In Http I had

<files qc2.rexx> 
  AuthName          SAFSurrogateUser 
  AuthType          Basic 
  AuthBasicProvider saf 
  Require           valid-user 
  SAFRunAs          PKISERV 
</files>

This worked fine.

I created a new rexx exec, without a defintion, and this caused the error messages because it ran with the WEBSRV userid, which did not have access, and so failed.I changed the SAFRunAs to %%CLIENT%% and it worked.

FSUM7351 not found

echo: /usr/lpp/ihsa_zos/bin/apachectl 88: FSUM7351 not found

At line 88 in the file, the command “echo” was not found. Check the path and libpath and check that /bin:/usr/sbin: are both specified

EZD0860I Stack INET is not available : errno 1011 (EDC8011I A name of a PFS was specified that either is not configured or is not a Sockets PFS.) errnojr 0x11B3005A

Programs like TCPIP ipsec could not find the default IP name.

For example /etc/resolv.conf was missing TCPIPJOBNAME

nameserver 127.0.0.1 
TCPIPJOBNAME  TCPIP

See Configuring TCPIP.DATA, Configuration statements in TCPIP.DATA, and TCPIPJOBNAME.

EZY2642E Unknown keyword:

I got this with FTP

EZY2642E Unknown keyword: PASSIVEDATAPORTS(8000,8100)

There needs to be a blank between PASSIVEDATAPORTS and the values (8000,8100)

PASSIVEDATAPORTS (8000,8100)

IKJ56529I SYMBOLIC PARMS IN VALUE LIST IGNORED
IKJ56529I COMMAND PROCEDURE HAS NO PROC STMT

In my TSO rexx program I had

/* REXX */ 

address tso 
if userid = "" then userid =  SYSVAR("SYSUID") 
say "userid="userid"." 
x = outtrap("var.") 
"TSO RACDCERT  LISTRING(TN3270) ID(PAICE)"

and got

IKJ56529I SYMBOLIC PARMS IN VALUE LIST IGNORED – RACDCERT LISTRING(TN3270) ID(START1 )+
IKJ56529I COMMAND PROCEDURE HAS NO PROC STMT

The problem was the TSO in front of the command. In effect the command was

address tso TSO RACDCERT LISTRING(TN3270) ID(PAICE)

and the TSO Command processor was unable to parse the statement.

Removing the TSO in “TSO RACDCERT LISTRING(TN3270) ID(PAICE)” solved the problem

DFDSS messages

ADR374E (001)-OPNCL(14), UNABLE TO OPEN DDNAME TARGET, 14

The target data set had a RACF profile which meant it would be encrypted. For example had

DFP INFORMATION                                  
---------------                                  
RESOWNER= NONE                                   
DATAKEY= COLINBATCHAES

Action: Use a different data set name.

ADR412E DATA SET …IN CATALOG … ON VOLUME … FAILED SERIALIZATION

You need TOL(ENQF) 

DUMP  - 
   DATASET(INCLUDE(USER.Z25D.PROCLIB        - 
                   USER.Z25D.PARMLIB   - 
                   USER.Z25D.CLIST  )- 
          )    - 
   TOL(ENQF) - 
   OUTDDNAME(TARGET) - 
   COMPRESS

ADR380E DATA SET … NOT PROCESSED, 31

Code 31 means it did not know where to put it. Use

//S1  EXEC PGM=ADRDSSU,REGION=0M PARM='TYPRUN=NORUN'               
//TARGET   DD DSN=COLIN.BACKUP.CSF,DISP=SHR                        
//SYSPRINT  DD SYSOUT=*                                            
//DASD2    DD    UNIT=3390,VOL=(PRIVATE,SER=D5CFG1),DISP=OLD       
//SYSIN DD *                                                       
 RESTORE -                                                         
    DATASET(INCLUDE(CSF.**)  )                   -                 
    REPLACE  -                                                     
    OUTDDNAME(DASD2 ) -                                            
    INDDNAME(TARGET)                                               
/*                                                                 
DATASET(INCLUDE(CSF.*) ) - REPLACE - OUTDDNAME(DASD2 ) - INDDNAME(TARGET) /

ADR380E DATA SET … NOT PROCESSED, 18

Code 18 means you need replace

//S1  EXEC PGM=ADRDSSU,REGION=0M PARM='TYPRUN=NORUN'           
//TARGET   DD DSN=COLIN.BACKUP.CSF,DISP=SHR                    
//SYSPRINT  DD SYSOUT=*                                        
//DASD2    DD    UNIT=3390,VOL=(PRIVATE,SER=D5CFG1),DISP=OLD   
//SYSIN DD *                                                   
 RESTORE -                                                     
    DATASET(INCLUDE(CSF.**)  )                   -             
    OUTDDNAME(DASD2 ) -                                        
    REPLACE  -                                                 
    INDDNAME(TARGET)                                           
/*

AZF2606E Failed to listen on loopback address (port:…. rc:112, rsn:0x112b00b6)

rc:112 means EAGAIN – resource temporarily unavailable.

112b and 00b6

I got this when TCPIP was down, and so a connect to a socket failed.

TSO

IKJ56251I USER NOT AUTHORIZED FOR SUBMIT

The userid needs access to JCL

permit JCL     class(TSOAUTH)id(COLIN) access(REAd) 
permit CONSOLE class(TSOAUTH)id(COLIN) access(REAd) 
setropts raclist(TSOAUTH) refresh 
setropts raclist(ACCTNUM) refresh

Binder

IEW2469E 9907 THE ATTRIBUTES OF A REFERENCE TO … FROM SECTION … DO NOT MATCH THE ATTRIBUTES OF THE TARGET SYMBOL. REASON 2

Message IEW2469E reason 2 is The xplink attributes of the reference and target do not match.

I was compiling this from a 64 bit C program (so is XPLINK). I needed a

#pragma linkage(IRR… ,OS)

in my program to say the program is a stub/ assembler program.

VSAM

IDC3009I ** VSAM CATALOG RETURN CODE IS 80 – REASON CODE IS IGG0CLAT-4

DEFINE PATH -
 (NAME( COLIN.ISM400.UTIL.ZFS ) -
  PATHENTRY( ISM400.UTIL.ZFS ))


IDC3022I INVALID RELATED OBJECT
IDC3009I ** VSAM CATALOG RETURN CODE IS 80 – REASON CODE IS IGG0CLAT-4
IDC3003I FUNCTION TERMINATED. CONDITION CODE IS 12

it needs

   DEFINE PATH  - 
      (NAME( COLIN.ISM400.UTIL.ZFS ) - 
      PATHENTRY( ISM400.UTIL.ZFS )) - 
      CATALOG(USERCAT.Z25D.PRODS)

Abend 0C4 in CELQLIB

I got SYSTEM COMPLETION CODE=0C4 REASON CODE=00000010 while my program was starting up.

This was caused by having a 64 bit C program linkedited with

// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31'

instead of AMODE=64.

AWSEMI307I Warning! Disabled Wait CPU 0 = 00020000 00000000 00000000 00000088

I got this WAIT088 reIPLing a system after a migration. It is described here. It means I did not have a LOADxx member corresponding to the IPL parm with xx.

This is not to be confused with System Abend code 088 (in the same manual) The auxiliary storage manager (ASM) detected a paging I/O error when attempting to read from or write to storage-class memory (SCM). Which is an Abend code, not a wait code.

C compiler

ERROR CCN3166 file:line Definition of function … requires parentheses

I had code 

#include <findkey.h> 
#include <keytype.h>

I had a definition typedef union keyTYPE…. in keytype.h but I used it in findkey before it was defined.

Solution:

Move the definition before use, or add #include <keytype.h> at the start of findkey.h

ERROR CCN3277 COLIN.ICSF.C.HELPERS(KEYTEST):31 Syntax error: possible missing
ERROR CCN3045 COLIN.ICSF.C.HELPERS(KEYTEST):32 Undeclared identifier rule.

At list 31 in COLIN.ICSF.C.HELPERS(KEYTEST) I had

char8 rule[2] = {"AES ","KEY-LEN "};

but char8 was not defined.

In my program I then defined typedef char char8[8]; and it worked. The clue was in the second message – not the first.

I put the following in my code 

#ifndef  char8 
  #error  char8 not defined
#endif

and the compilation produced

ERROR CCN3205 COLIN.ICSF.C(DELETE):36    char8 not defined

Abend S206-c0

I got the system 20c abend rc c0. The documentation says A parameter was not addressable or was in the wrong storage key.

I got this compiling a 64 bit C program, which was not XPLINK. I changed EDCCB to EDCQCB, and it worked.

EINVAL 0x0717014A

I was getting 0717014A when using shmatt. Eventually I changed my program to be 64 bit and it worked. ( I compiled it with EDCQCB <Compile, bind, and run a 64-bit C program>). It may be the original shared memory was defined in 64 bit mode.

TCP/IP

EZZ8342I gethostbyname(ABCD: Unknown host)

The names server had not been set up properly.

F RESOLVER,display

gives information like

F RESOLVER,DISPLAY                                                 
EZZ9298I RESOLVERSETUP - ADCD.Z31B.TCPPARMS(GBLRESOL)              
EZZ9298I DEFAULTTCPIPDATA - ADCD.Z31B.TCPPARMS(GBLTDATA)           
EZZ9298I GLOBALTCPIPDATA - ADCD.Z31B.TCPPARMS(GBLTDATA)            
EZZ9298I DEFAULTIPNODES - ADCD.Z31B.TCPPARMS(ZPDTIPN1)             
EZZ9298I GLOBALIPNODES - ADCD.Z31B.TCPPARMS(ZPDTIPN1)              
EZZ9304I COMMONSEARCH                                              
EZZ9304I CACHE                                                     
EZZ9298I CACHESIZE - 200M                                          
EZZ9298I MAXTTL - 2147483647                                       
EZZ9298I MAXNEGTTL - 2147483647                                    
EZZ9304I NOCACHEREORDER                                            
EZZ9298I UNRESPONSIVETHRESHOLD - 25                                
EZZ9293I DISPLAY COMMAND PROCESSED

The configuration file is RESOLVERSETUP – ADCD.Z31B.TCPPARMS(GBLRESOL).

The definitions of for the local name server are in DEFAULTIPNODES and GLOBALIPNODES.

You can either change one of these files, and use the command

F RESOLVER,refresh

to pick up the change. I did not want to change “production” so

  • I copied ADCD.Z31B.TCPPARMS(GBLRESOL) to USER.Z31B.TCPPARMS(GBLRESOL).
  • Created USER.Z31B.TCPPARMS(ZPDTIPN1) from ADCD.Z31B.TCPPARMS(ZPDTIPN1) , and added in my changes
  • Changed USER.Z31B.TCPPARMS(GBLRESOL) to have DEFAULTIPNODES – USER.Z31B.TCPPARMS(ZPDTIPN1)
  • Made the change active F RESOLVER,refresh,setup=’user.Z31B.TCPPARMS(GBLRESOL)’

When it worked, I copied my change from USER.Z31B.TCPPARMS(ZPDTIPN1) to ADCD.Z31B.TCPPARMS(ZPDTIPN1), and used F RESOLVER,refresh,setup=’user.Z31B.TCPPARMS(GBLRESOL)’ to go back to the system definitions.

IEF450I GPMSERVE GPMSERVE – ABEND=S0C4 U0000 REASON=00000011

After I got this message I started RMF, and used F RMF,START III and it worked successfully.

ERB944I Report is not available, reason code 3.

Using the RMF III option ZFSSUM I got the message, which has

3 Backlevel data or no data from the zFS interface.

I could not find what the problem was. I did wonder if it was because I had ZFS running within the OMVS address space (for performance), and so RMF could not find the ZFS job.

REXX on TSO

I got RC(-2168) from issuing a RACF command.

On the z/OS console I had 

IEA705I ERROR DURING GETMAIN SYS CODE = 878-10 IBMKEYR STEPNAME 40      
IEA705I 00FC6000 008B9368 008B9368 00015610 00003000

Abend 878-10 is There is not enough virtual private area storage 

I changed my JCL to add a region size

//IBMKEYR JOB 1,MSGCLASS=H 
//STEPNAME EXEC PGM=IKJEFT01,PARM='LRING START1',REGION=0M

ZD&T ZPDT OPRMSG no console

Problem: I tried to start a new ZD&T system, but the console did not go into a full screen mode, but was like a line printer on the Linux terminal. You could issue commands using OPRMSG ‘d a,l’

Solution: I did not have 3270port 3270 in the devmap, so no 3270s were defined. I added it

[system]
processors      3
memory  8192m
system_name     VS01
# ipl DE27 DE28NVM
3270port    3270            # port number for TN3270 connections

[manager]

The 3270 port matched the x3270 definition

x3270 -model 5  mstcon@localhost:3270  &