Why does one ping work, and the same ping doesn’t?

I was trying to check connectivity from z/OS running on my laptop. For some remote sites I could issue ping and get a response back. For some other sites I issue the ping and did not get a response back.

When I issued the pings from Linux – they both worked.

I noticed that for the pings from z/OS the field Timestamp from icmp data (relative): was 27 seconds behind. This was caused by z/OS adding leap seconds. It was not the problem.

By comparing all the fields in a successful and an unsuccessful ping, I could see that z/OS send 256 bytes of data, and Linux sent only 40 bytes of data.

On Linux, when I used

ping …. -s 256

it failed. When I used

ping …. -s 20

it worked.

Similarly on z/OS.

ping .... (length 20

The defaults lengths are different between z/OS and Linux.

The moral of this tale is

If ping does not return any data – try a very short ping.

Creating a ZFS – which way should I do it? IDCAMS LINEAR or IDCAMS ZFS?

When I looked into creating a ZFS (so I could run use it in the Unix environment) I found there were two ways of doing it both have the same end result.

The “old” way – a three step process

You use DEFINE CLUSTER …LINEAR to create the data set, then use PGM=IOEAGFMT to format it, then mount it.

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE COLIN.ZOPEN.ZFS CLUSTER
SET MAXCC=0
DEFINE -
CLUSTER -
(NAME(COLIN.ZOPEN.ZFS)-
LINEAR -
VOLUMES(USER10 ) -
STORCLAS(SGBASE ) -
MEGABYTES(6000 1000) -
SHAREOPTIONS(3 3))
/* -
//FORMATFS EXEC PGM=IOEAGFMT,REGION=0M,COND=(0,NE,DEFINE),
// PARM=('-aggregate COLIN.ZOPEN.ZFS ')
//* PARM=('-aggregate COLIN.ZOPEN.ZFS -compat')
//SYSPRINT DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//*
//*
//MOUNT EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE),(0,NE,FORMATFS))
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('COLIN.ZOPEN.ZFS') TYPE(ZFS) +
MOUNTPOINT('/u/zopen') +
MODE(RDWR) PARM('AGGRGROW') AUTOMOVE
/*

The define took less than a second, the format took about 16 seconds, and the mount took less than one second

The Two step (sounds like a dance for system administrators)

You create the dataset with type ZFS, you then mount it, and the mount formats it.

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE COLIN.ZOPEN.ZFS CLUSTER
SET MAXCC=0
DEFINE -
CLUSTER -
(NAME(COLIN.ZOPEN.ZFS)-
ZFS -
VOLUMES(USER10 ) -
STORCLAS(SGBASE ) -
MEGABYTES(6000 1000) -
SHAREOPTIONS(3 3))
/*
//MOUNT EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE))
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('COLIN.ZOPEN.ZFS') TYPE(ZFS) +
MOUNTPOINT('/u/zopen') +
MODE(RDWR) PARM('AGGRGROW') AUTOMOVE
/*

The define took less than a second – the mount took 17 seconds, because it had to do the format.

What’s the difference?

Overall the time to execute the job was the same.

I think I prefer the first way of doing it, as I have more control, and can check the format was as I expected.

If you used the second way of doing it, defined the ZFS in parmlib, I don’t know if the formatting would hold up OMVS startup.

And don’t forget

And do not forget to update your parmlib member so the ZFS is mounted automatically at IPL.

Wow I can have member generations

We have had the capability of having multiple generations of data sets on z/OS for years.
For example with three generations you can have

  • the current data set
  • the one before that
  • and the one before that.

If you create a new data set the oldest gets deleted, and they all move along one.

This has been around for years.
What I found recently was you can have this with members within a V2 PDSE (not a PDS) since 2015.

System wide set up

When you create the data set, the number of generations is limited by MAXGENS_LIMIT in the IGDSMSxx member of PARMLIB.

Use the command

D SMS,OPTIONS

This displays information like

ACDS     = SYS1.S0W1.ACDS               
COMMDS = SYS1.S0W1.COMMDS
ACDS LEVEL = z/OS V3.1
SMS PARMLIB MEMBER NAME = IGDSMS00
...
HONOR_DSNTYPE_PDSE = NO PDSE_VERSION = 2
USER_ACSVAR = ( , , ) BYPASS_CLUSTER_PREFERENCING = NO
USE_MOST_FREE_SPACE = NO MAXGENS_LIMIT = 0

To change the value of MAXGENS_LIMIT you need to change the parmlib member and use T SMS=nn (or just wait till the next IPL).

I used Which parmlib/proclib library has my member? to find the member and added

 MAXGENS_LIMIT(3)        

I then used

set sms=00

to activate it

Using the support

For example to allocate a dataset to support this.

Example JCL

//SAM00001 DD DISP=(NEW,CATLG),DSN=IBMUSER.TEST1.PDSE00,
// DSNTYPE=(LIBRARY,2),LRECL=80,BLKSIZE=8000,RECFM=FB,
// MAXGENS=3

Where

  • dsntype=(LIBRARY,2) says this a LIBRARY ( also known as PDSE) type 2
  • MAXGENS=3 this will support up to 3 generation

Using ISPF

This works in z/OS 3.1, I do not know if earlier releases have the ISPF support.

I used ISPF 3.2 and specified

──────────────────────────────────────────────────────────────────────────────
Allocate New Data Set
Command ===>

Data Set Name . . . : COLIN.PDSE2

Management class . . . (Blank for default management class)
...
Data set name type LIBRARY (LIBRARY, PDS, LARGE, BASIC, *
EXTREQ, EXTPREF or blank)
Data set version . : 2
Num of generations : 3
Extended Attributes (NO, OPT or blank)
Expiration date . . . (YY/MM/DD, YYYY/MM/DD
YY.DDD, YYYY.DDD in Julian form
DDDD for retention period in days
or blank)

I then edited a member, saved it, and then reedited it several times.

ISPF 3;4 member list gave

DSLIST            COLIN.PDSE2                           Row 0000001 of 0000001
Command ===> Scroll ===> CSR
Name Prompt Size Created Changed ID
_________ AA 2 2025/11/09 2025/11/09 09:22:35 COLIN

Using the line command b to browse the data set showed the latest content.

Using the line command N gave me

GENLIST           (AA)COLIN.PDSE2                       Row 0000001 of 0000004
Command ===> Scroll ===> CSR
RGEN Prompt Size Created Changed ID
_ 00000000 5 2025/11/09 2025/11/09 09:31:44 COLIN
_ 00000001 4 2025/11/09 2025/11/09 09:31:32 COLIN
_ 00000002 3 2025/11/09 2025/11/09 09:31:17 COLIN
_ 00000003 2 2025/11/09 2025/11/09 09:22:35 COLIN

There is

  • (AA)COLIN.PDSE2 showing member AA of data set ( library) COLIN.PDSE2
  • RGEN showing the generations
  • Generation 3 is the oldest

In the line command you can type / which lists all of the valid commands

           Action for Generation 0              

Generation Action
1. Edit
2. View
3. Browse
4. Delete
5. Info
6. Print

Prompt Action . . (For prompt field)

Select a choice and press ENTER to continue

Info gave me

   Menu  Functions  Confirm  Utilities  Help  
─────────────────────────────────────────────
EDIT USER.Z31B.PARMLIB
. . . . . . . . . . . . . . .
Member Informat
Command ===>

Data Set Name . . . : COLIN.PDSE2

General Data
Member name . . . . . . . . : AA
Concatenation number . . . . : 1
Version . Modification . . . : 01.07
...

Non-current Generations
Maximum . . . . : 3
Saved . . . . . : 3
Newest Absolute : 7
Oldest Absolute : 5

See Version and modification level numbers. You can use the commands

To set these values

Deleting a member

Using the D line command against the oldest member gave the prompt

           Confirm Member Generation Delete          

Data Set Name:
COLIN.PDSE2

Member Name:
AA

Generation to be Deleted:
-3

__Set member generation delete confirmation off

Only the specified generation will be deleted.

Press ENTER to confirm delete.
Press CANCEL or EXIT to cancel delete.

Editing a member

When you edit a member the screen is like

   File  Edit  Edit_Settings  Menu  Utilities  Compilers  Test  Help
────────────────────────────────────────────────────────────────────
EDIT COLIN.PDSE2(AA) - 01.04
Command ===>
****** ********************************* Top of Data ***************
000100 ddd
000200 bbbb
000300 44444

with the version. release at the top.

The future is already here. GS UK 2025

I was at the GS UK conference recently (which was bigger than ever) and learned so many new things.

I’ll give some short descriptions of what I learned. There is no order to these topics. Some items come in more than one area.

  • Python is very popular and widely used.
    • pyracf for issuing RACF commands from OMVS
    • pysdsf accessing SDSF from OMVS (z/OS 3.2)
  • Use ssh to access z/OS instead of ISPF
    • Many Unixy commands ported to z/OS through zopen project
    • /dfds to access data sets
    • Possibly faster than through ISPF
  • vscode is the most commonly used IDE with lots of plugins. Can edit z/OS data sets, files, submit jobs and look at spool – via Zowe
    • Git is the standard repository
    • Edit in vscode
    • check-in to Git
    • on z/os pull from Git
    • compile and run from ssh window
    • can edit on your workstation and process on z/OS
    • use Zowe/vscode to edit datasets and files in vscode, submit JCL and look at the spool. Can use zowe command line interface for issuing stuff to z/OS ( eg list files, issue operator commands)
  • People like my blog posts – Wow ! I never really knew. If you like/use anyone’s post please “like it” so the author knows. If it has been really helpful make a comment “I found this very useful”. Steven P. pointed out that you need o be logged on to a WordPress account to be able to “like” or raise a comment – this would explain why I got so few likes!
  • Lots of capturing data and displaying it in tools like grafana.
    • Python used to capture data
  • Monitoring dashboards are so last year.
    • Now have modern tools (AI) to detect changes from historical data, then alert people to differences, who then use the dashboards.
  • SDSF version 3.2 can intercept RACF writes to SMF and can display the activity, so if RACF is configured you can display OK access to resources. You just get the failures reported on syslog
  • You’ve been hacked
    • Often there is evidence months before hack – you just need to spot it
      • Pat is a z/OS sysprog who comes to work, has a coffee and starts working at 0930. Today there were two password validation failures at 0430. Is this unusual – yes. Do something
      • The password failures occurred at 0925 and 0926 – is this unusual.. you might want to check
      • You had a connection from an IP address you’ve never seen before – what do you do? Slow down their traffic and notify someone
    • Prepare for this
      • Have an integrated playbook.
        • Populate panels with the suspicious userid, and have a pull down to disable. It takes longer to type data into a RACF command than use from pre populated fields. (Eg userid COLIN Click here to disable it. )
        • Have people use the play book so they know what to do when an incident occurs. You do not have time to learn as you go along.
      • You have minutes to act. Getting someone out of bed is too long.
    • What software is running where? File Integrity Monitoring
      • I thought this was module ABC CSECT CS123 PTF level UK987654. No. If someone has zapped the module how do you know? And when did they do it? This helps you know how far you need to restore from,
      • Take each module and CSECT and create an encrypted checksum for each component. Store the information system id/library/module/CSECT/hash code. Check it weekly If someone has zapped the module – it will have a different hash. You can also see which systems are back level.
      • Do the same for configuration files eg parmlib.
      • If it has changed there should be a matching change request.
  • Regulations are in. If you have hacker insurance you will have to comply with regulation standards- such as
    • have you implemented File Integrity Monitoring (above).
    • Do you follow the standards for isolation of responsibilities.
    • “Yes” is the wrong answer. You need to demonstrate proof.
      • eg password strength. You need tests to validate it
      • prove this TCPIP connection is encrypted
  • Certificates should be reissued every 30-60 days. Not the n years it used to be.
  • OpenTelemetry tracing system. Configure applications and subsystems, to emit “here I am” to central monitoring to show path transaction took. Eg MQ client, into CICS transaction… to another CICS and back. Can do it for all – or just a sample of some requests.

  • Lots of youngster involved with z/OS.
  • Lots of old familiar faces who love working with z/OS, and should have retired years ago . This includes me (grin).

The Like box below only works if you are logged on to wordPress.

You should be able to click on the stars and give a score without logging on

Go back

Your message has been sent

Rating(required)
Warning

Using ISMF and looking at volumes

The ISMF ISPF dialogs provide a good way of displaying information about SMS,volumes data sets etc.

Like many tools – it is easy once you know how to use it.

Where is there free disk space?

Using the ISMF dialogs (as a storage administrator)

                  ISMF PRIMARY OPTION MENU - z/OS DFSMS 3.1       
Selection or Command ===>


0 ISMF Profile - Specify ISMF User Profile
1 Data Set - Perform Functions Against Data Sets
2 Volume - Perform Functions Against Volumes

Option 2 Volume –>, 1 for DASD

                          VOLUME SELECTION ENTRY PANEL              Page 1 of 3
Command ===>

Select Source to Generate Volume List . . 2 (1 - Saved list, 2 - New list)
1 Generate from a Saved List Query Name To
List Name . . COLIN Save or Retrieve
2 Generate a New List from Criteria Below
Specify Source of the New List . . 1 (1 - Physical, 2 - SMS)
Optionally Specify One or More:
Enter "/" to select option Generate Exclusive list
Type of Volume List . . . 1 (1-Online,2-Not Online,3-Either)
Volume Serial Number . . USER* (fully or partially specified)
Device Type . . . . . . . (fully or partially specified)
Device Number . . . . . . (fully specified)
To Device Number . . . (for range of devices)
Acquire Physical Data . . Y (Y or N)
Acquire Space Data . . . Y (Y or N)
Storage Group Name . . . * (fully or partially specified)
CDS Name . . . . . . . 'ACTIVE'

Where Acquire Physical Data obtains

  • Device type such as 3390-3
  • Device number such as 0a94
  • Shared DASD such as YES
  • Use Attributes such as PRIV

and Acquire Space Data obtains and calculates

  • Free space
  • Free extents,
  • % Free index status
  • Allocated space
  • Free DSCBs
  • Fragmentation Index
  • Free VIRs (VTOC Index Records)
  • Largest extent
  • Physical status

gave

                                                           VOLUME LIST           
Command ===> Scroll ===> HALF
Entries 1-8 of 8
Enter Line Operators below: Data Columns 3-8 of 45

LINE VOLUME FREE % ALLOC FRAG LARGEST FREE
OPERATOR SERIAL SPACE FREE SPACE INDEX EXTENT EXTENTS
---(1)---- -(2)-- ---(3)--- (4)- ---(5)--- -(6)- ---(7)--- --(8)--
USER00 81233K 1 8233268K 674 6308K 135
USER02 100822K 4 2670678K 252 53953K 31
USER03 5202K 0 2766298K 483 2213K 9
USER04 1457936K 53 1313564K 49 1328284K 20
USER05 78577K 3 2692923K 83 63802K 4
USER06 289020K 10 2482480K 179 105581K 16
USER07 500735K 18 2270765K 92 356142K 11
USER08 953881K 34 1817619K 86 509478K 3
---------- ------ ----------- BOTTOM OF DATA ----------- ------ ----

If you press PF1 for help, press “enter to continue” and select option 6 Data Column Descriptions it gives all of the columns names. They have a name and number.

Back in the data display, you can scroll sideways to display these field.

You can use the command view 3,23 to display just those fields, or view * to display all of the fields.

You can use view save name and view select name to save the current view configuration as name.

I could not find a way of displaying all of the attributes for a volume in a vertical list.

Sort

You can say sort 3 to sort by column 3 ascending, or just sort where you can specify the columns and ascending or descending.

The columns you can sort by depend on what are displayed via the view command.

How difficult is it to delete a data set – it can be harder than you think.

I was using a volume on one zD&T system, and used the same volume on the next version of zD&T. Some of the datasets are SMS managed, and were cataloged in the first system’s master catalog.

Now I’m on the latest level of zD&T, I am trying to delete unwanted data sets, and this was a challenge, as all of the usual methods did not work.

I can use ISPF 3.4 and specify the volume USER05. There are some old datasets, such as SYS1.S0W1.Z24C.DMP00001. These are not cataloged. They are only visible in ISPF 3.4 if you specify the volume. If you try to browse the data set it gives data set not cataloged.

I failed to delete the dataset using the usual techniques….

From ISPF

If you try to delete it from ISPF 3.4 it says “Data set not cataloged”,

From JCL

For example

//IBMDEL   JOB 1,MSGCLASS=H 
//TSO EXEC PGM=IEFBR14
//DDS DD DSN=SYS1.S0W1.Z24C.DMP00001,VOL=SER=USER05,
// DISP=(OLD,DELETE)
/*

This fails with the error message

IEFA107I IBMDEL TSO DDS – DATA SET SYS1.S0W1.Z24C.DMP00001 NOT FOUND

From IDCAMS

//IBMDEL2  JOB 
//STEP1 EXEC PGM=IDCAMS
//DD1 DD VOL=SER=USER05,UNIT=3390,DISP=OLD
//* DSN=SYS1.S0W1.Z24C.DMP00003
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
DELETE -
SYS1.S0W1.Z24C.DMP00001 -
FILE (DD1) -
PURGE
/*

Fails with

IDC3012I ENTRY SYS1.S0W1.Z24C.DMP00001 NOT FOUND
IDC3009I ** VSAM CATALOG RETURN CODE IS 8 – REASON CODE IS IGG0CLEG-42

The problem…

The dataset is on an SMS managed volume. All data sets managed by SMS have to be cataloged. The catalog that owns the dataset is on a non existant system – the old master catalog from the system from 3 years ago. The outcome is that there is no catalog available to locate the data set.

On the SMS volume USER05 is the SMS dataset SYS1.VVDS.VUSER05. This is like a catalog for the volume, and contains information about all the data sets on the SMS volume. You can use IDCAMS to print this data set (in hex).

The solution…

You need to recatalog it – then delete it

recatalog

//IBMRECAT JOB 
//RECATSMS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE NONVSAM(NAME(SYS1.S0W1.Z24C.DMP00001 ) -
OWNER(COLIN ) -
VOLUMES(USER05) -
DEVT(3390 ) -
RECATALOG)
LISTCAT ENT(SYS1.S0W1.Z24C.DMP00001) ALL
/*

This gave

  DEFINE NONVSAM(NAME(SYS1.S0W1.Z24C.DMP00001   ) -                                           
OWNER(COLIN ) -
VOLUMES(USER05) -
DEVT(3390 ) -
RECATALOG)
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

LISTCAT ENT(SYS1.S0W1.Z24C.DMP00001) ALL
NONVSAM ------- SYS1.S0W1.Z24C.DMP00001
IN-CAT --- CATALOG.Z31B.MASTER
HISTORY
DATASET-OWNER------COLIN CREATION--------2025.296
RELEASE----------------2 EXPIRATION------0000.000
ACCOUNT-INFO-----------------------------------(NULL)
SMSDATA
STORAGECLASS -----SCBASE MANAGEMENTCLASS---(NULL)
DATACLASS --------(NULL) LBACKUP ---0000.000.0000
ENCRYPTIONDATA
DATA SET ENCRYPTION-----(NO)
VOLUMES
VOLSER------------USER05 DEVTYPE------X'3010200F' FSEQN------------------0
ASSOCIATIONS--------(NULL)
ATTRIBUTES

Having done this, I can now browse SYS1.S0W1.Z24C.DMP00001 in ISPF without specifying a volume, so proving it is now cataloged.

Delete it

If I now try to delete it I get

IEC331I 050-088(,USER05),COLIN,ISPFPROC,VCMP,IGG0CLE4
IEC614I SCRATCH FAILED – RC 008, DIAGNOSTIC INFORMATION IS (040942D1),
ISPFPROC,USER05,SYS1.S0W1.Z24C.DMP00001

The codes for IEC331I are given in IEC3009I. If you search for “code 50” then page down till you get code 88, it says

Explanation: A VVR or NVR with the correct component name was found, but the catalog name did not match. On a delete request, the BCS record will be deleted, but the VVR or NVR and the format 1 DSCB will not be scratched. There is no SFI data.

I had to use

//IBMDEL2  JOB 
//STEP1 EXEC PGM=IDCAMS
//DD1 DD VOL=SER=USER05,UNIT=3390,DISP=OLD
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
DELETE SYS1.S0W1.Z24C.DMP00001 NVR FILE(DD1)
/*

Which says delete the data set (and remove it from the VVDS dataset).
This job was successful, and the data set was no longer on the volume.

Sometimes I had to remove the NVR from the delete statement.
My overall JCL was

//IBMDEL   JOB  1 
//E1 EXPORT SYMLIST=*
// SET NAME='IZUSVR.JVM.IZUSVR1.D250809.T080903.X001'
// SET VOL='USER06'
//RECATSMS EXEC PGM=IDCAMS
//DD1 DD VOL=SER=&VOL,UNIT=3390,DISP=OLD
//SYSPRINT DD SYSOUT=*
//SYSIN DD *,SYMBOLS=JCLONLY

DEFINE NONVSAM(NAME(&NAME.) -
OWNER(COLIN ) -
VOLUMES(&VOL.) -
DEVT(3390 ) -
RECATALOG)

LISTCAT ENT(&NAME. ) ALL

DELETE &NAME. NVR FILE(DD1)

DELETE &NAME. FILE(DD1)
/*

Configuring and using the RMF GPM Server

RMF provides information on the usage of system resources, such as CPU, Channel usage, Disk response time etc. You can get reports from an attached 3270 screen, from a web server, and from a REST request.

For the web server and REST requests, you need the GPM server running. It took me a while to get this running, and to get useful data out of it.

GPMServer uses basic authority checking of userid and password. Alternatively it can use certificates from the client to authenticate on z/OS.

There are two versions of GPMSERVE. It looks like the newer one is written in Java. I only have access to the old version.

GPM Setup

I used

//GPMSERVE PROC MEMBER=00 
//STEP1 EXEC PGM=GPMDDSRV,REGION=128M,TIME=1440,
// PARM='TRAP(ON)/&MEMBER'
//* PARM='TRAP(ON),ENVAR(ICLUI_TRACETO=STDERR)/&MEMBER'
//*
//*STEPLIB DD DISP=SHR,DSN=CEE.SCEERUN
//* DD DISP=SHR,DSN=CBC.SCLBDLL
//GPMINI DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMINI)
//GPMHTC DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMHTC)
//GPMPPJCL DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMPPJCL)
//CEEDUMP DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
// PEND

CACHESLOTS(4)                   /* Number of timestamps in CACHE     */ 
DEBUG_LEVEL(3) /* informational messages */
SERVERHOST(10.1.1.2)
HTTPS(ATTLS) /* AT-TLS setup required */
MAXSESSIONS_HTTP(20) /* MaxNo of concurrent HTTP requests */
HTTP_PORT(8803) /* Port number for HTTP requests */
HTTP_ALLOW(*) /* Mask for hosts that are allowed */
HTTP_NOAUTH() /* No server can access without auth.*/
CLIENT_CERT(NONE)
/* CLIENT_CERT(ACCEPT) */

The essence of my AT-TLS definitions is (from my Easy-ATTLS)

LocalPortRange : 8803
Direction : Both
ApplicationControlled : Off
TTLSEnabled : On
CtraceClearText : On
Trace : 2
HandshakeRole : Server
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
HandshakeTimeout : 3
ClientECurves : Any
ServerCertificateLabel : NISTECCTEST
V3CipherSuites : [
1302 TLS_AES_256_GCM_SHA384,
1301 TLS_AES_128_GCM_SHA256,
003D TLS_RSA_WITH_AES_256_CBC_SHA256,
C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
]

I used CtraceClearText : On so I could trace the flows and see the encrypted traffic.

The Chrome browser used ECDHE* cipher specs. I had specified C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, and I could this was being used.

The Chrome browser prompted for userid and password which was passed up to the server.

Issuing commands

You start the server with

S GPMSERVE

If it abends with

IEF450I GPMSERVE GPMSERVE - ABEND=S0C4 U0000 REASON=00000011

Check RMF is active. And check you have issued F RMF,START III to start the data collection.

You stop the server

p gpmserve

You can display information about the server

f gpmserve,display

The newer version of GPMSERVE uses commands like F GPMSERVE,APPL=DISPLAY

The output is like

+GPM062I DDS-REFR 01/02 084125 CYCLE=314. WAITING 10 SEC
+GPM062I HTTP-LIS 01/02 084119 MAX=20 ACTIVE=0 SUSPEND=1
+GPM062I RMF_DDS_ATTLS 01/02 074900 STARTING …
+GPM062I RMF_DDS_OPTS 01/02 074900 STARTING …
+GPM062I HTTP-CLI 01/02 083219 ::FFFF:10.1.0.2 TERMINATED. SUSPENDED.

Where 01/02 is Jan 2nd. 074900 is 07:49:00

Certificate and keyring set up

I reused an existing keyring. The AT-TLS definitions give the keyring is start1/TN3270 and the certificate to use is NISTECCTEST.

List the ring contents

tso RACDCERT listring(TN3270) id(START1)

The keyring included the CA for my NISTECCTEST certificate, and the CA for the client’s certificate (on Linux).

My certificate authentication to work, I needed the client certificate connected to the keyring.

On Linux I had

  • ca256.pem the Certificate Authority
  • colinpaice.pem

I FTPed these to z/OS as VB data sets, COLIN.CA256.PEM, and COLIN.PAICE.PEM.

Import the CA into z/OS

//IBMRACFI JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.CA256.PEM')
RACDCERT DELETE -
(LABEL('CA256')) CERTAUTH
RACDCERT CERTAUTH ADD('COLIN.CA256.PEM') -
WITHLABEL('CA256') TRUST
RACDCERT CERTAUTH LISTCHAIN(LABEL('CA256'))

RACDCERT CONNECT(CERTAUTH LABEL('CA256') -
RING(TN3270) ) ID(START1)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH
/*

and import the users .pem file.

//IBMRACFI JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.PAICE.PEM')
RACDCERT DELETE -
(LABEL('RMFCERT')) ID(COLIN)
RACDCERT ID(COLIN) ADD('COLIN.PAICE.PEM') -
WITHLABEL('RMFCERT') TRUST
RACDCERT ID(COLIN) LISTCHAIN(LABEL('RMFCERT'))
RACDCERT ID(START1) CONNECT(ID(COLIN ) LABEL('RMFCERT') -
RING(TN3270))
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH
/*

When a user connects with a certificate, GPMSERVE looks in the keyring for the passed certificate, and finds the userid for it.

Setting up the security profiles

You need to set up a CLASS(APPL) profile for GPMSERVE. Give any authorised userids read access to the profile.

//IBMRACF  JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
* Delete and redefine the profile
* List it first
RLIST APPL GPMSERVE authuser
RDELETE APPL GPMSERVE
SETROPTS RACLIST(APPL) refresh
RDEFINE APPL GPMSERVE UACC(NONE) NOTIFY(COLIN)
PERMIT GPMSERVE CLASS(APPL) ID(IBMUSER) ACCESS(READ)
PERMIT GPMSERVE CLASS(APPL) ID(COLIN ) ACCESS(READ)
PERMIT GPMSERVE CLASS(APPL) ID(ADCDB ) ACCESS(NONE)
SETROPTS RACLIST(APPL) refresh
RLIST APPL GPMSERVE authuser
SETROPTS RACLIST(APPL) refresh
/*

I specified RDEFINE APPL GPMSERVE UACC(NONE) NOTIFY(COLIN) so the userid COLIN gets notified if anyone tries to use the profile and fails. Using WARNING does not work.

Changing security

If you give a userid read permission to the CLASS(APPL) GPMSERVE profile, you need to stop and restart GPMSERVE to pick up the changes. It looks like GPMSERVE caches the access after first use, and there is no refresh security command.

When tracing a job it helps to trace the correct address space.

The title When tracing a job it helps to trace the correct address space is a clue – it looks obvious, but the problem was actually subtle.

The scenario

I was testing the new version of Zowe, and one of the components failed to start because it could not find a keyring. Other components could find it ok. I did a RACF trace and there were no records. The question is why were there no records?

The execution environment.

I start Zowe with S ZOWE33. This spawns some processes such as ZOWE335. This runs a Bash script which starts a Java program.

I start a GTF trace with

s gtf.gtf,m=gtfracf
#set trace(callable(type(41)),jobname(Zowe*))

Where callable type 41 is for r_datalib services to access a keyring.

No records were produced

What is the problem?
Have a few minute pause to think about it.

Solution

After 3 days I stumbled on the solution – having noticed, but ignored the evidence. I wondered if the Java code to process keyrings, did not use the R_datalib API, I wondered if Java 21 uses a different jar file for processing keyrings – yes – but this didn’t solve the problem.

The solution was I should have been tracing job ZWE33CS! Whoa – where did that come from?

The Java program was started with

_BPX_JOBNAME=ZWE33CS /usr/lpp/java/J21.0_64/bin/java

See here which says

When a new z/OS® UNIX process is started, it runs in a z/OS UNIX initiator (a BPXAS address space). By default, this address space has an assigned job name of userIDx, where userID is the user ID that started the process, and x is a decimal number. You can use the _BPX_JOBNAME environment variable to set the job name of the new process. Assigning a unique job name to each … process helps to identify the purpose of the process and makes it easier to group processes into a WLM service class.

If I use the command D A,L it lists all of the address spaces running on the system. I had seen the ZOWE33* ones, and also the ZWE* ones – but ignored the ZWE* ones. Once I knew the solution is was so obvious.

What is my Unix process doing?

I was familiar with the USS command ps -ef which displays output like

     UID        PID       PPID  C    STIME TTY       TIME CMD 
WEBSRV 16842766 1 - 07:23:05 ? 0:00 -sh -c /web/httpd1/bin/apachectl -k start -f /web/httpd1/conf/httpd.conf -DNO_f

For Zowe threads I was getting

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/inter

which was annoyingly truncated.

The command ps -e -o args > aa gives the whole command line (up to 1024 bytes) such as

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/internal/start/component/cli.js

Another useful command when you know it.

How do I logon to ISPF and allocate my data sets?

Yes, I know you do not logon to ISPF, but the title is shorter than how do I logon to TSO, and start ISPF so my data sets are allocated as I want them.
I wrote this blog post because I was trying to use ISMF and save information into ISPF tables, but I could not use the information in the tables because my table data set was not in the ISPTLIB concatenation.

When I used TSO ISRDDN to display the data sets allocated to my TSO session I had

ISPTABL -> COLIN.S0W1.ISPF.ISPPROF
ISPTLIB -> ISP.SISPTENU
-> SYS1.DGTTLIB
-> SYS1.SBLSTBL0
...

COLIN.S0W1.ISPF.ISPPROF was not in the list of data sets in the ISPTLIB concatenation.

This lead me to the question – how do I add COLIN.S0W1.ISPF.ISPPROF to the ISPTLIB concatenation?

How do I allocate my datasets to ISPF

When I logon to ISPF I get

------------------------------- TSO/E LOGON -----------------------------------


Enter LOGON parameters below: RACF LOGON parameters:
Userid ===> COLIN
Password ===>
Procedure ===> ISPFPROC Group Ident ===>
Acct Nmbr ===> ACCT#
Size ===> 2096128
Perform ===>
Command ===> ex 'colin.zlogon.clist'

You can influence what happens by specifying a different Procedure, or specifying a command in Command.

The PROCEDURE ===> ISPFPROC is JCL to start a TSO address space and allocate system wide datasets.

Once ISPF has started, you can issue the command TSO ISRDDN to display all of the datasets allocated to TSO.
The ISRDDN command member ISPFPROC will find and show you which of the allocated data sets contain the member.
it gave me

                           Current Data Set Allocation         Member was found
Command ===> Scroll ===> PAGE

Message Act DDname Data Set Name Actions: B E V M F C I Q
Member: ISPFPROC >_ SYSPROC ADCD.Z31B.PROCLIB

You can enter the B command in the >_ field to browse the member directly

Aside:

The Actions: B E V M F C I Q are commands for

  • B Browse the first sixteen data sets or a single data set.
  • E Edit the first sixteen data sets or a single data set.
  • V View the first sixteen data sets or a single data set.
  • M Show an enhanced member list for the first sixteen data sets or a single data set.
  • F Free the entire DDNAME.
  • C Compress a PDS using the existing allocation.
  • I Provide additional data set information.
  • Q Display list of users or jobs using a data set.

Browse the member

This member has

//********************************************************************    
//*
//* ISPF FULL-FUNCTION LOGON PROC
//*
//*********************************************************************
//ISPFPROC PROC ROOT='/usr/lpp/zosmf' /* ZOSMF INSTALL ROOT */
// EXPORT SYMLIST=(XX)
// SET QT=''''
// SET XX=&QT.&ROOT.&QT.
//ISPFPROC EXEC PGM=IKJEFT01,REGION=0M,DYNAMNBR=200,
// PARM='%ISPFCL'
//CEEOPTS DD *,SYMBOLS=JCLONLY
ENVAR("PATH=/bin:&XX./bin")
//SYSUADS DD DISP=SHR,DSN=SYS1.UADS
//SYSLBC DD DISP=SHR,DSN=SYS1.BRODCAST
//SYSPROC DD DISP=SHR,DSN=USER.&SYSVER..CLIST
// DD DISP=SHR,DSN=FEU.&SYSVER..CLIST
// DD DISP=SHR,DSN=ADCD.&SYSVER..CLIST
// DD DISP=SHR,DSN=ISP.SISPCLIB
...
//ISPTLIB DD DISP=SHR,DSN=ISP.SISPTENU
// DD DISP=SHR,DSN=SYS1.DGTTLIB
...
//SDSFMENU DD DSN=ISF.SISFPLIB,DISP=SHR
//ISPTABL DD DSN=SYS1.SMP.OTABLES,DISP=SHR

This JCL

  • creates the environment PATH=/bin/:/usr/lpp/zosmf/bin
  • Allocates lots of data sets, for example SYSPROC has USER…..CLIST depending on the value of the global symbol &SYSVER (Z31B at the moment). If I IPL a different level of z/OS it may have a different level, such as Z24C
  • Allocates fixed name data sets such as ISP.SISPCLIB
  • Allocates lots of ISPF tables for input
  • Allocates an SDSF menu data set
  • Allocates a table ISPTABL for ISPF
  • But does not allocate an ISPTABL for my personal tables.

In the JCL it has

//ISPFPROC EXEC PGM=IKJEFT01,REGION=0M,DYNAMNBR=200,          
// PARM='%ISPFCL'

Which says invoke TSO (IKJEFT01) and execute the %ISPFCL Clist (or REXX).

Use PF3 to return from ISRDDN.

Where is ISPFCL?

The above JCL uses CLIST/REXX ISPFCL as a profile to do additional processing, such as allocating additional data sets.

You could allocate datasets in the ISPF JCL instead of through the CLIST – but the CLIST allows conditional processing, such as if the ISPFPROF data set does not exist, then allocate it.

You can use TSO ISRDDN again and specify member ISPFCL . The member was found, in four places (see the Member: below)

                           Current Data Set Allocations           Row 98 of 118
Command ===> _____________________ Scroll ===> PAGE

Message Act DDname Data Set Name Actions: B E V M F C I Q
Member: ISPFCL >_ SYSPROC USER.Z31B.CLIST
>_ FEU.Z31B.CLIST
Member: ISPFCL >_ ADCD.Z31B.CLIST
>_ ISP.SISPCLIB
Member: ISPFCL >_ USER.Z31B.PROCLIB
>_ FEU.Z31B.PROCLIB
Member: ISPFPROC >_ ADCD.Z31B.PROCLIB
>_ ISM403.SFMNEXEC
>_ AUT430.SINGREXX
>_ SYSUADS SYS1.UADS
>_ SYSUDUMP ---------- JES2 Subsystem file -------------

The member is found in 4 places. You can browse a member by entering B in the >_

The first ISPFCL member has

PROC 0 VOL(B3SYS1)                                                       
CONTROL NOMSG NOFLUSH ASIS
PROFILE NOMODE MSGID PROMPT INTERCOM WTPMSG
WRITE *****************************************************************
...
FREE FILE(ISPPROF ISPTABL)
SET &SDSFTAB= &STR(&SYSUID..SDSF.ISFTABL)
ALLOC DA('&SDSFTAB') SHR FILE(ISFTABL)

SET &DSNAME = &STR(&SYSUID..&SYSNAME..ISPF.ISPPROF)
ALLOC DA('&DSNAME') SHR FILE(ISPPROF)
ALLOC DA('&DSNAME') SHR FILE(ISPTABL)
IF &LASTCC ¬= 0 THEN DO
/* Allocate the ISPF Prof dataset */
...
END
  • The FREE FILE(ISPPROF ISPTABL) says drop (ignore) the existing definitions for ISPPROF and ISPTABL. The CLIST will reallocate them.
  • The ALLOC DA(‘&DSNAME’) SHR FILE(ISPTABL) allocates my dataset to the ISPTABL ddname.
  • The problem is that you cannot easily concatenate my data sets to the ISPTLIB concatenation. You can use the TSO ALLOCate command to allocate a list of data sets to a DDNAME, but not just to add one data set to an existing allocated DDNAME. See Adding a data set to an existing DDNAME in TSO.

Starting ISPF

When you logon to the TSO Logon panel it has

Command   ===> ex 'colin.zlogon.clist'       

The command (if specified) will be processed after any command found in the PARM field of the EXEC JCL statement in your logon procedure.

You can specify ISPF, a clist, or other command.
If you want to invoke ISPF from your clist you will need to invoke the ISPF command for example

/* Rexx */                                                        
trace r
say "in colin.zlogon.clist"
address TSO

"alloc fi(ISPTLIB) DA('COLIN.S0W1.ISPF.ISPPROF') SHR "
zl =userid.SDSF.isftabl /* so we get colin.zlogon.clist */
if SYSDSN(zl) = OK then
do
"alloc fi(isftabl) da('"zl"') shr reus"
end
req = "ALLOC FI(tmp) DA('COLIN.S0W1.ISPF.ISPPROF') SHR "
if bpxwdyn(req ) =0 then
call bpxwdyn "concat ddlist(ISPTLIB,tmp) "
"ispf"

With this, ISPF starts with my data sets allocated as I want them!