Linux mapping the keyboard and on z/OS SSH

I wanted to configure the keyboard in zsh and bash on z/OS so the delete key (top right of the keyboard) deleted the current character. (Backspace deletes the previous character) I could not find any good documentation on how to do it.

I found some bash for Linux documentation and have based this blog post on that. I’ve given the command I used, then sections on the background of how it works.

Note the binding of key to action depends on the shell. The shells zsf and bash support binding. shell does/may not.

Having written the original blog post, I found some additional useful information which will make the document clearer.

How to set the delete key to delete

With my Bash shell, I initially had to do it in stages:

  • Type bind “ that’s bind space double quote
  • Control+V, delete key to insert the value of the key. This made it bind “^[[3~
  • Type “:delete-char with no space after the :
  • It now looks like bind “^[[3~”:delete-char
  • Press enter
  • The delete key should delete the character under the cursor

The command bind -p |grep delete gave

"\C-h": backward-delete-char
"\C-?": backward-delete-char
"\C-d": delete-char
"\M-[3~": delete-char
"\\": delete-char
# delete-char-o

Which shows that Ctrl+d and \\ also deletes the current character.

In a shell (in ASCII) this is (in ISPF edit with hex on)

bind '" ∇3~":delete-char' 
6666222153723666676266672222
29E4072BB3E2A45C545D38127000

Where the incantation is x1b5b337e.

What does the mapping mean?

It took me many hours of looking for a good description of what the key mapping is.

The Wikipedia page ANSI escape codes, was very helpful and clear.

  • You can press the “a” key – and “a” is displayed in the command window.
  • You can press shift “a” – and “A” is displayed in the command window
  • You can press Control (Ctrl) “a” – and this may do something
  • You can press Meta “the Windows” key “a” and this may do something
  • You can press Alt + “a” and it may do something.
  • You can press combinations of those keys, such as Ctrl+shift + “a” and this may do something.

The operating system may intercept the key and not pass it to the window. For example on my Linux, Ctrl+Alt+Delete is logout.

In the command window the shell code will intercept a key press combination and take an action, for example move the cursor left, clear the screen, delete word, or just the the character.

You can change the mapping using the bind or bindkeys command for bash and zsf shells.

You can find the code for a key combination by using ctrl+v. For example with the left cursor key

Ctrl+v, < gives ^[[D

How to decode ^[[D ?

There are several formats.

  • <esc>[ number(;modifier)~
  • <esc>[ letters

The string starts with an escape sequence. ^[ This can be written as \e ,\033,  0x1b, or ^[. Where \0 is octal 33, which is hex 1B, (and decimal 27). When specifying a key sequence any of the values can be used. I mentioned above the incantation x1b5b337e.

Because there is no trailing ~ we need to lookup the “[D” in the Wikipedia page section xterm sequences. This has

<esc>[D     - Left 

So for ^[[D we have <esc>, Left key pressed

For ^[[3;5~ we have

  • ^[ is escape
  • [3 isDelete key
  • ; is a modifier
  • 5 is Control + shift
  • ~ end of escape

so the key presses was Control + Delete key (top right on my keyboard). Shift seems to be ignored!

Another popular key is ^M which is Carriage Return – (and process any data on the line) and move the cursor to column 0.

\C is the Ctrl key, \M is the meta key. For keyboards without the Meta key this is displayed as Escape \e.

Keys like \M-OD comes from

Keypad mode
"\M-OD":backward-char
"\M-OC":forward-char
"\M-OA":previous-history
"\M-OB":next-history

This information is not well documented.

What is key mapping

On Linux Ubuntu the control-right arrow combination moves the cursor right one word. The control-left arrow moves the cursor left one word. Control-L clears the screen, but leaves the command line intact.

You can list the possible actions

bind - l

This gave

...
arrow-key-prefix
backward-byte
backward-char
backward-delete-char
backward-kill-line
backward-kill-word
...

Get a list of functions and what keys use them

With upper case -P

bind -P |grep backward

This gave

backward-byte is not bound to any keys
backward-char can be found on "\C-b", "\eOD", "\e[D".
backward-delete-char can be found on "\C-h", "\C-?".

Where does \eOD come from ? See here.

Get a list of keys and their functions in bind format

with lower case -p

bind -p |grep backward

gave

"\eOD": backward-char
"\e[D": backward-char
"\C-h": backward-delete-char
"\C-?": backward-delete-char
...

where

  • Control-h moves the cursor left one character and deletes the character
  • \e is the escape key. The alt key is usually mapped to the escape key by terminal emulators.

This output is slightly untrue. If there is no blank after the : you can use bind….

If you have a blank after the : you need to enclose it in single quotes.

bind ‘”\C-h”: backward-delete-char’

to set the key.

What is the code for a key press combination?

The Linux command Ctrl-V (verbatim insert) followed by a key, gives the mapping for that key.

Ctrl+V LeftArrow
^[[D

Ctrl+V Ctrl+Leftarrrow
^[[1;5D

Where ^[ means the escape key and [D is OD.

^[[3~ is escape Delete key.

Verbatim insert gives the code of the key which was pressed. This works on z/OS if you have the Bash or zsf shell installed.

What keys are mapped?

In the zsh shell you can issue

bindkey -L

(With bash you can use the bind command).

This gives output like

bindkey "^E" end-of-line
...
bindkey "^L" clear-screen

Where ^ is the ctrl key.

If you type (on Linux) man bash , and locate Readline Command Names if lists the function names and what they mean.

The bash command bind -l lists all of the functions

$ bind -q backward-char
backward-char can be invoked via "\C-b", "\M-OD", "\M-[D".

It gets very confusing

A sequence can be created in different formats. For example many commands begin with the Escape (or Meta key). This can be written as \e ,\033,  0x1b, or ^[. Where \0 is octal 33 which is hex 1B, ( or decimal 27). It is confusing when you display information with different commands.

Where does \eOD (or \M-OD)come from?

It was a challenge to find this information.

In the Linux terminfo documentation it says

The codes sent by the left arrow, right arrow, up arrow, down arrow, and home keys can be given as kcub1, kcuf1, kcuu1, kcud1, and khome respectively. If there are function keys such as f0, f1, …, f10, the codes they send can be given as kf0, kf1, …, kf10. If these keys have labels other than the default f0 through f10, the labels can be given as lf0, lf1, …, lf10.

The linux command infocmp

infocmp|grep kcu 

gave

kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,

so this all says the left arrow key maps to backward character. – Phew

Problems

If there was a space after the : I got

readline: delete-char: no key sequence terminator

from the bind command.

Putting the command into my .profile file didn’t work because of the wrong character set.

I could put the command into a shell script, and invoke it – but I could not get it to work from the .profile.

Why does one ping work, and the same ping doesn’t?

I was trying to check connectivity from z/OS running on my laptop. For some remote sites I could issue ping and get a response back. For some other sites I issue the ping and did not get a response back.

When I issued the pings from Linux – they both worked.

I noticed that for the pings from z/OS the field Timestamp from icmp data (relative): was 27 seconds behind. This was caused by z/OS adding leap seconds. It was not the problem.

By comparing all the fields in a successful and an unsuccessful ping, I could see that z/OS send 256 bytes of data, and Linux sent only 40 bytes of data.

On Linux, when I used

ping …. -s 256

it failed. When I used

ping …. -s 20

it worked.

Similarly on z/OS.

ping .... (length 20

The defaults lengths are different between z/OS and Linux.

The moral of this tale is

If ping does not return any data – try a very short ping.

Creating a ZFS – which way should I do it? IDCAMS LINEAR or IDCAMS ZFS?

When I looked into creating a ZFS (so I could run use it in the Unix environment) I found there were two ways of doing it both have the same end result.

The “old” way – a three step process

You use DEFINE CLUSTER …LINEAR to create the data set, then use PGM=IOEAGFMT to format it, then mount it.

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE COLIN.ZOPEN.ZFS CLUSTER
SET MAXCC=0
DEFINE -
CLUSTER -
(NAME(COLIN.ZOPEN.ZFS)-
LINEAR -
VOLUMES(USER10 ) -
STORCLAS(SGBASE ) -
MEGABYTES(6000 1000) -
SHAREOPTIONS(3 3))
/* -
//FORMATFS EXEC PGM=IOEAGFMT,REGION=0M,COND=(0,NE,DEFINE),
// PARM=('-aggregate COLIN.ZOPEN.ZFS ')
//* PARM=('-aggregate COLIN.ZOPEN.ZFS -compat')
//SYSPRINT DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//*
//*
//MOUNT EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE),(0,NE,FORMATFS))
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('COLIN.ZOPEN.ZFS') TYPE(ZFS) +
MOUNTPOINT('/u/zopen') +
MODE(RDWR) PARM('AGGRGROW') AUTOMOVE
/*

The define took less than a second, the format took about 16 seconds, and the mount took less than one second

The Two step (sounds like a dance for system administrators)

You create the dataset with type ZFS, you then mount it, and the mount formats it.

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DELETE COLIN.ZOPEN.ZFS CLUSTER
SET MAXCC=0
DEFINE -
CLUSTER -
(NAME(COLIN.ZOPEN.ZFS)-
ZFS -
VOLUMES(USER10 ) -
STORCLAS(SGBASE ) -
MEGABYTES(6000 1000) -
SHAREOPTIONS(3 3))
/*
//MOUNT EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE))
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('COLIN.ZOPEN.ZFS') TYPE(ZFS) +
MOUNTPOINT('/u/zopen') +
MODE(RDWR) PARM('AGGRGROW') AUTOMOVE
/*

The define took less than a second – the mount took 17 seconds, because it had to do the format.

What’s the difference?

Overall the time to execute the job was the same.

I think I prefer the first way of doing it, as I have more control, and can check the format was as I expected.

If you used the second way of doing it, defined the ZFS in parmlib, I don’t know if the formatting would hold up OMVS startup.

And don’t forget

And do not forget to update your parmlib member so the ZFS is mounted automatically at IPL.

Wow I can have member generations

We have had the capability of having multiple generations of data sets on z/OS for years.
For example with three generations you can have

  • the current data set
  • the one before that
  • and the one before that.

If you create a new data set the oldest gets deleted, and they all move along one.

This has been around for years.
What I found recently was you can have this with members within a V2 PDSE (not a PDS) since 2015.

System wide set up

When you create the data set, the number of generations is limited by MAXGENS_LIMIT in the IGDSMSxx member of PARMLIB.

Use the command

D SMS,OPTIONS

This displays information like

ACDS     = SYS1.S0W1.ACDS               
COMMDS = SYS1.S0W1.COMMDS
ACDS LEVEL = z/OS V3.1
SMS PARMLIB MEMBER NAME = IGDSMS00
...
HONOR_DSNTYPE_PDSE = NO PDSE_VERSION = 2
USER_ACSVAR = ( , , ) BYPASS_CLUSTER_PREFERENCING = NO
USE_MOST_FREE_SPACE = NO MAXGENS_LIMIT = 0

To change the value of MAXGENS_LIMIT you need to change the parmlib member and use T SMS=nn (or just wait till the next IPL).

I used Which parmlib/proclib library has my member? to find the member and added

 MAXGENS_LIMIT(3)        

I then used

set sms=00

to activate it

Using the support

For example to allocate a dataset to support this.

Example JCL

//SAM00001 DD DISP=(NEW,CATLG),DSN=IBMUSER.TEST1.PDSE00,
// DSNTYPE=(LIBRARY,2),LRECL=80,BLKSIZE=8000,RECFM=FB,
// MAXGENS=3

Where

  • dsntype=(LIBRARY,2) says this a LIBRARY ( also known as PDSE) type 2
  • MAXGENS=3 this will support up to 3 generation

Using ISPF

This works in z/OS 3.1, I do not know if earlier releases have the ISPF support.

I used ISPF 3.2 and specified

──────────────────────────────────────────────────────────────────────────────
Allocate New Data Set
Command ===>

Data Set Name . . . : COLIN.PDSE2

Management class . . . (Blank for default management class)
...
Data set name type LIBRARY (LIBRARY, PDS, LARGE, BASIC, *
EXTREQ, EXTPREF or blank)
Data set version . : 2
Num of generations : 3
Extended Attributes (NO, OPT or blank)
Expiration date . . . (YY/MM/DD, YYYY/MM/DD
YY.DDD, YYYY.DDD in Julian form
DDDD for retention period in days
or blank)

I then edited a member, saved it, and then reedited it several times.

ISPF 3;4 member list gave

DSLIST            COLIN.PDSE2                           Row 0000001 of 0000001
Command ===> Scroll ===> CSR
Name Prompt Size Created Changed ID
_________ AA 2 2025/11/09 2025/11/09 09:22:35 COLIN

Using the line command b to browse the data set showed the latest content.

Using the line command N gave me

GENLIST           (AA)COLIN.PDSE2                       Row 0000001 of 0000004
Command ===> Scroll ===> CSR
RGEN Prompt Size Created Changed ID
_ 00000000 5 2025/11/09 2025/11/09 09:31:44 COLIN
_ 00000001 4 2025/11/09 2025/11/09 09:31:32 COLIN
_ 00000002 3 2025/11/09 2025/11/09 09:31:17 COLIN
_ 00000003 2 2025/11/09 2025/11/09 09:22:35 COLIN

There is

  • (AA)COLIN.PDSE2 showing member AA of data set ( library) COLIN.PDSE2
  • RGEN showing the generations
  • Generation 3 is the oldest

In the line command you can type / which lists all of the valid commands

           Action for Generation 0              

Generation Action
1. Edit
2. View
3. Browse
4. Delete
5. Info
6. Print

Prompt Action . . (For prompt field)

Select a choice and press ENTER to continue

Info gave me

   Menu  Functions  Confirm  Utilities  Help  
─────────────────────────────────────────────
EDIT USER.Z31B.PARMLIB
. . . . . . . . . . . . . . .
Member Informat
Command ===>

Data Set Name . . . : COLIN.PDSE2

General Data
Member name . . . . . . . . : AA
Concatenation number . . . . : 1
Version . Modification . . . : 01.07
...

Non-current Generations
Maximum . . . . : 3
Saved . . . . . : 3
Newest Absolute : 7
Oldest Absolute : 5

See Version and modification level numbers. You can use the commands

To set these values

Deleting a member

Using the D line command against the oldest member gave the prompt

           Confirm Member Generation Delete          

Data Set Name:
COLIN.PDSE2

Member Name:
AA

Generation to be Deleted:
-3

__Set member generation delete confirmation off

Only the specified generation will be deleted.

Press ENTER to confirm delete.
Press CANCEL or EXIT to cancel delete.

Editing a member

When you edit a member the screen is like

   File  Edit  Edit_Settings  Menu  Utilities  Compilers  Test  Help
────────────────────────────────────────────────────────────────────
EDIT COLIN.PDSE2(AA) - 01.04
Command ===>
****** ********************************* Top of Data ***************
000100 ddd
000200 bbbb
000300 44444

with the version. release at the top.

The future is already here. GS UK 2025

I was at the GS UK conference recently (which was bigger than ever) and learned so many new things.

I’ll give some short descriptions of what I learned. There is no order to these topics. Some items come in more than one area.

  • Python is very popular and widely used.
    • pyracf for issuing RACF commands from OMVS
    • pysdsf accessing SDSF from OMVS (z/OS 3.2)
  • Use ssh to access z/OS instead of ISPF
    • Many Unixy commands ported to z/OS through zopen project
    • /dfds to access data sets
    • Possibly faster than through ISPF
  • vscode is the most commonly used IDE with lots of plugins. Can edit z/OS data sets, files, submit jobs and look at spool – via Zowe
    • Git is the standard repository
    • Edit in vscode
    • check-in to Git
    • on z/os pull from Git
    • compile and run from ssh window
    • can edit on your workstation and process on z/OS
    • use Zowe/vscode to edit datasets and files in vscode, submit JCL and look at the spool. Can use zowe command line interface for issuing stuff to z/OS ( eg list files, issue operator commands)
  • People like my blog posts – Wow ! I never really knew. If you like/use anyone’s post please “like it” so the author knows. If it has been really helpful make a comment “I found this very useful”. Steven P. pointed out that you need o be logged on to a WordPress account to be able to “like” or raise a comment – this would explain why I got so few likes!
  • Lots of capturing data and displaying it in tools like grafana.
    • Python used to capture data
  • Monitoring dashboards are so last year.
    • Now have modern tools (AI) to detect changes from historical data, then alert people to differences, who then use the dashboards.
  • SDSF version 3.2 can intercept RACF writes to SMF and can display the activity, so if RACF is configured you can display OK access to resources. You just get the failures reported on syslog
  • You’ve been hacked
    • Often there is evidence months before hack – you just need to spot it
      • Pat is a z/OS sysprog who comes to work, has a coffee and starts working at 0930. Today there were two password validation failures at 0430. Is this unusual – yes. Do something
      • The password failures occurred at 0925 and 0926 – is this unusual.. you might want to check
      • You had a connection from an IP address you’ve never seen before – what do you do? Slow down their traffic and notify someone
    • Prepare for this
      • Have an integrated playbook.
        • Populate panels with the suspicious userid, and have a pull down to disable. It takes longer to type data into a RACF command than use from pre populated fields. (Eg userid COLIN Click here to disable it. )
        • Have people use the play book so they know what to do when an incident occurs. You do not have time to learn as you go along.
      • You have minutes to act. Getting someone out of bed is too long.
    • What software is running where? File Integrity Monitoring
      • I thought this was module ABC CSECT CS123 PTF level UK987654. No. If someone has zapped the module how do you know? And when did they do it? This helps you know how far you need to restore from,
      • Take each module and CSECT and create an encrypted checksum for each component. Store the information system id/library/module/CSECT/hash code. Check it weekly If someone has zapped the module – it will have a different hash. You can also see which systems are back level.
      • Do the same for configuration files eg parmlib.
      • If it has changed there should be a matching change request.
  • Regulations are in. If you have hacker insurance you will have to comply with regulation standards- such as
    • have you implemented File Integrity Monitoring (above).
    • Do you follow the standards for isolation of responsibilities.
    • “Yes” is the wrong answer. You need to demonstrate proof.
      • eg password strength. You need tests to validate it
      • prove this TCPIP connection is encrypted
  • Certificates should be reissued every 30-60 days. Not the n years it used to be.
  • OpenTelemetry tracing system. Configure applications and subsystems, to emit “here I am” to central monitoring to show path transaction took. Eg MQ client, into CICS transaction… to another CICS and back. Can do it for all – or just a sample of some requests.

  • Lots of youngster involved with z/OS.
  • Lots of old familiar faces who love working with z/OS, and should have retired years ago . This includes me (grin).

The Like box below only works if you are logged on to wordPress.

You should be able to click on the stars and give a score without logging on

Go back

Your message has been sent

Rating(required)
Warning

Using ISMF and looking at volumes

The ISMF ISPF dialogs provide a good way of displaying information about SMS,volumes data sets etc.

Like many tools – it is easy once you know how to use it.

Where is there free disk space?

Using the ISMF dialogs (as a storage administrator)

                  ISMF PRIMARY OPTION MENU - z/OS DFSMS 3.1       
Selection or Command ===>


0 ISMF Profile - Specify ISMF User Profile
1 Data Set - Perform Functions Against Data Sets
2 Volume - Perform Functions Against Volumes

Option 2 Volume –>, 1 for DASD

                          VOLUME SELECTION ENTRY PANEL              Page 1 of 3
Command ===>

Select Source to Generate Volume List . . 2 (1 - Saved list, 2 - New list)
1 Generate from a Saved List Query Name To
List Name . . COLIN Save or Retrieve
2 Generate a New List from Criteria Below
Specify Source of the New List . . 1 (1 - Physical, 2 - SMS)
Optionally Specify One or More:
Enter "/" to select option Generate Exclusive list
Type of Volume List . . . 1 (1-Online,2-Not Online,3-Either)
Volume Serial Number . . USER* (fully or partially specified)
Device Type . . . . . . . (fully or partially specified)
Device Number . . . . . . (fully specified)
To Device Number . . . (for range of devices)
Acquire Physical Data . . Y (Y or N)
Acquire Space Data . . . Y (Y or N)
Storage Group Name . . . * (fully or partially specified)
CDS Name . . . . . . . 'ACTIVE'

Where Acquire Physical Data obtains

  • Device type such as 3390-3
  • Device number such as 0a94
  • Shared DASD such as YES
  • Use Attributes such as PRIV

and Acquire Space Data obtains and calculates

  • Free space
  • Free extents,
  • % Free index status
  • Allocated space
  • Free DSCBs
  • Fragmentation Index
  • Free VIRs (VTOC Index Records)
  • Largest extent
  • Physical status

gave

                                                           VOLUME LIST           
Command ===> Scroll ===> HALF
Entries 1-8 of 8
Enter Line Operators below: Data Columns 3-8 of 45

LINE VOLUME FREE % ALLOC FRAG LARGEST FREE
OPERATOR SERIAL SPACE FREE SPACE INDEX EXTENT EXTENTS
---(1)---- -(2)-- ---(3)--- (4)- ---(5)--- -(6)- ---(7)--- --(8)--
USER00 81233K 1 8233268K 674 6308K 135
USER02 100822K 4 2670678K 252 53953K 31
USER03 5202K 0 2766298K 483 2213K 9
USER04 1457936K 53 1313564K 49 1328284K 20
USER05 78577K 3 2692923K 83 63802K 4
USER06 289020K 10 2482480K 179 105581K 16
USER07 500735K 18 2270765K 92 356142K 11
USER08 953881K 34 1817619K 86 509478K 3
---------- ------ ----------- BOTTOM OF DATA ----------- ------ ----

If you press PF1 for help, press “enter to continue” and select option 6 Data Column Descriptions it gives all of the columns names. They have a name and number.

Back in the data display, you can scroll sideways to display these field.

You can use the command view 3,23 to display just those fields, or view * to display all of the fields.

You can use view save name and view select name to save the current view configuration as name.

I could not find a way of displaying all of the attributes for a volume in a vertical list.

Sort

You can say sort 3 to sort by column 3 ascending, or just sort where you can specify the columns and ascending or descending.

The columns you can sort by depend on what are displayed via the view command.

How difficult is it to delete a data set – it can be harder than you think.

I was using a volume on one zD&T system, and used the same volume on the next version of zD&T. Some of the datasets are SMS managed, and were cataloged in the first system’s master catalog.

Now I’m on the latest level of zD&T, I am trying to delete unwanted data sets, and this was a challenge, as all of the usual methods did not work.

I can use ISPF 3.4 and specify the volume USER05. There are some old datasets, such as SYS1.S0W1.Z24C.DMP00001. These are not cataloged. They are only visible in ISPF 3.4 if you specify the volume. If you try to browse the data set it gives data set not cataloged.

I failed to delete the dataset using the usual techniques….

From ISPF

If you try to delete it from ISPF 3.4 it says “Data set not cataloged”,

From JCL

For example

//IBMDEL   JOB 1,MSGCLASS=H 
//TSO EXEC PGM=IEFBR14
//DDS DD DSN=SYS1.S0W1.Z24C.DMP00001,VOL=SER=USER05,
// DISP=(OLD,DELETE)
/*

This fails with the error message

IEFA107I IBMDEL TSO DDS – DATA SET SYS1.S0W1.Z24C.DMP00001 NOT FOUND

From IDCAMS

//IBMDEL2  JOB 
//STEP1 EXEC PGM=IDCAMS
//DD1 DD VOL=SER=USER05,UNIT=3390,DISP=OLD
//* DSN=SYS1.S0W1.Z24C.DMP00003
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
DELETE -
SYS1.S0W1.Z24C.DMP00001 -
FILE (DD1) -
PURGE
/*

Fails with

IDC3012I ENTRY SYS1.S0W1.Z24C.DMP00001 NOT FOUND
IDC3009I ** VSAM CATALOG RETURN CODE IS 8 – REASON CODE IS IGG0CLEG-42

The problem…

The dataset is on an SMS managed volume. All data sets managed by SMS have to be cataloged. The catalog that owns the dataset is on a non existant system – the old master catalog from the system from 3 years ago. The outcome is that there is no catalog available to locate the data set.

On the SMS volume USER05 is the SMS dataset SYS1.VVDS.VUSER05. This is like a catalog for the volume, and contains information about all the data sets on the SMS volume. You can use IDCAMS to print this data set (in hex).

The solution…

You need to recatalog it – then delete it

recatalog

//IBMRECAT JOB 
//RECATSMS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE NONVSAM(NAME(SYS1.S0W1.Z24C.DMP00001 ) -
OWNER(COLIN ) -
VOLUMES(USER05) -
DEVT(3390 ) -
RECATALOG)
LISTCAT ENT(SYS1.S0W1.Z24C.DMP00001) ALL
/*

This gave

  DEFINE NONVSAM(NAME(SYS1.S0W1.Z24C.DMP00001   ) -                                           
OWNER(COLIN ) -
VOLUMES(USER05) -
DEVT(3390 ) -
RECATALOG)
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

LISTCAT ENT(SYS1.S0W1.Z24C.DMP00001) ALL
NONVSAM ------- SYS1.S0W1.Z24C.DMP00001
IN-CAT --- CATALOG.Z31B.MASTER
HISTORY
DATASET-OWNER------COLIN CREATION--------2025.296
RELEASE----------------2 EXPIRATION------0000.000
ACCOUNT-INFO-----------------------------------(NULL)
SMSDATA
STORAGECLASS -----SCBASE MANAGEMENTCLASS---(NULL)
DATACLASS --------(NULL) LBACKUP ---0000.000.0000
ENCRYPTIONDATA
DATA SET ENCRYPTION-----(NO)
VOLUMES
VOLSER------------USER05 DEVTYPE------X'3010200F' FSEQN------------------0
ASSOCIATIONS--------(NULL)
ATTRIBUTES

Having done this, I can now browse SYS1.S0W1.Z24C.DMP00001 in ISPF without specifying a volume, so proving it is now cataloged.

Delete it

If I now try to delete it I get

IEC331I 050-088(,USER05),COLIN,ISPFPROC,VCMP,IGG0CLE4
IEC614I SCRATCH FAILED – RC 008, DIAGNOSTIC INFORMATION IS (040942D1),
ISPFPROC,USER05,SYS1.S0W1.Z24C.DMP00001

The codes for IEC331I are given in IEC3009I. If you search for “code 50” then page down till you get code 88, it says

Explanation: A VVR or NVR with the correct component name was found, but the catalog name did not match. On a delete request, the BCS record will be deleted, but the VVR or NVR and the format 1 DSCB will not be scratched. There is no SFI data.

I had to use

//IBMDEL2  JOB 
//STEP1 EXEC PGM=IDCAMS
//DD1 DD VOL=SER=USER05,UNIT=3390,DISP=OLD
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
DELETE SYS1.S0W1.Z24C.DMP00001 NVR FILE(DD1)
/*

Which says delete the data set (and remove it from the VVDS dataset).
This job was successful, and the data set was no longer on the volume.

Sometimes I had to remove the NVR from the delete statement.
My overall JCL was

//IBMDEL   JOB  1 
//E1 EXPORT SYMLIST=*
// SET NAME='IZUSVR.JVM.IZUSVR1.D250809.T080903.X001'
// SET VOL='USER06'
//RECATSMS EXEC PGM=IDCAMS
//DD1 DD VOL=SER=&VOL,UNIT=3390,DISP=OLD
//SYSPRINT DD SYSOUT=*
//SYSIN DD *,SYMBOLS=JCLONLY

DEFINE NONVSAM(NAME(&NAME.) -
OWNER(COLIN ) -
VOLUMES(&VOL.) -
DEVT(3390 ) -
RECATALOG)

LISTCAT ENT(&NAME. ) ALL

DELETE &NAME. NVR FILE(DD1)

DELETE &NAME. FILE(DD1)
/*

Configuring and using the RMF GPM Server

RMF provides information on the usage of system resources, such as CPU, Channel usage, Disk response time etc. You can get reports from an attached 3270 screen, from a web server, and from a REST request.

For the web server and REST requests, you need the GPM server running. It took me a while to get this running, and to get useful data out of it.

GPMServer uses basic authority checking of userid and password. Alternatively it can use certificates from the client to authenticate on z/OS.

There are two versions of GPMSERVE. It looks like the newer one is written in Java. I only have access to the old version.

GPM Setup

I used

//GPMSERVE PROC MEMBER=00 
//STEP1 EXEC PGM=GPMDDSRV,REGION=128M,TIME=1440,
// PARM='TRAP(ON)/&MEMBER'
//* PARM='TRAP(ON),ENVAR(ICLUI_TRACETO=STDERR)/&MEMBER'
//*
//*STEPLIB DD DISP=SHR,DSN=CEE.SCEERUN
//* DD DISP=SHR,DSN=CBC.SCLBDLL
//GPMINI DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMINI)
//GPMHTC DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMHTC)
//GPMPPJCL DD DISP=SHR,DSN=SYS1.SERBPWSV(GPMPPJCL)
//CEEDUMP DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
// PEND

CACHESLOTS(4)                   /* Number of timestamps in CACHE     */ 
DEBUG_LEVEL(3) /* informational messages */
SERVERHOST(10.1.1.2)
HTTPS(ATTLS) /* AT-TLS setup required */
MAXSESSIONS_HTTP(20) /* MaxNo of concurrent HTTP requests */
HTTP_PORT(8803) /* Port number for HTTP requests */
HTTP_ALLOW(*) /* Mask for hosts that are allowed */
HTTP_NOAUTH() /* No server can access without auth.*/
CLIENT_CERT(NONE)
/* CLIENT_CERT(ACCEPT) */

The essence of my AT-TLS definitions is (from my Easy-ATTLS)

LocalPortRange : 8803
Direction : Both
ApplicationControlled : Off
TTLSEnabled : On
CtraceClearText : On
Trace : 2
HandshakeRole : Server
Keyring : start1/TN3270
TLSv1.1 : Off
TLSv1.2 : On
TLSv1.3 : Off
HandshakeTimeout : 3
ClientECurves : Any
ServerCertificateLabel : NISTECCTEST
V3CipherSuites : [
1302 TLS_AES_256_GCM_SHA384,
1301 TLS_AES_128_GCM_SHA256,
003D TLS_RSA_WITH_AES_256_CBC_SHA256,
C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
]

I used CtraceClearText : On so I could trace the flows and see the encrypted traffic.

The Chrome browser used ECDHE* cipher specs. I had specified C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, and I could this was being used.

The Chrome browser prompted for userid and password which was passed up to the server.

Issuing commands

You start the server with

S GPMSERVE

If it abends with

IEF450I GPMSERVE GPMSERVE - ABEND=S0C4 U0000 REASON=00000011

Check RMF is active. And check you have issued F RMF,START III to start the data collection.

You stop the server

p gpmserve

You can display information about the server

f gpmserve,display

The newer version of GPMSERVE uses commands like F GPMSERVE,APPL=DISPLAY

The output is like

+GPM062I DDS-REFR 01/02 084125 CYCLE=314. WAITING 10 SEC
+GPM062I HTTP-LIS 01/02 084119 MAX=20 ACTIVE=0 SUSPEND=1
+GPM062I RMF_DDS_ATTLS 01/02 074900 STARTING …
+GPM062I RMF_DDS_OPTS 01/02 074900 STARTING …
+GPM062I HTTP-CLI 01/02 083219 ::FFFF:10.1.0.2 TERMINATED. SUSPENDED.

Where 01/02 is Jan 2nd. 074900 is 07:49:00

Certificate and keyring set up

I reused an existing keyring. The AT-TLS definitions give the keyring is start1/TN3270 and the certificate to use is NISTECCTEST.

List the ring contents

tso RACDCERT listring(TN3270) id(START1)

The keyring included the CA for my NISTECCTEST certificate, and the CA for the client’s certificate (on Linux).

My certificate authentication to work, I needed the client certificate connected to the keyring.

On Linux I had

  • ca256.pem the Certificate Authority
  • colinpaice.pem

I FTPed these to z/OS as VB data sets, COLIN.CA256.PEM, and COLIN.PAICE.PEM.

Import the CA into z/OS

//IBMRACFI JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.CA256.PEM')
RACDCERT DELETE -
(LABEL('CA256')) CERTAUTH
RACDCERT CERTAUTH ADD('COLIN.CA256.PEM') -
WITHLABEL('CA256') TRUST
RACDCERT CERTAUTH LISTCHAIN(LABEL('CA256'))

RACDCERT CONNECT(CERTAUTH LABEL('CA256') -
RING(TN3270) ) ID(START1)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH
/*

and import the users .pem file.

//IBMRACFI JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RACDCERT CHECKCERT('COLIN.PAICE.PEM')
RACDCERT DELETE -
(LABEL('RMFCERT')) ID(COLIN)
RACDCERT ID(COLIN) ADD('COLIN.PAICE.PEM') -
WITHLABEL('RMFCERT') TRUST
RACDCERT ID(COLIN) LISTCHAIN(LABEL('RMFCERT'))
RACDCERT ID(START1) CONNECT(ID(COLIN ) LABEL('RMFCERT') -
RING(TN3270))
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH
/*

When a user connects with a certificate, GPMSERVE looks in the keyring for the passed certificate, and finds the userid for it.

Setting up the security profiles

You need to set up a CLASS(APPL) profile for GPMSERVE. Give any authorised userids read access to the profile.

//IBMRACF  JOB 1,MSGCLASS=H 
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
* Delete and redefine the profile
* List it first
RLIST APPL GPMSERVE authuser
RDELETE APPL GPMSERVE
SETROPTS RACLIST(APPL) refresh
RDEFINE APPL GPMSERVE UACC(NONE) NOTIFY(COLIN)
PERMIT GPMSERVE CLASS(APPL) ID(IBMUSER) ACCESS(READ)
PERMIT GPMSERVE CLASS(APPL) ID(COLIN ) ACCESS(READ)
PERMIT GPMSERVE CLASS(APPL) ID(ADCDB ) ACCESS(NONE)
SETROPTS RACLIST(APPL) refresh
RLIST APPL GPMSERVE authuser
SETROPTS RACLIST(APPL) refresh
/*

I specified RDEFINE APPL GPMSERVE UACC(NONE) NOTIFY(COLIN) so the userid COLIN gets notified if anyone tries to use the profile and fails. Using WARNING does not work.

Changing security

If you give a userid read permission to the CLASS(APPL) GPMSERVE profile, you need to stop and restart GPMSERVE to pick up the changes. It looks like GPMSERVE caches the access after first use, and there is no refresh security command.

When tracing a job it helps to trace the correct address space.

The title When tracing a job it helps to trace the correct address space is a clue – it looks obvious, but the problem was actually subtle.

The scenario

I was testing the new version of Zowe, and one of the components failed to start because it could not find a keyring. Other components could find it ok. I did a RACF trace and there were no records. The question is why were there no records?

The execution environment.

I start Zowe with S ZOWE33. This spawns some processes such as ZOWE335. This runs a Bash script which starts a Java program.

I start a GTF trace with

s gtf.gtf,m=gtfracf
#set trace(callable(type(41)),jobname(Zowe*))

Where callable type 41 is for r_datalib services to access a keyring.

No records were produced

What is the problem?
Have a few minute pause to think about it.

Solution

After 3 days I stumbled on the solution – having noticed, but ignored the evidence. I wondered if the Java code to process keyrings, did not use the R_datalib API, I wondered if Java 21 uses a different jar file for processing keyrings – yes – but this didn’t solve the problem.

The solution was I should have been tracing job ZWE33CS! Whoa – where did that come from?

The Java program was started with

_BPX_JOBNAME=ZWE33CS /usr/lpp/java/J21.0_64/bin/java

See here which says

When a new z/OS® UNIX process is started, it runs in a z/OS UNIX initiator (a BPXAS address space). By default, this address space has an assigned job name of userIDx, where userID is the user ID that started the process, and x is a decimal number. You can use the _BPX_JOBNAME environment variable to set the job name of the new process. Assigning a unique job name to each … process helps to identify the purpose of the process and makes it easier to group processes into a WLM service class.

If I use the command D A,L it lists all of the address spaces running on the system. I had seen the ZOWE33* ones, and also the ZWE* ones – but ignored the ZWE* ones. Once I knew the solution is was so obvious.

What is my Unix process doing?

I was familiar with the USS command ps -ef which displays output like

     UID        PID       PPID  C    STIME TTY       TIME CMD 
WEBSRV 16842766 1 - 07:23:05 ? 0:00 -sh -c /web/httpd1/bin/apachectl -k start -f /web/httpd1/conf/httpd.conf -DNO_f

For Zowe threads I was getting

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/inter

which was annoyingly truncated.

The command ps -e -o args > aa gives the whole command line (up to 1024 bytes) such as

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/internal/start/component/cli.js

Another useful command when you know it.