Category: Z/OS

Why can’t java use my key ring?

I had a problem with z/OSMF. I configured it to use an exiting keyring, but it consistently refused to use it. I had messages like

[WARNING ] CWPKI0809W: There is a failure loading the defaultKeyStore keystore. If an SSL configuration references the defaultKeyStore keystore, then the SSL configuration will fail to initialize.

This blog post covers how I debugged this situation.

What seemed strange was this only occurred when an Elliptic Curve certificate was being used – and not an RSA certificate.

Even more curiouser was the documentation mentioned access to the <ringOwner>.<ringName>.LST resource in the RDATALIB class. See here. I didn’t have this defined and yet RSA certificates would work! So curiouser and curiouser (or for the people who like correct grammar, curiouser and more curiouser).

All applications needing access to certificates and private keys use the R_datalib callable service.

The bottom line

  • z/OSMF has userid IZUSVR
  • I had a keyring and used two certificates
    • An RSA certificate, CCPKeyring.IZUDFLT, belonging to userid IZUSVR – based on the sample JCL provided by z/OSMF
    • An existing Elliptic Curve certificate NISTEC224 belonging to userid COLIN. This works else where.
  • Without <ringOwner>.<ringName>.LST defined the class(RDATALIB) the RSA certificate worked
  • Without <ringOwner>.<ringName>.LST defined the class(RDATALIB) the Elliptic Curve certificate failed
  • Once I found the problem I defined <ringOwner>.<ringName>.LST in class(RDATALIB), and gave the userid IZUSVR Update access to it – and the Elliptic curve worked
  • The reasons (being wise after the event)
    • R_datalib checks access on one profile in the RDATALIB class first – <ringowner>.<ringname>.LST. If there is none, it will fall back to check on two profiles in the FACILITY class – IRR.DIGTCERT.LISTRING and IRR.DIGTCERT.GENCERT. If the certificate is not owned by the accessing ID (except CERTAUTH or SITE), RDATALIB class has to be used for private key access.
    • This is true for the RSA certificate, used the IRRDIGTCERT.LISTRING class(FACILITY) and had access. So this worked.
    • For the Elliptic Curve, the caller’s userid (IZUSVR) is not the associated with the certificate (COLIN) so this fails, and the logic drops through to the RDATALIB checking.
    • The caller’s user ID has READ or UPDATE authority to the ..LST resource in the RDATALIB class. READ access enables retrieving one’s own private key, UPDATE access enables retrieving other’s. The ring did not exist, and so this access was not given.

How did I debug this? – Using Java trace

Adding configuration to z/OSMF

I copied /global/zosmf/configuration/local_override.cfg to /global/zosmf/configuration/local_override.colin

I edited/global/zosmf/configuration/local_override.cfg and changes the JVM options line to

JVM_OPTIONS=”-Xoptionsfile=’/global/zosmf/configuration/local_override.colin'”

I edited the local_override.colin, deleted all but the JVM options line, then split the line at \n so it looks like

-Dcom.ibm.ws.classloading.tcclLockWaitTimeMillis=300000
-Xscmx150M
-Xquickstart

Add debug information to the configuraton file

I added

-Djava.security.auth.debug=pkcs11keystore
-Dlog.level=Error

The output

[err] Jan 17, 2025 8:18:52 AM com.ibm.crypto.ibmjcehybrid.provider.HybridRACFKeyStore engineLoad 
TRACE: Loading keyring CCPKeyring.IZUDFLT as a JCECCARACFKS type keystore. 
...
[err] Jan 17, 2025 8:19:02 AM com.ibm.crypto.hdwrCCA.provider.RACFInputStream getEntry 
FINER: The private key of NISTEC224 is not available or no authority to access the private key 
[err] Jan 17, 2025 8:19:02 AM com.ibm.crypto.ibmjcehybrid.provider.HybridRACFKeyStore engineLoad 
TRACE: Error loading and storing certificates and key material from underlying JCECCARACFKS keyring CCPKeyring.IZUDFLT 
java.io.IOException: The private key of NISTEC224 is not available or no authority to access the private key . This can be expected if the IBMJCECCA is not setup correctly or 
ICSF is down. Will now attempt to load the keyring as a JCERACFKS keyring.

Which is not a very helpful message.

How did I debug this? – Using RACF trace

R_datalib is the callable service to ALL the exploiters which need access to a RACF keyring (certificates and private keys). It is r_datalib or its alias irrsdl00 with callable type number 41.

Enable the RACF trace

#SET TRACE(CALLABLE(TYPE(41))JOBNAME(IZU*))

Start GTF

S GTF.GTF,M=GTFRACF

This reported

IEF403I GTF - STARTED - TIME=08.17.03                                  
IEF188I PROBLEM PROGRAM ATTRIBUTES ASSIGNED                            
AHL121I  TRACE OPTION INPUT INDICATED FROM MEMBER GTFRACF  OF PDS      
USER.Z24C.PROCLIB                                                      
TRACE=USRP                                                             
USR=(F44)                                                              
END                                                                    
AHL103I  TRACE OPTIONS SELECTED --USR=(F44)                            
AHL906I THE OUTPUT BLOCK SIZE OF    27998 WILL BE USED FOR OUTPUT 702  
        DATA SETS:                                                     
          SYS1.TRACE

I started z/OSMF until it failed.

Stop GTF

p GTF 
AHL006I GTF ACKNOWLEDGES STOP COMMAND                    
AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA :
          SYS1.TRACE

Use IPCS to look at the dump, using command GTF USR(ALL). Go to the bottom of the output, use the command report view. This gives an ISPF edit session.

  • x all
  • f ‘RACF Reason code:’ all
    • You are interested in the non zero codes. “Label” each line of interest using the line prefix command .a, .b etc.
  • reset
  • loc .a
    • This will position you by the labelled line. Look up the RACF return and reason codes here. I had Reason Code 2c, which is decimal 44. Look for the keyring, or other information. I do not know which data tells you which sub operation r_datalib was doing, but for me it had the keyring name “CCPKeyring.IZUDFLT “. The description in the reason code documentation does not cover the situation of not having update access to the keyring, so I’ve raised a doc comment on it.

Whoops I just used the wrong 3270 window.

For my z/OS system, I have multiple 3270 session. For example one has an all powerful userid, one has my normal userid, and one has a userid with no authority. I usually position them left to right so it is obvious which session I am using.

I recently had an incident where I disconnected my external monitor, used z/OS, then reconnected my external monitor. The 3270 sessions were not in their usual places, and I used the right hand session to do something, to find I was using the all powerful userid.

I’ve now fixed this by using 

x3270 -model 5 -bd red tso@localhost:3270 &

where -bd red says give the window a red border. Of course if I do not look at the border, it will not help – but I hope it will.

My session now looks like

If you display the x3270 options, you will not find -bd mentioned. x3270 uses some of xterm, which has options which include:

  • -bd color This option specifies the color to use for the border of the window. The default is “black.”
  • -bg color This option specifies the color to use for the background of the window. The default is “white.”
  • -bw number This option specifies the width in pixels of the border surrounding the window.
  • -fg color This option specifies the color to use for displaying text. The default is “black.”
  • -fn font This option specifies the font to be used for displaying normal text. The default is fixed.
  • -name name This option specifies the application name under which resources are to be obtained, rather than the default executable file name. Name should not contain “.” or “*” characters.
  • -rv This option indicates that reverse video should be simulated by swapping the foreground and background colors.
  • +rv Disable the simulation of reverse video by swapping foreground and background colors.
  • -title string This option specifies the window title string, which may be displayed by window managers if the user so chooses. The default title is the command line specified after the -e option, if any, otherwise the application name.

Using and debugging RACF CLASS(APPL) and pthread_security_np

I’ve blogged I want to be someone else – or using pthread_security_np, how a server can be passed a userid or password to logon to a server to do some work. For example I was logging on from a web server to the RMF GPMSERVE for displaying RMF reports in a a web browser.

I had followed the documentation, but all my userids had access, when none of them should have done. This blog post covers the steps I used to dig into this.

The RACF calls used, have a flag set “Suppress any RACF Messages”, which makes it harder to diagnose problems.

With most RACF calls you can define a profile

ADDSD 'COLIN.PROT.DATASET' UACC(NONE) WARNING

where the WARNING option says “If the userid does not have access – give the userid access, but write a message on the console”. This allows you to find when you need to grant access. Once the profile is established, you can use the ALTSD … NOWARNING. Userids requesting access, but which are not permitted will now fail.

Defining a profile with CLASS(APPL), the warning has no effect. I had to use NOTIFY(COLIN) to get notified of problems.

Tracing the requests

To trace all of the RACF calls made by my job COLINNT I issued

#set trace(callable(all),racroute(all),jobname(COLINNT))

This gave me many hundreds of calls.

Once I had been through the whole process, I found the trace for the pthread_security_np etc calls was just 

set trace(callable(type(38)),jobname(COLINNT))

Collect a trace

See Collecting and understanding a RACF GTF trace output.

Looking at the trace output

There is a trace record “before” (PRE) matching “after” (POST). Many parameters are the same (as you might expect).

The output of these traces is verbose, with unnecessary information and with extra blanks lines. For example for one parameter

   Area length:                  00000008 
                                                                                                                         
   Area value: 
   D6C6C6E2  C5E30000                                | OFFSET..                         | 
                                                                                                                         
   Area length:                  00000004 
                                                                                                                         
   Area value: 
   00000000                                         | ....                             | 
                                                                                                                         
   Area length:                  00000008

In the description below, I’ve squashed this down to a single line

area length  8  OFFSET0000 length 4 00000000

It looks like the field OFFSET0000 has the offset in hex from somewhere. I didn’t find this information useful.

A compressed trace record is given below. For the interpretation of the fields see the following trace record.

RTRACE
OMVSPRE
Service number 00000026
Parameters
area length x6c  
area length  8  OFFSET0000 length 4 00000000
area length  8  OFFSET0004 length 4 00000000
area length  8  OFFSET0008 length 4 00000000
area length  8  OFFSET000C length 4 00000000
area length  8  OFFSET0010 length 4 40404040
area length  8  OFFSET0014 length 4 00000000
area length  8  OFFSET0018 length 4 40404040
area length  8  OFFSET001C length 1 01
area length  8  OFFSET0020 length 4 C4800000
area length  8  OFFSET0024 length 6 05c1c2c3c4C2 =  .ABCDB
area length  8  OFFSET0028 length 4 00000000 
area length  8  OFFSET002C length 9 08C7D7D4  E2C5D9E5  C5   = .GPMSERVE
Internal information
area length  8  OFFSET0034 length x37 = "Server Userid=IBMUSER created a full ACEE"
area length  8  OFFSET0048 length 4 00000000
area length  8  OFFSET0050 length 9 08000000  00000000  00
area length  8  OFFSET0054 length 1 00 
area length  8  OFFSET0018 length 4 40404040
Internal data
area length xc0 ACEE
area length x50 userid information
area length x90 ACEX
area length x50 USP

hex dump of record

The RACF commands documentation says for a callable service, 0x00000026 is function number for IRRSIA00. This page gives the name of the function name IRRSIA00 and the description z/OS kernel on behalf of servers that use pthread_security_np servers or __login, or MVS servers that do not use z/OS UNIX services.

The parameters are for the initACEE (IRRSIA00) call.

The matching “after” record, OMVSPOST was (with the parameters of the the IRRSIA00 call format, parameters) are

RTRACE
OMVSPOST
Service number:               00000026 
RACF Return code:             00000008 
RACF Reason code:             00000020 
Parameters
area length  8  OFFSET0000 length 4 00000000 >work_area<
area length  8  OFFSET0004 length 4 00000000 >ALET<
area length  8  OFFSET0008 length 4 00000008 >SAF_return_code<
area length  8  OFFSET000C length 4 00000000 >ALET< 
area length  8  OFFSET0010 length 4 00000008 >RACF_return_code<
area length  8  OFFSET0014 length 4 00000000 >ALET<
area length  8  OFFSET0018 length 4 00000020 >RACF_reason_code<
area length  8  OFFSET001C length 1 01       >Function_code 1 = Create an ACEE<
area length  8  OFFSET0020 length 4 C4800000 >Attributes - see below<
area length  8  OFFSET0024 length 6 05c1c2c3c4C2  >Userid .ABCDB<
area length  8  OFFSET0028 length 4 00000000 >ACEE_pointer<
area length  8  OFFSET002C length 9 08C7D7D4  E2C5D9E5  C5  >APPLID = .GPMSERVE<
Internal information
area length  8  OFFSET0034 length x37 = "Server Userid=IBMUSER created a full ACEE"
area length  8  OFFSET0048 length 4 00000000
area length  8  OFFSET0050 length 9 08000000  00000000  00
area length  8  OFFSET0054 length 1 00 
area length  8  OFFSET0018 length 4 40404040
Internal data
area length xc0 ACEE
area length x50 userid information
area length x90 ACEX
area length x50 USP

There the attributes C480000 mean

  • X80000000 – Create the ACEE
  • X40000000 – Createthe USP for the userid
  • X04000000 – Suppress any RACF Messages
  • X00800000 – Return an OUSP in the output area

Note the flag:X04000000 – Suppress any RACF Messages.

The return code 

area length  8  OFFSET0008 length 4 00000008 >SAF_return_code<
...
area length  8  OFFSET0010 length 4 00000008 >RACF_return_code<
area length  8  OFFSET0014 length 4 00000000 >ALET<
area length  8  OFFSET0018 length 4 00000020 >RACF_reason_code<

8,8,32 means The user does not have appropriate RACF access to either the SECLABEL, SERVAUTH profile, or APPL specified in the parmlist.

For other RACF services the trace entries follow a similar format.

Tracing encrypted data to z/OS

I had blogged Collecting a wire-shark trace with TLS active for a browser where you could specify an environment variable export SSLKEYLOGFILE=$HOME/sslkeylog.log. OpenSSL would write the key to this file, and Wireshark could decrypt the traffic using this data.

Unfortunately this only worked with RSA keys. I could not get it to work with modern Elliptic Curve keys.

I’ve updated my zWireshark program to capture AT-TLS application data in clear text from the z/OS side. It uses an IBM provided API, and captures the traffic between AT-TLS and the application.

You need to set up security profiles, for example

permit  EZB.TRCSEC.*.*.ATTLS            - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit  EZB.TRCCTL.S0W1.TCPIP.DATTRACE  - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit EZB.TRCCTL.S0W1.TCPIP.OPEN  - 
   CL(SERVAUTH) id(ADCDB) access(READ) 
permit EZB.TRCCTL.*.*.* CL(SERVAUTH) id(ADCDB) access(READ) 
SETROPTS RACLIST(SERVAUTH) refresh

and change the AT-TLS configuration to include CtraceClearText On

For my web browser traffic it produced (printed with the ASCII switch)

Data trace.  Data length 345.  ATTLS Clear Text.                             
<GPMSERVE  11:01:57.258946 Src 10.1.1.2 Port 8803 Dst 10.1.0.2 Port 45168    
Warning: 199 RMF-DDS-Server SeverityCode(03) Data(0)                         
Content-Location: perform_20250102110157_20250102110157.xml                  
Cache-Control: max-age=30                                                    
Date: Thu, 02 Jan 2025 11:01:57 GMT                                          
Connection: close                                                            
Content-Length: 1468                                                         
Content-Type: application/xml                                                
X-UA-Compatible: IE=edge                                                     
X-Content-Type-Options: nosniff                                              
X-XSS-Protection: 1; mode=block

When the AT-TLS option CtraceClearText was Off, the output was

Data trace.  Data length 0.  PTH_Flag.Confidential.                           
<GPMSERVE  11:27:48.733616 Src 10.1.1.2 Port 8803 Dst 10.1.0.2 Port 52860

So no confidential data was displayed.

The JCL is

//COLINC5    JOB 1,MSGCLASS=H,COND=(4,LE) 
// SET LOADLIB=COLIN.ZWIRESHA.LOAD 
//RUN      EXEC PGM=TCPDATA,REGION=0M,PARMDD=MYPARMS 
//STEPLIB  DD DISP=SHR,DSN=&LOADLIB 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=* 
//SYSERR   DD SYSOUT=* 
//* MYPARMS needs a / to split LE parms from program parms 
//*         needs a blank before each parm, because trailing blanks removed 
//MYPARMS DD * 
/ 
 --IP   10.1.0.2 
 --WAIT 60 
 --DEBUG 0 
 --DISCARD 1 
 --PRINT A 
/*

Please let me know of any problems or suggestions

I want to be someone else – or using pthread_security_np

There are times when you are running a job, and you want to do some work as a different userid.

You can configure a userid so it acts as a surrogate – I can submit jobs with your userid, without knowing your password.

You can have a server and have a thread within the server run as different userid by specifying a userid and password, using the pthread_security_applid_np function. You can also have it logon using a certificate, and it looks up the userid from the certificate.

You can extend/restrict this by specifying an RACF APPL which the user must have access to. 

RDEFINE APPL YYYY 
PERMIT  CLASS(APPL) YYYY ID(COLIN) ACCESS(READ) 
SETROPTS RACLIST(appl ) REFRESH

My job running with userid IBMUSER specifies a userid COLIN, and COLIN’s password. If applid YYYY is specified, userid COLIN must have read access to the YYYY in CLASS(APPL).

If you use pthread_security_np() is the same as pthread_security_applid_np() and uses the default OMVSAPPL.

My example C program is

Main

int main( int argc, char *argv[]) 
{ 
  pthread_t thid; 
  int rc; 
  void *ret; 
  if (pthread_create(&thid, NULL, thread, "thread 1") != 0) { 
    perror("pthread_create() error"); 
    exit(1); 
  } 
  rc =pthread_join(thid, &ret); 
  printf("Pthread join %d\n",rc); 
  if (rc  != 0) { 
 // perror("pthread_create() error"); 
 // return(3); 
  } 
  printf("thread exited with '%s'\n", ret); 
  return 0  ; 
} 

  #pragma runopts(POSIX(ON)) 
  /*Include standard libraries */ 
  #include <stdio.h> 
  #include <stdlib.h> 
  #include <string.h> 
  #include <stdarg.h> 
  #include <errno.h> 
  #define _OPEN_SYS 1 
  #include <pthread.h> 
                                                                    
  void *thread(void *arg) { 
  char *ret = ""; 
  int rc; 
  printf("thread() entered with argument '%s'\n", arg); 
  rc = pthread_security_applid_np(__CREATE_SECURITY_ENV, 
              __USERID_IDENTITY, 
              5, 
              "COLIN", 
              "PASSW0RD", 
              0,"AAAA"); 
  perror("perror"); 
  printf("colin rc = %d errno %d\n",rc,errno); 
  if (rc == 0) 
  { 
     FILE * f2  ; 
     f2 = fopen("DD:TEST","r                      ") ; 
     printf("DD:TEST  %d\n",f2); 
   } 
  pthread_exit(ret); 
  } 



int main( int argc, char *argv[]) 
{ 
  pthread_t thid; 
  int rc; 
  void *ret; 
  if (pthread_create(&thid, NULL, thread, "thread 1") != 0) { 
    perror("pthread_create() error"); 
    exit(1); 
  } 
  rc =pthread_join(thid, &ret); 
  printf("Pthread join %d\n",rc); 
  if (rc  != 0) { 
    perror("pthread_create() error"); 
   return(3); 
  } 
  printf("thread exited with '%s'\n", ret); 
  return 0  ; 
}

When the above program ran it gave me

ICH408I USER(COLIN   ) GROUP(SYS1    ) NAME(COLIN PAICE         ) 
  IBMUSER.TRY.PEM CL(DATASET ) VOL(C4USR1)                        
  INSUFFICIENT ACCESS AUTHORITY                                   
  ACCESS INTENT(READ   )  ACCESS ALLOWED(NONE   )

which shows that the userid COLIN was used.

EDC5167I Access to the UNIX System Services version of the C RTL is denied. (errno2=0xC10F0001 C10F0001) .

You cannot use the pthread_security… functions in the main thread. You have to attach a subtask using pthread_create().

EDC5143I No such process. (errno2=0x0BE80000 0BE80000)

Invalid userid specified.

EDC5111I Permission denied. (errno2=0x0BE80000 0BE80000)
Invalid password.

EDC5163I SAF/RACF extract error. (errno2=0x0BE8081C 0BE8081C)

Revoked userid.

EDC5168I Password has expired. (errno2=0x0BE80000 0BE80000)

Obvious.

EDC5163I SAF/RACF extract error. (errno2=0x0BE80820 0BE80820)

No OMVS segment, or the new user does not have access to the specified applid class.

rc = pthread_security_np() defaults to pthread_security_applid_np() with an applid of OMVSAPPL

ICH70004I 
ATTEMPTED 'READ' ACCESS OF USER(ADCDE) GROUP(TEST) NAME(ADCDE) ENTITY 'OMVSAPPL' IN CLASS 'APPL'

EDC5139I Operation not permitted. (errno2=0x0BE802AF 0BE802AF)

ICH420I PROGRAM CERT FROM LIBRARY COLIN.LOAD CAUSED THE ENVIRONMENT TO
BECOME UNCONTROLLED.
BPXP014I ENVIRONMENT MUST BE CONTROLLED FOR SERVER (BPX.SERVER)
PROCESSING.

Needed

RALTER PROGRAM CERT ADDMEM('COLIN.LOAD'//NOPADCHK)
SETROPTS WHEN(PROGRAM) REFRESH

Python could not read a data set I sent from z/OS USS.

I created a file in Unix System Services, and FTPed it down to my Linux box. I could edit it, and process it with no problems, until I came to read in the file using Python.

Python gave me

File “<frozen codecs>”, line 322, in decode
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb8 in position 3996: invalid start byte

The Linux command file pagentn.txt gave me

pagentn.txt: ISO-8859 text

whereas other files had ASCII text.

I changed my Python program to have

with open(“/home/colinpaice/python/pagentn.txt”,encoding=”ISO-8859-1″) as file:

and it worked!

I browsed the web, and found a Python way of finding the code page of a file

import chardet    
rawdata = open(infile, 'rb').read()
result = chardet.detect(rawdata)
charenc = result['encoding']

it returned a dict with

result {‘encoding’: ‘ISO-8859-1’, ‘confidence’: 0.73, ‘language’: ”}

One minute mvs: data set file system (/dsfs) on z/OS

There is a good overview of dsfs here.

My understanding of how dsfs works is that you can access z/OS datasets, members and spool, through a Unix file system interface, and so use commands like “ls”. When you read a member, it is read into the dsfs file system, and the Unix commands work on this data. When you write to the Unix file, it is written to the dsfs file system, and when the file is closed, the data is written to the dataset.

Some information is cached in the dsfs file system, and some definitions, such as DCB information on a per userid basis is also stored.

DSFS is defined to OMVS as a Physical File System. Part of the definition of the PFS is the started task name. When the PFS is defined to OMVS, it starts the DSFS started task.

To stop it, you issue a command to OMVS to stop the PFS.

Permissions

The dsfs started task has threads which do work. When you issue a request, one of the threads becomes your userid and accesses the data sets, so there is no change to the standard RACF data set protection.

User defaults

You can define defaults for when data sets are created. These are user dependant. To issue these for another user, you have to become a super user to become that id, and then issue the dfsadm command.

Changing configuration

You can change the configuration file and restart DSFS (which will be disruptive).

You can make changes to the active system (which are not saved) using the omvs command

Diagnosis

Messages are produced, and are documented in the M&C manual. if you get a reason code, you can use bpxmtext code to display the meaning of the code.

Creating data sets and members from /dsfs

You can configure defaults when using dsfs to create data sets and members. For example

Creating a PDS(E)

cd /dsfs/txt/colin
dsadm createparm -path . -pdsmodel "dsntype(library) dsorg(po) lrecl(133) recfm(vb)      
blksize(0)"

For path /dsfs/txt/colin (-path .) it creates a model for pds (-pdsmodel) with the parameters dsntype…blksize(0)

Note:blksize(0) says “pick the best block size”

You can display this using

dsadm fileinfo   -path /dsfs/txt/colin

and get 

path: /dsfs/txt/colin                                                                                
fid                    49,1          anode                  1285,4548                                
length                 16384         format                 BLOCKED                                  
1K blocks              16            dir tree status        VALID                                    
PDS model anode        31,60         PS model anode         0,0                                      
object type            DIR           object linkcount       181                                      
object genvalue        0x00000000    dir version            1                                        
dir name count         431           data set name type     HLQ DIR                                  
recfm                  na            lrecl                  na                                       
data mode              TEXT          data set status        RETRIEVED                                
data set name          COLIN                                                                         
ENQ held               NO                                                                            
direct blocks          0x00001BAF    0x00001BB0    0xFFFFFFFF    0xFFFFFFFF                          
                       0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF                          
indirect blocks        0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF                                        
mtime       Sep  9 10:09:55 2024     atime        Sep  9 10:09:55 2024                               
ctime       Sep  9 10:09:55 2024     create date  Sep  9 2024                                        
not encrypted                        not compressed                                                  
PDS model              dsntype(library) dsorg(po) lrecl(133) recfm(vb) blksize(0)                  
PS model               na                                                                            
vnode,vntok          0x00000050,,0x072003C0      0x0099A7F0,,0x00000000                              
opens                oi=0          rd=0          wr=0                                                
file segments        na         file unscheduled        na                                           
meta buffers         0          dirty meta buffers      2

Creating a sequential file

dsadm createparm -path /dsfs/txt/colin -psmodel "dsorg(ps) lrecl(80) recfm(fb) blksize(3200)"

The command

dsadm fileinfo   -path /dsfs/txt/colin

now gives

PS model               dsorg(ps) lrecl(80) recfm(fb) blksize(3200)

The command

echo "hi" > /dsfs/txt/colin/seq2

creates a data set COLIN.SEQ2 with characteristics

Record format . . . : FB
Record length . . . : 80
Block size  . . . . : 3200

If you specify blksize(0) the system applies a default. In my case Block size . . . . : 27920, which makes good usage of the disk space.

Different data set types

There are three data set types

  • bin – binary for example load modules
  • rec – records
  • txt – text files, with a new line character “at the end of the line”

You can define the data set characteristics for each of these.

Problems I experienced when using /dsfs

I played around with /dsfs and found a couple of problems.

My userid is COLIN.

Data set not seen for a time

I created a data set COLIN.ENCR2.DSN in ISPF (and also in batch).

The command

ls -ltr /dsfs/txt/colin/en*

found the data set /dsfs/txt/colin/encrypt.jcl, but it did not find /dsfs/txt/colin/encr2.dsn.

A while later (a minute or two) it found /dsfs/txt/colin/encr2.dsn . This is because of the DIRECTORY_REFRESH_TIMEOUT that controls it. Its default value is 60. You can dynamically change it via command dsadm config -directory_refresh_timeout <number>. The expected value is a number in the range 30-600.

Encrypted data set not seen

I had set up a data set which was encrypted, and empty. I could not list it using /dsfs

I had to set up an encryption ZFS for DSFS (as the documentation says) for it to be seen. I expected the dataset name to be visible, but not the contents.

Create the label in ICSF

//IBMICSF  JOB 1,MSGCLASS=H 
//STEP10 EXEC PGM=CSFKGUP 
//  SET CKDS=COLIN.SCSFCKDS 
//CSFCKDS DD DISP=OLD,DSN=&CKDS 
//* LENGTH(32) GENERATES A 256 BIT KEY 
//CSFIN DD *,LRECL=80 
ADD TYPE(DATA) ALGORITHM(AES), 
LABEL(DSFSAES                ) LENGTH(32) 
/* 
DELETE TYPE(DATA) LABEL(COLINBATCHAES ) 
//CSFDIAG DD SYSOUT=*,LRECL=133 
//CSFKEYS DD SYSOUT=*,LRECL=1044 
//CSFSTMNT DD SYSOUT=*,LRECL=80 
//* REFRESH THE IN MEMORY DATA 
//REFRESH  EXEC PGM=CSFEUTIL,PARM='&CKDS,REFRESH'

Give the DSFS userid access to the label

//IBMRACF2 JOB 1,MSGCLASS=H RESTART=S2 
// EXPORT SYMLIST=(*) 
// SET KEY=DSFSAES 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD *,SYMBOLS=(JCLONLY) 
RDELETE CSFKEYS &KEY 
RDEFINE CSFKEYS &KEY                          + 
  ICSF(SYMCPACFWRAP(YES) SYMCPACFRET(YES))    + 
  UACC(NONE) 
PERMIT  &KEY                                  + 
  CLASS(CSFKEYS) ID(DSFS         )            + 
  ACCESS(READ) 
SETROPTS RACLIST(CSFKEYS) REFRESH 
                                                             
RLIST  CSFKEYS &KEY      AUTHUSER         ICSF 
                                                             
ADDSD  'COLIN.ZFS.DSFS'  UACC(NONE) 
ALTDSD 'COLIN.ZFS.DSFS'  UACC(NONE)    DFP(DATAKEY(&KEY)) 
/*

Define the ZFS for DSFS

I stopped DSFS using the operator command

f omvs,stoppfs=dsfs

I could then delete and recreate the ZFS. If I wanted to keep the content of the DSFS ZFS, I think I could have using VSAM REPRO to copy the contents to a new file.

//IBMCLUST  JOB 1 
//DEFINE  EXEC PGM=IDCAMS,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
  DELETE          COLIN.ZFS.DSFS 
  DEFINE CLUSTER (NAME(COLIN.ZFS.DSFS)       - 
                  VOLUMES(*     )            - 
                  CYL(100 1000)              - 
                  KEYLABEL(DSFSAES)          - 
                  DATACLASS(DCEATTR)         - 
                  ZFS    ) 
LISTCAT ENT(COLIN.ZFS.DSFS) ALL 
/*

The LISTCAT output had

LISTCAT ENT(COLIN.ZFS.DSFS) ALL                                                                            
CLUSTER ------- COLIN.ZFS.DSFS                                                                                     
 ...                                               
     ENCRYPTIONDATA                                                                                                
       DATA SET ENCRYPTION----(YES)                                                                                
       DATA SET KEY LABEL-----DSFSAES

I restarted DSFS.

With the DSFS ZFS encrypted, the ls command for the encrypted data set worked, and I was able to read the content.

When my userid COLIN which had access to the encrypted data set, but not the encryption key, tried to display the contents there were messages on the system log

ICH408I USER(COLIN   ) GROUP(SYS1    ) NAME(####################) 008    
  COLINBATCHAES CL(CSFKEYS )                                             
  INSUFFICIENT ACCESS AUTot HORITY                                          
  ACCESS INTENT(READ   )  ACCESS ALLOWED(NONE   )                        
IDFS00338E RACROUTE REQUEST=FASTAUTH call to authorize access to the keylabel COLINBATCHAES                                                   
for data set COLIN.ENCR.DSN for user COLIN failed with SAF rc 8 RACF rc 8 Reason Code 0.

and the OMVS session got

cat: /dsfs/txt/colin/encr.dsn: EDC5164I SAF/RACF error.

It is hard work getting error messages

I got 

COLIN:/u: >dsadm config -hlq_list_add SYS1.PARMLIB                                                                                
IDFS00324E Could not add the specified high-level qualifier directory names to the HLQ list, error code 139 reason code ED04618D.

To get the full description of the error I had to use

COLIN:/u: >bpxmtext ED04618D                                                                                                      
DSFS Tue Sep 12 19:21:54 EDT 2023                                                                                                 
Description: The caller is not UID 0.                                                                                             
                                                                                                                                  
Action: Ensure that you are properly authorized for this operation. This                                                          
subcommand requires UID 0 or READ access to the SUPERUSER.FILESYS.PFSCTL                                                          
profile in the z/OS UNIXPRIV class.

Where’s the $?

Because the OMVS shell uses $ to replace variables, you have to quote the data set name.

IBMUSER:/u/ibmuser: >ls /dsfs/txt/sys1.proclib/$$$COIBM                           
ls: FSUM6785 File or directory "/dsfs/txt/sys1.proclib/50397215" is not found

You have to use

ls '/dsfs/txt/sys1/parmlib/$$$COIBM'

with quotes around it, or escape the special characters.

ls /dsfs/t*/sys1/proclib/\$\$\$coibm

Setting up data set file system /dsfs on z/OS

There is a good overview of dsfs here.

The documentation is pretty good. It is clear – but I would have ordered the sections differently.

Concept

My understanding of how dsfs works is that you can access z/OS datasets, members and spool, through a Unix file system interface, and so use commands like “ls”. When you read a member, it is read into the dsfs file system, and the Unix commands work on this data. When you write to the Unix file, it is written to the dsfs file system, and the data is written to the dataset when the file is closed.

Some information is cached in the dsfs file system, and some definitions, such as DCB information on a per userid basis is also stored.

Setup the RACF definitions

//IBMRACFL JOB 1,MSGCLASS=H                                
//S1  EXEC PGM=IKJEFT01,REGION=0M                          
//SYSPRINT DD SYSOUT=*                                     
//SYSTSPRT DD SYSOUT=*                                     
//SYSTSIN DD *                                             
 ADDGROUP DSFSGRP SUPGROUP(SYS1)                           
 ADDUSER DSFS DFLTGRP(DSFSGRP) AUTHORITY(USE) UACC(NONE)   
 RDEFINE STARTED DSFS.** STDATA(USER(DSFS))                
 SETROPTS RACLIST(STARTED) 
 SETROPTS RACLIST(STARTED) REFRESH 
/*                                                         
//

The started task procedure is in IOE.SIOESAMP(DSFS)

You can configure it to use the parmlib concatenation. I configured it to use a member in user….parmlib.

//DSFS     PROC REGSIZE=0M                                            
//*                                                                   
//DSFSGO   EXEC PGM=BPXVCLNY,REGION=&REGSIZE,TIME=1440                
//*                                                                   
//*STEPLIB DD DISP=SHR,DSN=hlq.SIEALNKE               <--DSFS LOADLIB 
//IDFZPRM DD DISP=SHR,DSN=USER.Z31A.PARMLIB(DSFS) <--DSFS PARM FILE

This is automatically started when you activate the dsfs physical file system (in a bpxprmxx member).

Create the ZFS

If you are going to use encrypted data sets the ZFS needs to be encrypted. See One minute MVS – Using individual data set encryption on z/OS.

I do not know how much space you actually need. If you specify a secondary extent, the ZFS can expand as needed.

If you want the ZFS to be larger than 4GB you need to specify Extended Addressability. On ADCD there is an SMS Data Class DCEATTR with extended attributes.

The documentation talks about having the z/OS image name as part of the data set when part of a sysplex. I specified the name as part of DSFS startup.

If you are using ADCD and you use a “system” HLQ such as ZFS.DSFS, then when you move to the next release of ADCD you’ll have to migrate it. By using COLIN as a HLQ, it gets picked up automatically in the next release after I have imported the user catalog.

//IBMCLUST  JOB 1 
//DEFINE  EXEC PGM=IDCAMS,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
  DEFINE CLUSTER (NAME(COLIN.ZFS.DSFS)       - 
                  VOLUMES(*     )            - 
                  CYL(100 1000)              - 
                  DATACLASS(DCEATTR)          - 
                  ZFS    ) 

/* 
 LISTCAT ENT(COLIN.ZFS.DSFS) ALL
 DELETE          COLIN.ZFS.DSFS

The ZFS is formatted when it is first used, and as it expands.

Startup parameters

In USER.Z31A.PARMLIB(DSFS), referred to in the DSFS started procedure, I have

* 
UTFS_NAME=COLIN.ZFS.DSFS

Without the UTFS_NAME, it will look for a data set with the image name. If you are running in a sysplex each system needs its own ZFS.

Define the root

In TSO OMVS use mkdir /dsfs

Define the physical file system, and mount the file

I created USER.Z31A.PARMLIB(BPXPRMFS) with

FILESYSTYPE TYPE(DSFS) ENTRYPOINT(IDFFSCM) ASNAME(DSFS)

MOUNT TYPE(DSFS)
MODE(RDWR)
NOAUTOMOVE
MOUNTPOINT('/dsfs')
FILESYSTEM('COLIN.ZFS.DSFS')

Activate and start it

You can use the operator command set omvs=(FS) to activate the member bpxprmfs (defined above). This starts the address space DSFS. You could have this activated as part of IPL in the usual way.

The first time I started DSFS, I got messages

SET OMVS=(FS)                                                         
IEE252I MEMBER BPXPRMFS FOUND IN USER.Z31A.PARMLIB                    
BPXO032I THE SET OMVS COMMAND WAS SUCCESSFUL.                         
IEF196I IEF285I   USER.Z31A.PARMLIB                            KEPT   
IEF196I IEF285I   VOL SER NOS= A3CFG1.                                
...                           
$HASP100 DSFS     ON STCINRDR                                         
$HASP373 DSFS     STARTED                                             
IEF403I DSFS - STARTED - TIME=08.51.23                                
IDFS00153I DSFS kernel: Initializing z/OS    DSFS 279                 
Version 03.01.00 Service Level OA65531 - HZFS510.                     
Created on Tue Sep 12 19:21:54 EDT 2023.                              
Address space asid x47.                                               
IDFS00064I USER.Z31A.PARMLIB(DSFS) is the configuration data set currently in use.                                                     
IDFS00352I DSFS is using HLQ EXCLUDE mode.                            
IDFS00039I DSFS kernel: initialization complete.                      
BPXF025I FILE SYSTEM COLIN.ZFS.DSFS IS BEING MOUNTED.                                                     
IDFS00320I Formatting to 4096 control interval 18000 for primary 
IDFS00003I Primary extent loaded successfully for COLIN.ZFS.DSFS. 
IDFS00034I Detaching utility file system COLIN.ZFS.DSFS.

Stop dsfs

If you get messages saying data set in use etc, this may be due to the DSFS program has started. You can stop it using f omvs,stoppfs=dsfs, fix any problems, and restart it.

Setting up permissions

Some of the dsadm commands, for example which change the configuration, need permission. The issuer must be logged in as a root user (UID=0) or have READ authority to the
SUPERUSER.FILESYS.PFSCTL resource in the z/OS UNIXPRIV class. Note: If you are permitted READ to the BPX.SUPERUSER resource in the RACF FACILITY class, you can become a UID of 0 by issuing the su command to get uid 0.

PERMIT SUPERUSER.FILESYS.PFSCTL CLASS(UNIXPRIV) ID(userid) ACCESS(READ)
SETROPTS RACLIST(UNIXPRIV) REFRESH

PERMIT BPX.SUPERUSER CLASS(FACILITY) ID(userid) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH

Once it has started – now what?

In OMVS you can use the ls -ltr command

IBMUSER:/u/ibmuser: >ls -ltr  /dsfs                           
total 0                                                       
drwxrwxrwx   2 OMVSKERN SYS1           0 Sep  9 03:51 txt     
drwxrwxrwx   2 OMVSKERN SYS1           0 Sep  9 03:51 sysout  
drwxrwxrwx   2 OMVSKERN SYS1           0 Sep  9 03:51 rec     
drwxrwxrwx   2 OMVSKERN SYS1           0 Sep  9 03:51 bin

You can use the dsfs commands in Unix Services

dsadm fsinfo
   File System Name:          COLIN.ZFS.DSFS                         
                                                                     
   System:              S0W1            Devno:                  0    
   Size:                72000K          Free 8K Blocks:         8732 
   Free 1K Fragments:   7               Log File Size:          2056K
   Bitmap Size:         16K             Anode Table Size:       8K   
   File System Objects: 7               Version:                1    
   Overflow Pages:      0               Overflow HighWater:     0    
   Space Monitoring:    0,0                                          
   ENOSPC Errors:       0               Disk IO Errors:         0    
   Status:              NE,NC                                        
                                                                     
   File System Creation Time: Sep  9 03:51:25 2024                   
   Mount Time:                Sep  9 03:51:25 2024                   
                                                                     
   Last Grow Time:            n/a