When is an error not an err?

If you step off the golden path of trying to read a file – you can quickly end up in in trouble and the diagnostics do not help.

I had some simple code

FILE * hFile = fopen(...); 
recSize = fread(pBuffer ,1,bSize,hFile); 
if (recSize == 0)
{
  // into the bog!
 if (feof(hFile))printf("end of file\n");
 else if (ferror(hFile)) printf("ferror(hFile) occurred\n");
 else printf("Cannot occur condition\n");
 
}

When running a unit test of the error path of passing a bad file handle, I got the “cannot occur condition because the ferror() returned “OK – no problem “

The ferror() description is

General description: Tests for an error in reading from or writing to the specified stream. If an error occurs, the error indicator for the stream remains set until you close the stream, call rewind(), or call clearerr().
If a non-valid parameter is given to an I/O function, z/OS XL C/C++ does not turn the error flag on. This case differs from one where parameters are not valid in context with one another.

This gave me 0, so it was not able to detect my error. ( So what is the point of ferror()?)

If I looked at errno and used perror() I got

errno 113
EDC5113I Bad file descriptor. (errno2=0xC0220001)

You may think that I need to ignore ferror() and check errno != 0 instead. Good guess, but it may not be that simple.

The __errno2 (or errnojr – errno junior)) description is

General description: The __errno2() function can be used when diagnosing application problems. This function enables z/OS XL C/C++ application programs to access additional diagnostic information, errno2 (errnojr), associated with errno. The errno2 may be set by the z/OS XL C/C++ runtime library, z/OS UNIX callable services or other callable services. The errno2 is intended for diagnostic display purposes only and it is not a programming interface. The __errno2() function is not portable.
Note: Not all functions set errno2 when errno is set. In the cases where errno2 is not set, the __errno2() function may return a residual value. You may use the __err2ad() function to clear errno2 to reduce the possibility of a residual value being

If you are going to use __errno2 you should clear it using __err2ad() before invoking a function that may set it.

I could not find if errno is clean or if it may return a residual value, so to be sure to set it before every use of a C run time library function.

Having got your errno value what do you do with it?

There are #define constants in errono.h such as

#define EIO 122 /* Input/output error */

You case use if ( errno == EIO ) …

Like many products there is no mapping of 122 to “EIO”, but you can use strerror(errno) to map the errno to the error string like EDC5113I Bad file descriptor. (errno2=0xC0220001). This also provides the errno2 string value.

fopen trace etc is not so useful

If you can specify an environment variable you can trace the C file operations.

This did not give much useful information, as it did not give the name of the file being processed, and I could not trace the file which was causing fopen problems, so overall a good idea – but a poor implementation.

How to set it up

See File I/O trace, Locating the file I/O trace and the environment variable _EDC_IO_TRACE

For example

export _EDC_IO_TRACE="(*,2,1M)"

Where filter is

Filter Indicates which files to trace.

  • //DD:filter Trace will include the DD names matching the specified filter string.
  • //filter Trace will include the MVS™ data sets matching the specified filter string. Member names of partitioned data sets cannot be matched without the use of a wildcard. filter Trace will include the Unix files matching the specified filter string.
  • //DD:* Trace will include all DD names.
  • //* Trace will include all MVS data sets. This is the default setting.
  • /* Trace will include all Unix files.
  • * Trace will include all MVS data sets and Unix files.

Detail – use 2.

Buffer size such as 1M or 50K .

The output goes to a file such as /tmp, but you can change this with

export _CEE_DMPTARG=”.”

This worked for me … but initially I could not read the output file. (It may because it came from Python which has been compiled with ASCII option.

The command ls -ltrT showed the file was tagged in ASCII, so I used

chtag -r EDC*

to reset it, and I could edit the file.

Sample output

Trace details for ((POSIX)):
        Trace detail level:  2 
        Trace buffer size:   1024K                                                                  
        fdopen(10,r)                                                                 
        fldata: 
            __recfmF:1........ 0            __dsorgVSAM:1..... 0 
            __recfmV:1........ 0            __dsorgHFS:1...... 1 
            __recfmU:1........ 1            __openmode:2...... 1 
...

Which is not very helpful as it does not tell you the file that has been opened!

When I traced a Python program, I only got information on 5 files – instead of the hundreds I was expecting.

Various abends and problems

I’ll list them here for search engines to find.

CEE3250C The system or user abend U4000 R=00007017 was issued.

U4000

  • Explanation: The assembler user exit could have forced an abend for an unhandled condition. These are user-specified abend codes.
  • System action:Task terminated.
  • Programmer response:
  • Check the Language Environment message file for message output. This will tell you what the original abend was.

There were no other messages. BPXBATCH ended with return code 2304 which means a kill -9 was issued.

If I remove the _EDC_IO_TRACE it works.

I also got a file BST-1.20220809.110241.83951661 etc which is tagged as ASCII – but is not.

This file had the trace for they Python file which was being run – including the name of the file.

If you cannot open a data set, amrc may help

I had a C program which opened a dataset and read from it. I enhanced it, by adding comments and other stuff, and after lunch it failed to open

I undid all my changes, and it still it failed to open! Weird.

I got message

  • EDC5061I
    • An error occurred when attempting to define a file to the system. (errno2=0xC00B0403)
    • Programmer response : Check the __amrc structure for more information. See z/OS XL C/C++ Programming Guide for more information on the __amrc structure.
  • C00B0403:
    • The filename argument passed to fopen() or freopen() specified dsname syntax. Allocation of a ddname for the dsname was attempted, but failed.
    • Programmer response: Failure information returned from SVC 99 was recorded in the AMRC structure. Use the information there to determine the cause of the failure.

This feels like the unhelpful messages Ive seen. “An error has occurred – we know what the error is – but we wont tell you” type messages.

To find the reason I had to add some code to my program.

 file =  fopen(fileName, mode     ); 
 __amrc_type save_amrc; 
 memcpy(&save_amrc,__amrc,sizeof(__amrc)); 
 printf("AMRC __svc99_info %hd error %hd\n",
         save_amrc.__code.__alloc.__svc99_info, 
         save_amrc.__code.__alloc.__svc99_error); 
                                                                                    

and it printed

AMRC __svc99_info 0 528

The DYNALLOC (dynamic allocation) which uses SVC 99 to allocate data sets, has a section Interpreting error reason codes from DYNALLOC. The meaning of 528 is Requested data set unavailable. The data set is allocated to another job and its usage attribute conflicts with this request.

And true enough, in one of the ISPF sessions in one of my TSO userid I was editing the file.

It looks like

printf(“__errno2 = %08x\n”, __errno2());

Would print the same information.

Thoughts

It appears that you cannot tell fopen to open it for read even if it has a write lock on it.

For DYNALLOC, if the request worked, these fields may have garbage in them – as I got undocumented values.

It would be nice if the developer of the fopen code produced messages like

EDC5061I: An error occurred when attempting to define a file to the system. (errno2=0xC00B0403) (AMRC=0x00000210)

Then it would be more obvious!

How do I use BPXBATCH? It is not that obvious.

I was running a Python script via BPXBATCH, with no problem. Then I extended it, and it was unable to find a load module. In getting this to work, I found out a lot about using BPXBATCH, and how things do not work as documented.

Getting started

You can run a shell program, or a “module” (program).

Use STDPARM

This can be used instead of specifying PARM=…. It avoids the 100 character restriction of PARM=…

You can use EXPORT SYMLIST, and SYMBOLS=EXECSYS for JCL variables to be passed into the DD * data.

//  EXPORT SYMLIST=*
// SET PY='/usr/lpp/IBM/cyp/v3r8/pyz/bin/python3'
// SET PR='/u/tmp/zos'
//STEP1 EXEC PGM=BPXBATSL,
//STDPARM DD *,SYMBOLS=EXECSYS             
pgm &PY &PR/my.py'
/*
//STDOUT   DD SYSOUT=* 
//STDERR   DD SYSOUT=* 
//SYSDUMP  DD SYSOUT=* 
//CEEDUMP  DD SYSOUT=* 
//STDIN    DD DUMMY 

This will execute program /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 and pass /u/tmp/zos/my.py

Run a program

You can use JCL like

// SET PY='/usr/lpp/IBM/cyp/v3r8/pyz/bin/python3'
// SET PR='/u/tmp/zos'
//STEP1 EXEC PGM=BPXBATSL,PARM='pgm &PY &PR/my.py'
//STDOUT   DD SYSOUT=* 
//STDERR   DD SYSOUT=* 
//SYSDUMP  DD SYSOUT=* 
//CEEDUMP  DD SYSOUT=* 
//STDIN    DD DUMMY 

This will execute program /usr/lpp/IBM/cyp/v3r8/pyz/bin/python3 and pass /u/tmp/zos/my.py

Run a shell

//STEP1   EXEC PGM=BPXBATSL,REGION=0M,TIME=NOLIMIT,MEMLIMIT=NOLIMIT, 
//   PARM='SH /u/tmp/zos/cc.sh' 
//STDENV   DD * 
PATH=/u/abc/ 
xyz=123 
//STDOUT   DD SYSOUT=* 
//STDOUT2  DD SYSOUT=* 
//STDERR   DD SYSOUT=* 
//SYSDUMP  DD SYSOUT=* 
//* SABEND DD SYSOUT=* 
//CEEDUMP  DD SYSOUT=* 
//STDIN    DD DUMMY 

This runs the shell script /u/tmp/zos/cc.sh.

Because this is a shell script, there are some profiles that may run before the script executes.

The documentation says in Customizing the shell environment variables

The places to set environment variables, in the order that the system sets them, are:

1. The RACF® user profile.
2.The /etc/profile file, which is a system-wide file that sets environment variables for all z/OS shell users. This file is only run for login shells.
3.The $HOME/.profile file, which sets environment variables for individual users. This file is only run for login shells.
4.The file named in the ENV environment variable. This file is run for both login shells and subshells.
5.A shell command or shell script.

Later settings take precedence. For example, the values set in $HOME/.profile override those in /etc/profile.

Colin’s notes

  • I cant find how to set any environment variables in the RACF profile.
  • The /etc/profile is only run if a shell(sh) command is issued
  • The $HOME/.profile. This needs a home entry in the RACF userid OMVS segment ( Use the TSO RACF command LU userid OMVS to display the OMVS information)
  • I specified a file in the ENV environment variable – this was not used. If the file did not exist it did not produce an error message. When I had //STDENV DD *… in my JCL the statements were used

//STDENV

When I used

//STDPARM   DD * 
sh export 
//STDENV   DD * 
PATH=/u/abc/ 
xyz=1234 
yy="ABCD" 
xx=$xyz 
/*

The export command listed all of the environment variables. These included

PATH=”/bin:/usr/sbin:/usr/lpp/jav….
xx=”\$xyz”
xyz=”1234″
yy=”\”ABCD\””

  • My PATH statement was not used. It was overwritten by /etc/profile (or $HOME/.profile)
  • The special characters $ and ” have been escaped.
  • It is not doing shell processing, for example in a shell xx=$xyz, says assign to xxx the value of zyz. All that happens is xx is assigned the literal value $xyz

So overall – it didn’t work as I expected it to, and I need to do some redesign.

Using BPXBATSL

When I copied some JCL which used BPXBATSL I got

BPXM018I BPXBATCH FAILED BECAUSE SPAWN (BPX1SPN) OF /BIN/LOGIN FAILED WITH RETURN CODE 0000009D REASON CODE
0B1B0473

BPXBATSL is an alias of BPXBATCH. I do not think it supports the SH command.

There are many ways to fail to read a file in a C program.

The 10 minute task to read a file from disk using a C program took a week.

There are several options you can specify on the C fopen() function and it was hard to find the one that worked. I basically tried all combinations. Once I got it working, I tried on other files, and these failed, so my little task turned into a big piece of research.

Depending on the fopen options and the use of fgets or fread I got different data when getting from a dataset with two records in it!

  • AAAA – what I wanted.
  • AAAAx’15’ – with a hex 15 on the end.
  • AA
  • AAAAx’15’BBBBx’15’- both the records, with each record terminated in a line-end x’15’.
  • AAAABBBB – both records with no line ends.
  • x’00140000′ x’00080000’AAAAx’000080000’BBBB.
    • This has a Block Descriptor Word (x’00140000′) saying the length of the block is x14.
    • There is a Record Descriptor Word (RDW) (X’00080000′) of length x’0008′ including the 4 bytes for the RDW.
    • There is the data for the first record AAAA,
    • A second RDW
    • The second data
    • The size of this record is 4 for the BWD, 4 for the first RDW, 4 for the AAAA, 4 for the second RDW, and 4 for the data = 20 bytes = x14.
  • Nothing!

Sections in this blog post

There are different sorts of data

  • z/OS uses data sets which have records. Records can be blocked (for better disk utilisation) It is more efficient to write one block containing 40 * 80 bytes records than write 40 blocks each of 80 bytes. You can treat the data as one big block (of size 3200 bytes) – or as 40 blocks of 80 (or 2 blocks of 1600 …) . With good blocking you can get 10 times the amount of data onto a disk. (If you imagine each record on disk needs a 650 byte header you will see why blocking is good).
  • You can have “text” files, which contain printable characters. They often use the new line character (x’15’) to indicate end of line.
  • You can have binary files which should be treated as a blob. In these characters line x’15’ do not represent new line. For example it might just be part of a sequence x’13141516′.
  • Unix (and so OMVS) uses byte-addressable storage.
    • “Normal files” have data in EBCDIC
    • You can use enhanced ASCII. Where you can have files with data in ASCII (or other code page). The file metadata contains the type of data (binary or text) and the code page of the text data.
    • With environment _BPXK_AUTOCVT you can have automatic conversion which converts a text file in ASCII to EBCDIC when it is read.

From this you can see that you need to use the correct way of accessing the data(records or byte addressable), and of using the data(text or binary).

Different ways of getting the data

You can use

  • the fread function which reads the data, according to the file open options (binary|text, blocked)
  • the fgets function which returns text data (often terminated by a line end character).g

Introduction to fopen()

The fopen() function takes a file name and information on how the file should be opened and returns a handle. The handle can be used in an fread() function or to provide information about the file. The C runtime reference gives the syntax of fopen(), and there is a lot of information in the C Programming guide:

There is a lot of good information – but it didn’t help me open a file and read the contents.

The file name

The name can be, for example:

  • “DD:SYSIN” – a reference to a data set via JCL
  • “//’USER.PROCLIB(MYPROC)'” – a direct reference to a dataset
  • “/etc/profile” – an OMVS file.

If you are using the Unix shell you need to use “//’datasetname'” with both sets of quotes.

The options

This options string has one or more parameters.

The first parameter defines the operation, read, write, read+write etc. Any other parameters are in the format keyword=format.

The read operation can be

  • “rb” to read a binary file. In a binary file, the data is treated as a blob.
  • “rt” to read a text file. A text file can have auto conversion see FILETAG C run time option.
  • “r”

The type can be

  • type=record – read a record of data from the disk, and give the record to the application
  • type=blocked. It is more efficient in disk storage terms, to build up a record of 10 * 80 byte records into one 800 byte record (or even 3200 byte records). When a blocked record is read, the read request returns the first N bytes, then the next n bytes. When the end of the block is reached it will get retrieve the next block from disk.
  • not specified, which implies byte access, rather than record access, and use of fgets function.

An example fopen

FILE * f2 fopen(“//’COLIN.VB'”,”rt type=blocked”) ;

How much data is read at a time?

Depending on the type of data, and the open options a read request may return data

  • up to the end of the record
  • up to and including the new-line character
  • up to the size of the buffer you give it.

If you do not give it a big enough buffer you may have to issue several reads to get whole logical record.

The system may also “compress” data, for example remove trailing blanks from the end of a line. If you were expecting an 80 byte record – you might only get 20 bytes – so you may need to check this, and fill in the missing blanks if needed.

Reading the data

You can use the fread() function

The fread() parameters are:

  • the address of a buffer to hold the data
  • the unit of the buffer block size
  • the number of buffer blocks
  • the file handle.

It returns the number of completed buffer blocks used.

It is usually used

#define lBuffer 1024

char buffer[lBuffer];
size_t len = fread(buffer, 1 ,lBuffer ,fHandle );

This says the unit of the buffer is 1 character, and there are 1024 of them.

If the record returned was 80 bytes long, then len would be 80.

If it was written as

size_t len = fread(buffer, lBuffer, 1 ,fHandle );

This says the unit of buffer is 1024 – and there is one of them. The returned “length” is 0, as there were no full 1024 size buffers used.

It appears that a returned length of 0 means either end of file or a file error. You can tell if the read is trying to go past the end of the file (feof()) or was due to a file error (ferror()).

if (feof(hFile))…
else if (ferror(hFile))
{ int myerror = errno;
perror(“Colins error”);

}

You can use the fgets function.

The fgets() parameters are:

  • the address of a buffer to hold the data
  • the size of the buffer
  • the file handle.

This returns a null terminated string in the buffer. You can use strlen(buffer) to get the length of it. Any empty string would have length 0.

fgets()returns a pointer to the string, or NULL if there is an error. If there is an error you can use feof() or ferror() as above.

Results from reading data sets and files

I did many tests with different options, and configurations, and the results are displayed below.


The following are using in the tables:

  • Run.
    • job. The program was run in JCL using BPXBATSL or via PGM=…
    • unix shell – the program was run in OMVS shell.
  • BPXAUTO. This is used in OMVS shell. The environment variable _BPXK_AUTOCVT can have
    • “ON” – do automatic conversion from ASCII to EBCDIC where application
    • “OFF” do not do automatic conversion from ASCII to EBCDIC
  • open. The options used.
    • r read
    • rb read binary
    • rt read test
    • “t=r” is shorthand for type=record
  • read. The C function used to read the data.
    • fread – this is the common read function. It can read text files and binary files
    • fgets – this is used to get text from a file. The records usually have the new-line character a the end of the daya
    • aread is a C program which uses fread – but the program is compiled with the ASCII option.
  • data
    • E is the EBCDIC data I was expecting. For example reading from a FB dataset member gave me an 80 byte record with the data in it.
    • E15 is EBCDIC data, but trailing blanks have been removed. The last character is an EBDCIC line end (x15).
    • A0A. The data is returned in ASCII, and has a trailing line end character (x0A). You can convert this data from ASCII to EBCDIC using the C run time function __a2e_l(buffer,len) .
    • Abuffer – data up to the size of the buffer ( or size -1 for fgets) the data in ASCII.
    • Ebuffer – data up to the size of the buffer ( or size -1 for fgets) the data in EBCDIC. This data may have newlines within the buffer

To read a dataset

This worked with a data set and inline data within a JOB.

RunBPXAUTOopenreaddata
JobNArb,t=rfreadE
r|rtfreadEbuffer
rfgetsE15
rtfgetsE15
Unix shellAnyrb,t=rfreadE
r|rtfreadEbuffer
rfgetsE15
rtfgetsE15

Read a normal(EBCDIC) OMVS file

RunBPXAUTOopenreaddata
JobOFFrb,t=rfreadE
offrtfgetsE15
off*freadEbuffer
Unix shelloffrfgetsE15
offrtfgetsE15
offrbfgetsE15
off*freadE15
onrb,t=rfreadE
onrbfgetsE15

Read an ASCII file in OMVS

RunBPXAUTOopenreaddata
JoboffragetsA0A
offrbagetsA0A
offrtagetsA0A
Unix shellonrfgetsE15
onrbfgetsE15
onrtfgetsE15
on*freadbuffer
offragetsA0A
offrbagetsA0A
offrtagetsA0A
off*freadAbuffer

To read a binary file

To read a data set

RunBPXAUTOopenreaddata
JoboffrbfreadE
Unix shellonrbfreadE

Read a binary normal(EBCDIC) OMVS file

RunBPXAUTOopenreaddata
JoboffrbfreadE
Unix shellonrbfreadE
offrbfreadE

Read a binary ASCII file in OMVS

If use list the attributes of a binary file, for example “ls -T ecrsa.p12” it gives

b binary T=off ecrsa1024.p12

Which shows it is a binary file

RunBPXAUTOopenreaddata
Joboffrb,t=rfreadE
Unix shellonrb,=rfreadE
offrb,t=rfreadE

Reading data sets from a shell script

If you try to use fopen(“DD:xxxx”…) from a shell script you will get

FOPEN: EDC5129I No such file or directory. (errno2=0x05620062)

If you use fopen(“//’COLIN.VB'”…) and specify a fully qualified dataset name if will work.

fopen(“//VB”..) will put the RACF userid in front off they name. For example attempt to open “//’COLIN.VB.'”

How big a buffer do I need?

The the buffer is not large enough to get all of the data in the record, the buffer will be filled up to the size of the data. For example using fgets and a 20 byte buffer, 19 characters were returned. The 20th character was a null. The “new-line” character was present only when the end of the record was got.

How can I tell the size of the buffer I need?

There is data available which helps – but does not give the whole picture.

With the C fldata() — Retrieve file information function you can get information from the file handle such as

  • the name of the dataset (as provided at open time)
  • record format – fixed, variable
  • dataset organisation (dsorg) – Paritioned, Temporary, HFS
  • open mode – text, binary, record
  • device type – disk, printer, hfs, terminal
  • blksize
  • maximum record length

With fstat(), fstat64() — Get status information from a file handle and lstat(), lstat64() — Get status of file name or symbolic link you can get information about an OMVS file name (or file handle). This has the information available in the OMVS “ls” command. For example

  • file serial number
  • owner userid
  • group userid
  • size of the file
  • time last access
  • time last modified
  • time last file status changed
  • file tag
    • ccsid, value or 0x000 for untagged, or 0xffff for binary
    • pure text flag.

Example output

For data sets and files (along the top of the table)

  • VB a sequential data set Variable Blocked
  • FB a member of user.proclib which is Fixed Block
  • SYSIN inline data in a job
  • Loadlib a library containing load modules (binary file)
  • OMVS file for example zos.c
  • ASCII for example z.py which has been tagged as ASCII

Other values

  • V variable format data
  • F fixed format data
  • Blk it has blocked records
  • B “Indicates whether it was allocated with blocked records”. I do not know the difference between Blk and V
  • U Undefined format
  • PS It is a Physical Sequential
  • PO It is partitioned – it as members
  • PDSE it is a PDSE
  • HFS it is a file from OMVS
VBFBSYSINloadibOMVS fileASCII
fldatarecfmV Blk BF Blk BF Blk BUUU
dsorgPSPO PDSMemPSPO PDSMem PDSEHFSHFS
devicediskdiskotherdiskHFSHFS
blocksize614461080640000
maxreclen10248080640010241024
statccsid00000819
file size00004780824

To read a record, you should use the maxreclen size. This may not be the size of the data record but it is the best guess.

It look like the maxreclen for Unix files is 1024 – I guess this is the page size on disk.