What is my Unix process doing?

I was familiar with the USS command ps -ef which displays output like

     UID        PID       PPID  C    STIME TTY       TIME CMD 
WEBSRV 16842766 1 - 07:23:05 ? 0:00 -sh -c /web/httpd1/bin/apachectl -k start -f /web/httpd1/conf/httpd.conf -DNO_f

For Zowe threads I was getting

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/inter

which was annoyingly truncated.

The command ps -e -o args > aa gives the whole command line (up to 1024 bytes) such as

/u/tmp/zowep33//bin/utils/configmgr -script /u/tmp/zowep33//bin/commands/internal/start/component/cli.js

Another useful command when you know it.

Oh p*x, it didn’t copy across some files.

I had managed to mess up the files for a product, so I wanted to copy them across from an older system.

This worked for some of the files – but when I came to start the subsystem – it was missing some files! For example /u/my/zosmf/liberty/lib/native/zos/s390x/bbgzsrv

I copied the files across again – and they were still not there!

Once you know the answer it is obvious…

There is a directory /usr/lpp/zosmf/liberty – and it was this directory that was missing.

Once I looked into it more carefully – this was not a directory, but a symbolic link to another directory liberty -> ../liberty_zos/current

To fix this I used

# go to my version of zosmf
cd /u/my/zosmf
# remove the symbolic link
rm liberty
#make the new link
ln -s /usr/lpp/liberty_zos/current liberty

and now I could use ls /u/tmp/zosmfp/liberty/lib/native/zos/s390x/bbgzsrv and it found the file.

If I had checked this before I started, I would have save myself a half day of IPLing older systems!

Programming shared memory – more head banging.

I was trying to use shared memory (to look at Java Shared Classes), and it took me a day to get it working – better documentation would have helped.

I had two basic problems

  1. Using smctl to display information about the shared memory, gave the size as 0 bytes, even though the ipcs command showed me there were megabytes of data in the shared memory area.
  2. Trying to attach the shared memory gave me “invalid parameters” return code – even though the documented reasons for this error code did not apply to my program.

I tried many things, from using different userids, to running with a different storage key, running APF authorised….

I eventually got it to work by compiling my C program in 64 bit mode rather than 31 bit mode. There is no discussion about 31 bit/64 bit in the documentation. If the shared memory in 64 bit mode, you will need 64 bit addressability, so you need a 64 bit program. But there is no way of determining that the shared memory is 64 bit!

My basic program

{ 
//struct shmid_ds buf;
struct shmid_ds64 buf;
memset(&buf ,0,sizeof(buf));
int shmid = 8197;
int rc = 0;
long l;
int cmd = IPC_STAT;
char * fn = "COLIN";
int shmflg =0;
shmflg = IPC_STAT;
// rc =shmctl(shmid, cmd, &buf);
rc =shmctl64(shmid, cmd, &buf);
perror("shmctl " );
printf("shctl rc %i\n",rc);
l = buf.shm_segsz;
printf("size %ld\n",l);
printHex(stdout,&buf,sizeof(buf));
///////////////////////////////////////////////
// shmat
///////////////////////////////////////////////
char * pData = NULL;
pData = shmat(shmid, NULL , 0 );
printf("Address %ld\n",pData);
printHex(stdout,pData+4096*1024,1024*1024);
int e = errno;
perror("shmat ");
printf("Errno: %s\n",strerror(e));
return 0;
}

Originally I was using EDCCB to compile and bind this.

The EINVAL error return code was (from the documentation) for cases where the pointer in shmat was non NULL. I was passing NULL – so none of this made sense.

The reason code 0717014A was

JRInvalidAmode: An incorrect access mode was specified on the access service
Action: The access mode specified on the access service has unsupported bits turned on. Reissue the request and specify a valid access mode.

It turned out that my program was 31 bit. When I used 64 bit – it magically started working.

I compiled it with EDCQCB, and had to change a few things to be 64 bit mode.

  • shmid_ds buf -> shmid_ds64
  • shmctl -> shmctl64

When I ran it in 31 bit mode, the length of the storage returned was 0. In 64 bit mode, it gave the correct length. This looks like a way of telling what mode the shared memory is!

How do I diff on Z/OS with Unix files and directories

I wanted to compare the contents of two directories in Unix System Services on z/OS before I merged them. This took me some time to do because the documented is lacking.

With ISPF you can use SUPERC (3.13), give it two PDSs and it shows you the differences.

On Unix there is the diff command. This can compare individual files, or directories. It can display just the changes, or the changes in context.

File a

AOnly 
line1
Aline2
line3

File b

line1 
Bline2
line3
BOnly
BOnly2

Using ispf edit compare to show differences

You can use diff to show the differences in two files, but it is not easy to understand. ISPF EDIT has the compare facility. If you know two files are different you can use

  • oedit /etc/zexpl/rseapi.env
  • use the primary command compare
    • enter the fully qualified name /u/ibmuser/zexpl/rseapi.env,, in the “Name . . . . .” field. if you specify +/filename the + means the same directory.
    • press pf3 and it will show the differences

Because I tend to remove comments to make it easier to see the content, I tend to use

  • oview /etc/ssh/sshd_config you get into ISPF edit, but no changes are saved.
  • Comments start with a #. x all;f ‘#’ all 1 1; f ‘ ‘ 1 1 all;del all nx removes comment lines and blank lines
  • reset to show the hidden lines
  • compare I then specify my version of the file, and see the changes.

  • Lines like ====== TrustedUserCAKeys /etc/ssh/user_ca_key.pub are from my copy.
  • Lines in green with a line number are in both files 000006 Subsystem sftp /usr/lib/ssh/sftp-server
  • Lines like .OAAAA UseDNS yes are from the base file

Update your version, make a copy of the original, then replace the original with your version.

diff -c1 a b – show the files and the changes in context

The line prefix for input file going to output file

  • – to be removed
  • ! to be changed
  • + to be added

The command diff -c1 a b gives

*** a Tue Aug 27 02:48:56 2024              
--- b Tue Aug 27 02:50:41 2024
***************
*** 1,4 ****
- AOnly
line1
! Aline2
line3
--- 1,5 ----
line1
! Bline2
line3
+ BOnly
+ BOnly2
  • *** a Tue Aug 27 02:48:56 2024 the first file name, and last changed date
  • — b Tue Aug 27 02:50:41 2024 the second file name, and last changed date
  • *** 1,4 **** the*** show it is file 1, lines 1 to 4
  • – AOnly this line is in file a is not in file b, so would need to be removed(-) from file a
  • ! Aline2 this line is in file b – but different
  • — 1,5 —- this is file b, lines 1 to 5
  • ! Bline2 this line is also in file a – but different
  • + BOnly this line is in file b and and was additional(+) to file a

When one file exists but is empty you get output like

*** /etc/resolv.conf Wed Mar  6 11:54:50 2024                         
--- /u/ibmuser/temp/resolv.conf Thu Dec 7 05:40:24 2023
***************
*** 0 ****
--- 1,2 ----
+ nameserver 127.0.0.1
+ TCPIPJOBNAME TCPIP

which follows the rules I explained above. *** 0 **** shows the content after line 0 is empty, because the next line is — 1,2 —- from the other file.

diff a b – show just the changes

gives

1d0             
< AOnly

3c2
< Aline2
---
> Bline2

4a4,5
> BOnly
> BOnly2

The output can be split into sections. The first line of each section is like

  • 1d0 the first line of a needs to be deleted from b, line 0
  • 3c2 line 3 of a is changed from line 2 of b
  • 4a4,5 lines 4,5 of b need to be added to a

The < and > tell you which file the data came from

When data is changed it gives the lines

  • < content of file a
  • output divider
  • > content of file b

When the data is in file a and not file b

  • < contents of file a

When the data is in file b and not file a

  • > contents of file b

diff -s dir1 dir2 compare the directory contents

If you specify -s, or just specify two directories, it compares the directory content.

You can use

diff -c1 dir1 dir2 

the -c1 to display the contents (how I like it).

With the directory entries you get records like

Only in /u/ibmuser/temp: test.tar 
Common subdirectories: /etc/wbem and /u/ibmuser/temp/wbem
Only in /etc: yylex.c
diff -c1 /etc/hosts /u/ibmuser/temp/hosts
*** /etc/hosts Wed Mar 6 11:06:55 2024
--- /u/ibmuser/temp/hosts Tue Feb 28 12:43:07 2023

You can find which files are missing from /etc , by using grep ‘Only in /u/ibmuser/temp’ on the output.

It shows the command used for the individual files, and the output

diff -c1 /etc/hosts /u/ibmuser/temp/hosts
*** /etc/hosts Wed Mar 6 11:06:55 2024
--- /u/ibmuser/temp/hosts Tue Feb 28 12:43:07 2023
...

diff -s -r dir1 dir2 compare the directory contents

The -r option displays the data recursively.

Restoring pax files onto z/OS

Some products and packages that run in Unix System Services provide .pax files you download and install. Often the instructions are not very clear.

The steps are usually:

  1. Download the .pax file to your work station
  2. Upload the file to z/OS ( usually Unix Services, but a sequential data set can be used.
  3. Optionally create a ZFS file system, or find space on an existing file system
  4. Unload the file

Download the .pax file to your work station

This is usually pretty simple – just make sure you are downloading the correct file. I spent a short while trying to get a .tar.gz file for Linux installed.

Upload the file to z/OS ( usually Unix Services, but a sequential data set can be used.

  • Upload the file in binary
  • I usually upload it into a zfs. This means you do not need to allocate space, and DCB information.

Optionally create a ZFS file system, (or find space on an existing file system)

This is where it starts to get harder – as there is less useful information. The first question is how much space do you need? Sometimes you get information in (mega) bytes, sometimes in number of 512 byte records, sometimes in number of 8KB blocks.

If you need to create a ZFS, you need to create a directory on the file system where you want the file system mounted, for example

mkdir /usr/lpp/java/J11.0.19.0_64

My pax file was in /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z.

You can list the contents of the file using

pax -ppx -Evzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z 1>aout 2>b

where aout has content like

drwxr-xr-x      1 JENKINS USERGRP     0 May 16 20:31 J11.0_64/lib/ 
-rwxr-xr-x apsl 1 JENKINS USERGRP 98304 May 16 20:13 J11.0_64/lib/default/libcuda4j29.so

Each line has extended attributes and the file size.

  • If the first character is “d” then this is a directory,
  • if it is a “-“, it is a file.
  • The sixth column is the size of the file.

You can use the following command to give the size of all of the files.

cat aout | awk ' substr($0,1,1) == "-" {print $0}' | awk '{sum+=$6;}END{print sum;}'

This command

  • passes the file aout into the awk command
  • awk… if the first character is a “-” =, it is a file, so print the record
  • awk…add up the 6th column and display the final result.

This gave me 5.24403e+08 (bytes) for the size of all of the files. You need to add perhaps 5-10% for overhead (for example directory entries). If you specify a secondary extend to the data set, it will try to expand if the ZFS fills up.

I created the ZFS

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE   EXEC   PGM=IDCAMS 
//SYSPRINT DD     SYSOUT=* 
//SYSIN    DD     * 
  DELETE               JVB800.V11.ZFS    CLUSTER 
  SET MAXCC=0 
  DEFINE                - 
    CLUSTER             - 
    (NAME(JVB800.V11.ZFS)- 
    VOLUMES(USER02)                   - 
    LINEAR              - 
    MEGABYTES(500 254)  - 
    SHAREOPTIONS(3 3)) 
/* 
//FORMATFS EXEC   PGM=IOEAGFMT,REGION=0M,COND=(0,NE,DEFINE), 
// PARM=('-aggregate JVB800.V11.ZFS    -compat') 
//SYSPRINT DD     SYSOUT=* 
//STDOUT   DD     SYSOUT=* 
//STDERR   DD     SYSOUT=* 
//* 

and mounted it over the directory you created above

//MOUNT    EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE),(0,NE,FORMATFS))
//SYSTSPRT DD   SYSOUT=* 
//SYSTSIN  DD   * 
    MOUNT FILESYSTEM('JVB800.V11.ZFS') TYPE(ZFS) + 
    MOUNTPOINT('/usr/lpp/java/J11.0.19.0_64') + 
     MODE(RDWR) PARM('AGGRGROW') AUTOMOVE 
/* 

You can use the MOUNT command in the BPXPRMxx member

    MOUNT FILESYSTEM('JVB800.V11.ZFS') TYPE(ZFS) 
    MOUNTPOINT('/usr/lpp/java/J11.0.19.0_64')
     MODE(RDWR) PARM('AGGRGROW') AUTOMOVE

Note: without the “+” signs.

Size after unpacking is similar to the calculated value above.

After the file was unpacked the command

df -P /usr/lpp/java/J11.0_64

reported

Filesystem     512-blocks   Used  Available Capacity Mounted on
JVB800.V11.ZFS 1178880   1045634     133246       89% /usr....

“Used blocks” 1045634 * 512 bytes gives 535364608 = 5.4e+08

The size calculation above is close to the final space used.

Unload the file

When I first unpacked one of the files I got messages about not being authorised to set the shared library attribute(l) in the directory – even though my userid was a super user. I had to define a security profile, and give my userid access to it.

RDEFINE FACILITY BPX.FILEATTR.SHARELIB UACC(NONE)
PERMIT BPX.FILEATTR.SHARELIB CLASS(FACILITY) ID(IBMUSER) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH

Finally! In Unix System Services I issued the commands

cd /usr/lpp/java/J11.0.19.0_64
pax -ppx -rvzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z

This unloaded the files into /usr/lpp/java/J11.0_64/J11.0_64

Discussion about where to put the file system.

If I had an existing directory /usr/lpp/java/J11.0_64 I could have mounted the new ZFS at this address, and issued

cd /usr/lpp/java
pax -ppx -rvzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Zf

and this would have unload the files into /usr/lpp/java/J11.0_64, overwriting what was already there. See Follow the instructions to install Java on z/OS and screw up production. Think carefully where you want to put your files.