LPA for Unix System Services .so modules

I discovered a shared library facility for loading OMVS modules (for example .so) into storage which can be shared by all OMVS address spaces.

This is an unusual blog, in that I’ve written about a topic – then say “do not use it!

Phil Wakelyn of CICS strategy said….

The UNIX shared library region, this is not something we currently recommend – see this doc, and its now disabled by default in CICS at the JVM level using the variable

_BPXK_DISABLE_SHLIB=NO

The reasons  for this is that that:

  • It has negligible difference on JVM startup time
  • It has a substantial negative impact on virtual storage below the bar, as storage is allocated from the high private area in MB chunks for each library (dll), so there is lots of wasted space and you can quickly use up 100MB or more. This has been a large customer support issue for CICS customers where space is very tight below the bar and MVS private storage SOS conditions are fatal.
  • Shared libraries are loaded once per address space, so are cached at the address space level

Where are modules stored?

They are stored in a common area below the 31 bit line – so taking storage from all regions.

If you issue the command

D OMVS,L

it gives

BPXO051I 10.32.34 DISPLAY OMVS 293                              
OMVS     0010 ACTIVE             OMVS=(00,01,BP,IZ,RZ,BB,ZW,PY) 
SYSTEM WIDE LIMITS:         LIMMSG=NONE                         
                  CURRENT  HIGHWATER     SYSTEM                 
                    USAGE      USAGE      LIMIT                 
...               
SHRLIBRGNSIZE    58720256   65011712   67108864   
SHRLIBMAXPAGES          0          0     409600 *  

I made it larger than the default (4096) by using the operator command

SETOMVS SHRLIBMAXPAGES=409600

How do you load modules into the shared region?

You need to set the extended attribute +l for example

extattr +l /usr/lpp/java/J8.8_64/J8.0_64/bin/j9vm/libdbgwrapper80.so

When this module is loaded, it will be loaded into the shared region – if there is enough space.

Be careful how you reference the modules

You should use a consistent reference to files. For example

/usr/lpp/java/J8.8_64/libj9a2e.so is the same file as /Z24C/usr/lpp/java/J8.8_64/libj9a2e.so, but the shared library will treat them as two different objects, and load both of them. This will waste space.

How do you unload a module from the shared region

I reset the attribute using extattr -l. This did not unload it. When it was next loaded, it appeared to be unloaded from the shared region, and the file on disk was used.

How do you know what is in the shared region?

There is no IBM answer. There is a rexx exec OMVS command written by an IBMer

The syntax is

wjsigshl -p

This gives output

Usage  Meg Used-Unused-Pgs Pathname 
    1    1       8     248 647857ED xxx/libracfimp.so 
...
    1   25    6349      51 647857EC xxx/compressedrefs/libj9jit29.so 
...
 Total Storage (Meg)             60 
 Total Module (Pages)         10146 
 Total Unused (Pages)          5214 
 Total Module Count              28 

I changed /Z24C/usr/lpp/java/J8.8_64/J8.0_64/lib/s390x to xxx so the output would fit within the area.

Each modules gets storage in multiples of 1MB, so you waste space with small objects.

The used pages does not tie up directly with the size of the object on disk. For example for libjgsk8iccs_64.so

  • using ls -ltr the size is 913408
  • from the shared region mapping information is it 379648

It may be that there is information in the disk version which is not needed once the module is loaded.

How much space do I need?

I set the +l attribute for all of the .so objects in Java V8 SR 8. When I ran my Java program (which uses TLS) there were 28 modules loaded, and 60 MB of data used.

How do I turn it off for a job?

For your job, you can use the environment variable

_BPXK_DISABLE_SHLIB=YES

The documentation says

System shared libraries are disabled. When loading a program with the system shared library extended attribute (st_sharelib), the attribute is ignored and the program is loaded into the caller’s private storage. The _BPXK_DISABLE_SHLIB setting is propagated on both fork and spawn from the parent to the child process.

End words

Now you know how it works – do not use it.
I asked on one of the z/OS news groups if anyone used this facility and unusually I got no replies. It looks like it is not used in the z/OS community.

Turbo start your Java program on z/OS and save a bucket of CPU

This blog post follows on from Some of the mysteries of Java shared classes and gives some CPU figures.

This should help you with any of the Java applications running on z/OS, such as z/OSMF, z/OS Connect, MQWEB, RSEAPI, and ZOWE.

I ran the scenarios on z/OS on zPDT running on my Ubuntu Linux machine, and so the figures are nothing like you may expect on a real z/OS machine – but my figures should show you the potential.

Topics covered:

Overview of Java shared classes

With Java shared classes support, as a Java program starts, and reads the jar and class files and also copies them into memory somewhere. Successive start can use the in memory copy and avoid the read for disk and initial processing.

You can save the in-memory copy to disk, and restore this disk copy to memory, for example across IPLs.

Measurements

I measured the CPU user from the address space once the system was started

The Java program provides a high level trace. I note the time difference between the first message and the “I am up” message

Scenarios

I used three scenarios

  • IPL and start Java program with no share classes
  • Enable shared classes
  • IPL and restore the shared classes, and start the program

No shared classes

ScenarioCPUDuration seconds
First run after IPL 394172
Second run425183

Enable shared classes

I enabled shared classes by using the Java option

-Xshareclasses:verbose,name=rseapi,cachedir=/tmp/,groupAccess,nonpersistent,nonfatal,cacheDirPerm=0777″

ScenarioCPUDuration seconds
run after shared classes enabled 500200
Second run after shared classes enabled 292116
Third run after shared classes enabled25181

IPL and restore snapshot

ScenarioCPUDuration seconds
First run after (IPL and restore snapshot )27499
Second run272121
Third run279116
Fourth run264111

Analysis of the results

Using the shared classes saved CPU in the region of 25% and reduced the elapsed time by about a half.

The first time the Java program runs and creates the shared class data has a higher CPU cost, and increased elapsed time. The savings of CPU and elapsed time when the shared cache is reused outweighs this one time cost.

Observation

It appears that each time you restart using shared classes the CPU drops. I think this is due to the optimisation being done on the classes, but it may be some totally different effect – or it may just be co-incidence!

Setting up to use the shared classes

I added two job steps to my Java program JCL

Before – restore the share classes cache from the backup copy

// EXPORT SYMLIST=* 
// SET J='/usr/lpp/java/J8.8_64/J8.0_64/bin' 
// SET C='/tmp/' 
// SET N='rseapi' 
// SET V='restoreFromSnapshot'
// SET Q='cacheDirPerm=0777,groupAccess' 
//RESTORE  EXEC PGM=BPXBATCH,REGION=0M,PARMDD=PARMDD 
//PARMDD  DD *,SYMBOLS=(JCLONLY) 
SH &J/java -Xshareclasses:cacheDir=&C,name=&N,&V,&Q 
/* 

If the in-memory cache exists you get message

JVMSHRC726E Non-persistent shared cache “rseapi” already exists. It cannot be restored from the snapshot.

After – save the shared class cache to disk

// SET V='snapshotCache' 
// SET J='/usr/lpp/java/J8.8_64/J8.0_64/bin' 
//SAVECAC  EXEC PGM=BPXBATCH,REGION=0M, 
//   PARM='SH &J/java -Xshareclasses:cacheDir=&C,name=&N,&V' 
//STDERR   DD   SYSOUT=* 
//STDOUT   DD   SYSOUT=* 

Strange behaviour

By using the startup option -verbose:class,dynload you can get information about the classes as they are loaded.

When not using shared classes, there were records saying <Loaded ….. and giving durations of the loads etc.

When using shared classes there were still a few instances of <Loaded… . I could not find out why some classes were read from disk , and the rest were read from the shared classes cache.

If we could fix these, then the startup would be even faster!

After some investigation I can explain some of the strange behaviour.

  • When a jar is first used there is a <Loaded… for the class that requested the jar.
  • A class like <Loaded sun/reflect/GeneratedMethodAccessor1 with a number at the end gets a <Loader… entry.
  • Some other classes in a jar file get loaded using <Loader… though they do not look any different to classes which are loaded from the shared cache!

All in all, very strange.

Where do you harden the cache to?

By default the cache is saved to /tmp. As /tmp is often cleared at IPL, this means the cache will not exist across IPLs. You may wish to save it in an instance specific location such as /var/myprogram.

What happens if I change my Java program?

I had a small test program which I recompiled, and created the jar file. The Java source was

public class hw   { 
  public static void main(String[] args) throws Exception { 
    System.out.println("This will be printed"); 
    System.out.println("HELLo" )  ; 
    CPUtil.print(); // this prints Util.line 10 
    hw2.print(); 
  } 
} 

When I reran the program the output contained

JVMSHRC169I Change detected in /u/adcd/hw.jar... 
  ...marked 3 cached classes stale 
class load: sun/launcher/LauncherHelper$FXHelper from: .../lib/rt.jar 
<Loaded CPUtil> 
<  Class size 427; ROM size 416; debug size 0> 
<  Read time 4 usec; Load time 108 usec; Translate time 595 usec> 
class load: CPUtil from: file:/u/adcd/hw.jar 
Output from CPUtil.line 10 
<Loaded hw2> 
<  Class size 386; ROM size 368; debug size 0> 
<  Read time 3 usec; Load time 107 usec; Translate time 635 usec> 
class load: hw2 from: file:/u/adcd/hw.jar 

Where you can see output from my program is intermixed with the loader activity.

What happens internally

From the previous topic, it seems that Java has to read the files on disk for example to spot that a class has changed. This may just be a matter of reading the time stamp of the file on disk,or it may go into the file itself.

Should I use .class files or package the .class files into a .jar files?

This will be a hand waving type answer. Generally the answer is use a .jar file.

Use one .jar fileUse multiple .class files
One directory access and one security access check should reduce the CPU usage.Multiple directory access and multiple security checks are required.
Reading one large file may be faster than reading many smaller files. An I/O has “set-up I/O”, “transfer data”, “shutdown I/O” there is one set-up and one shutdown.Each file I/O has set-up and shutdown time as well as the transfer time and is generally slower than processing bigger files. (Think about large block sizes for data sets).
The .jar files are compressed so there is less data to transfer. The decompression of the jar file takes CPU.Files do not need to be decompressed
For integrity reasons you can have your .jar file cryptographically signed.You cannot sign .class files.

Should I use of BPXBATCH or BPXBATSL?

In the Tomcat script for starting the web server it issued

exec "/usr/lpp/java/J8.8_64/J8.0_64/bin/java" ...  &

The & makes it run in the background. As I was running this as a started task, this seemed unnecessary and removed the &.

I also used EXEC PGM=BPXBATSL instead of EXEC PGM=BPXBATCH

The combination of both reduced the start time significantly!

I had to specify environment variable _BPX_SPAWN_SCRIPT=YES to be able to run the script. Without it I got

BPXM047I BPXBATCH FAILED BECAUSE SPAWN (BPX1SPN) OF … FAILED WITH RETURN CODE 00000082 REASON CODE 0B1B0C27

Problems I experienced while setting this up.

Group access

When restoring from a snapshot I used

java -Xshareclasses:cacheDir=/tmp,name=rseapi’,restoreFromSnapshot’, cacheDirPerm=0777,groupAccess’

Which worked.

When I omitted the group Access I had the following messages in stderr of my Java program.

JVMSHRC020E An error has occurred while opening semaphore 
JVMSHRC336E Port layer error code = -197358 
JVMSHRC337E Platform error message: semget : EDC5111I Permission denied. 
JVMSHRC028E Permission Denied 
JVMSHRC670I Error recovery: attempting to use shared cache in readonly mode if the shared memory region exists, in response to "-Xshareclasses:nonfatal" option.                                                                                                                      
JVMSHRC659E An error has occurred while opening shared memory 
JVMSHRC336E Port layer error code = -393966 
JVMSHRC337E Platform error message: shmget : EDC5111I Permission denied. 
JVMSHRC028E Permission Denied 
JVMSHRC627I Recreation of shared memory control file is not allowed when running in read-only mode. 
JVMSHRC840E Failed to start up the shared cache. 
JVMSHRC686I Failed to startup shared class cache. Continue without using it as -Xshareclasses:nonfatal is specified c

The OMVS command ipcs -m gave

>ipcs -m
IPC status as of Mon Aug 21 17:33:54 2023
Shared Memory:
T ID KEY MODE OWNER GROUP
m 8196 0x6100c70e --rw-rw---- OMVSKERN SYS1
m 8197 0x6100c30e --rw------- OMVSKERN STCGROUP

When the correct group access was specified the ipcs -m command gave

>ipcs -m
IPC status as of Mon Aug 21 17:38:40 2023                         
Shared Memory:                                                    
T         ID     KEY        MODE       OWNER    GROUP             
m       8196 0x6100c70e --rw-rw---- OMVSKERN     SYS1             
m      73733 0x6100c30e --rw-rw---- OMVSKERN STCGROUP             

and the group mode has values -rw.

Wrong owner

I submitted a job to run Java which created the shared cache. I then tried running the same program using a started task with a different userid.

The cache on disk had access

-rw-rw----   1 COLIN    SYS1          32 Aug 25 11:05 C290M4F1A64_semaphore_zosmf_G41L00       
-rw-rw----   1 COLIN    SYS1          40 Aug 25 11:05 C290M4F1A64_memory_zosmf_G41L00          

But my started task was running with a different userid and group.

I got messages

JVMSHRC684E An error has occurred while opening semaphore. Control file could not be locked.         
JVMSHRC336E Port layer error code = -102                                                             
JVMSHRC337E Platform error message: EDC5111I Permission denied. (errno2=0xEF076015)                  
JVMSHRC028E Permission Denied                                                                        

I delete the cache entries, and restarted the started task. I also added another step to the started task to issue snapshotCache.

What options can I specify for Java on z/OS?

You can use the following command to list them all.

java -X -help 

On Java 8 SR 8 this gave me

The following options are non-standard and subject to change without notice. 
                                                                                                       
  -Xbootclasspath:<path>    set bootstrap classpath to <path> 
  -Xbootclasspath/p:<path>  prepend <path> to bootstrap classpath 
  -Xbootclasspath/a:<path>  append <path> to bootstrap classpath 
                                                                                                       
  -Xrun<library>[:options]  load native agent library 
                            (deprecated in favor of -agentlib) 
                                                                                                       
  -Xshareclasses[:options]  Enable class data sharing (use help for details) 
                                                                                                       
  -Xint           run interpreted only (equivalent to -Xnojit -Xnoaot) 
  -Xnojit         disable the JIT 
  -Xnoaot         do not run precompiled code 
  -Xquickstart    improve startup time by delaying optimizations 
  -Xfuture        enable strictest checks, anticipating future default
  -verbose[:(class|gcterse|gc|dynload|sizes|stack|debug)] 
                                                                                                       
  -Xtrace[:option,...]  control tracing use -Xtrace:help for more details 
                                                                                                       
  -Xcheck[:option[:...]]  control checking use -Xcheck:help for more details 
                                                                                                       
  -Xhealthcenter  enable the Health Center agent 
                                                                                                       
  -Xdiagnosticscollector enable the Diagnotics Collector 
                                                                                                       
  -XshowSettings                show all settings and continue 
  -XshowSettings:system 
                      (Linux Only) show host system or container 
                      configuration and continue 
  -XshowSettings:all            show all settings and continue 
  -XshowSettings:vm             show all vm related settings and continue 
  -XshowSettings:properties     show all property settings and continue 
  -XshowSettings:locale         show all locale related settings and continue 
                                                                                                         
Arguments to the following options are expressed in bytes. 
Values suffixed with "k" (kilo) or "m" (mega) will be factored accordingly. 
                                                                                                         
  -Xmca<x>        set RAM class segment increment to <x> 
  -Xmco<x>        set ROM class segment increment to <x> 
  -Xmn<x>         set initial/maximum new space size to <x> 
  -Xmns<x>        set initial new space size to <x> 
  -Xmnx<x>        set maximum new space size to <x> 
  -Xmo<x>         set initial/maximum old space size to <x> 
  -Xmos<x>        set initial old space size to <x> 
  -Xmox<x>        set maximum old space size to <x> 
  -Xmoi<x>        set old space increment to <x> 
  -Xms<x>         set initial memory size to <x> 
  -Xmx<x>         set memory maximum to <x> 
  -Xmr<x>         set remembered set size to <x> 
  -Xmrx<x>        set maximum size of remembered set to <x> 
  -Xmso<x>        set OS thread stack size to <x> 
  -Xiss<x>        set initial java thread stack size to <x> 
  -Xssi<x>        set java thread stack increment to <x> 
  -Xss<x>         set maximum java thread stack size to <x> 
  -Xscmx<x>       set size (or soft max size if option -XX:SharedCacheHardLimit= is 
                  present) of new shared class cache to <x> 
                                                                                                        
   -Xscminaot<x>   set minimum shared classes cache space reserved for AOT data to <x> 
   -Xscmaxaot<x>   set maximum shared classes cache space allowed for AOT data to <x> 
   -Xmine<x>       set minimum size for heap expansion to <x> 
   -Xmaxe<x>       set maximum size for heap expansion to <x> 
                                                                                                        
 Arguments to the following options are expressed as a decimal from 0 to 1. 
 A value of 0.3 represents a request of 30% 
                                                                                                        
   -Xminf<x>       minimum percentage of heap free after GC 
   -Xmaxf<x>       maximum percentage of heap free after GC

 Arguments to the following options are expressed a decimal numbers. 
                                                                                                                
   -Xgcthreads<x>                set number of GC threads 
   -Xnoclassgc                   disable dynamic class unloading 
   -Xclassgc                     enable dynamic class unloading 
   -Xalwaysclassgc               enable dynamic class unloading on every GC 
   -Xnocompactexplicitgc         disable compaction on a system GC 
   -Xcompactexplicitgc           enable compaction on every system GC 
   -Xcompactgc                   enable compaction 
   -Xnocompactgc                 disable compaction 
   -Xlp                          enable large page support 
   -Xrunjdwp:<options>           enable debug, JDWP standard options 
   -Xjni:<options>               set JNI options 
                                                                                                                  

I noticed that some products using Java have the nice option

-Xoptionsfile=/var/xxxf/configuration/colin_override.cfg

Where the file has a list of JVM options one on each line.

This option is not generally available.

Restoring pax files onto z/OS

Some products and packages that run in Unix System Services provide .pax files you download and install. Often the instructions are not very clear.

The steps are usually:

  1. Download the .pax file to your work station
  2. Upload the file to z/OS ( usually Unix Services, but a sequential data set can be used.
  3. Optionally create a ZFS file system, or find space on an existing file system
  4. Unload the file

Download the .pax file to your work station

This is usually pretty simple – just make sure you are downloading the correct file. I spent a short while trying to get a .tar.gz file for Linux installed.

Upload the file to z/OS ( usually Unix Services, but a sequential data set can be used.

  • Upload the file in binary
  • I usually upload it into a zfs. This means you do not need to allocate space, and DCB information.

Optionally create a ZFS file system, (or find space on an existing file system)

This is where it starts to get harder – as there is less useful information. The first question is how much space do you need? Sometimes you get information in (mega) bytes, sometimes in number of 512 byte records, sometimes in number of 8KB blocks.

If you need to create a ZFS, you need to create a directory on the file system where you want the file system mounted, for example

mkdir /usr/lpp/java/J11.0.19.0_64

My pax file was in /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z.

You can list the contents of the file using

pax -ppx -Evzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z 1>aout 2>b

where aout has content like

drwxr-xr-x      1 JENKINS USERGRP     0 May 16 20:31 J11.0_64/lib/ 
-rwxr-xr-x apsl 1 JENKINS USERGRP 98304 May 16 20:13 J11.0_64/lib/default/libcuda4j29.so

Each line has extended attributes and the file size.

  • If the first character is “d” then this is a directory,
  • if it is a “-“, it is a file.
  • The sixth column is the size of the file.

You can use the following command to give the size of all of the files.

cat aout | awk ' substr($0,1,1) == "-" {print $0}' | awk '{sum+=$6;}END{print sum;}'

This command

  • passes the file aout into the awk command
  • awk… if the first character is a “-” =, it is a file, so print the record
  • awk…add up the 6th column and display the final result.

This gave me 5.24403e+08 (bytes) for the size of all of the files. You need to add perhaps 5-10% for overhead (for example directory entries). If you specify a secondary extend to the data set, it will try to expand if the ZFS fills up.

I created the ZFS

//IBMUZFS  JOB ,' ',COND=(4,LE) RESTART=MOUNT 
//DEFINE   EXEC   PGM=IDCAMS 
//SYSPRINT DD     SYSOUT=* 
//SYSIN    DD     * 
  DELETE               JVB800.V11.ZFS    CLUSTER 
  SET MAXCC=0 
  DEFINE                - 
    CLUSTER             - 
    (NAME(JVB800.V11.ZFS)- 
    VOLUMES(USER02)                   - 
    LINEAR              - 
    MEGABYTES(500 254)  - 
    SHAREOPTIONS(3 3)) 
/* 
//FORMATFS EXEC   PGM=IOEAGFMT,REGION=0M,COND=(0,NE,DEFINE), 
// PARM=('-aggregate JVB800.V11.ZFS    -compat') 
//SYSPRINT DD     SYSOUT=* 
//STDOUT   DD     SYSOUT=* 
//STDERR   DD     SYSOUT=* 
//* 

and mounted it over the directory you created above

//MOUNT    EXEC PGM=IKJEFT1A,COND=((0,NE,DEFINE),(0,NE,FORMATFS))
//SYSTSPRT DD   SYSOUT=* 
//SYSTSIN  DD   * 
    MOUNT FILESYSTEM('JVB800.V11.ZFS') TYPE(ZFS) + 
    MOUNTPOINT('/usr/lpp/java/J11.0.19.0_64') + 
     MODE(RDWR) PARM('AGGRGROW') AUTOMOVE 
/* 

You can use the MOUNT command in the BPXPRMxx member

    MOUNT FILESYSTEM('JVB800.V11.ZFS') TYPE(ZFS) 
    MOUNTPOINT('/usr/lpp/java/J11.0.19.0_64')
     MODE(RDWR) PARM('AGGRGROW') AUTOMOVE

Note: without the “+” signs.

Size after unpacking is similar to the calculated value above.

After the file was unpacked the command

df -P /usr/lpp/java/J11.0_64

reported

Filesystem     512-blocks   Used  Available Capacity Mounted on
JVB800.V11.ZFS 1178880   1045634     133246       89% /usr....

“Used blocks” 1045634 * 512 bytes gives 535364608 = 5.4e+08

The size calculation above is close to the final space used.

Unload the file

When I first unpacked one of the files I got messages about not being authorised to set the shared library attribute(l) in the directory – even though my userid was a super user. I had to define a security profile, and give my userid access to it.

RDEFINE FACILITY BPX.FILEATTR.SHARELIB UACC(NONE)
PERMIT BPX.FILEATTR.SHARELIB CLASS(FACILITY) ID(IBMUSER) ACCESS(READ)
SETROPTS RACLIST(FACILITY) REFRESH

Finally! In Unix System Services I issued the commands

cd /usr/lpp/java/J11.0.19.0_64
pax -ppx -rvzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Z

This unloaded the files into /usr/lpp/java/J11.0_64/J11.0_64

Discussion about where to put the file system.

If I had an existing directory /usr/lpp/java/J11.0_64 I could have mounted the new ZFS at this address, and issued

cd /usr/lpp/java
pax -ppx -rvzf /tmp/ibm-semeru-certified-jdk_s390x_zos_11.0.19.0.pax.Zf

and this would have unload the files into /usr/lpp/java/J11.0_64, overwriting what was already there. See Follow the instructions to install Java on z/OS and screw up production. Think carefully where you want to put your files.

Adding more disk space to z/OS, creating volumes and adding them to SMS.

I had managed to fill up my user volumes for my data set, and needed to add more space. This is quite easy once you know the steps. There are three parts

  1. Find some space
  2. Format it
  3. Make it available to SMS

Find some space

If you have real disks, you need to get some space created for your volume. Talk to your storage administrator.

z/D&T

I was running on zD&T, running on Linux, so my disks are emulated.

I followed the documentation and issued:

alcckd USER02 -d3390-3

This create a file called USER02 with size of a 3390-3. The file can have any valid name, USER02.33903 would also be valid, it has no connection to the z/OS volume ID.

You need to mount it on z/OS. Look in your devmap for an entry like

device 0Ac5 3390 3390       # Available for dynamic mounts
device 0Ac6 3390 3390 # Available for dynamic mounts

and mount it

awsmount 0ac5 -m USER02

Where 0ac5 is a free device in the device map and USER02 is my filename.

Within my devmap I had

device 0A9E 3390 3390 /home/zPDT/C4USS2
...
device 0AC5 3390 3390       # Available for dynamic mounts

Device 0ac5 does not have a file specified, so can be used for dynamic mounts.

For future IPLs, I added my USER02 to my device map.

Format it

I had to vary on line from a z/OS perspective

v 0ac5,online

This gave me message

v 0ac5,online
IEF503I UNIT 0Ac5 I/O ERROR

I formatted the volume using ICKDSF: (check the unit very carefully)

//IBMUSERO  JOB 1,COLIN,MSGCLASS=X,REGION=40M 
// EXEC PGM=ICKDSF,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD * 
 INIT UNIT(AC5) NOVERIFY VOLID(USER02) VTOC(0,1,14) 
/* 

When running this JCL a WTOR came up asking if I was sure I wanted to format it.

Once this has completed, vary the device online

V AC5,ONLINE

You can specify STORAGEGROUP to define which SMS storage group it will be used in.

ISMF gave me some JCL(which I didn’t use) which seems to do the same thing. I used this to reformat the volume

//IBMUSERA JOB  (ACCOUNT),'NAME'                                      
//STEP1  EXEC PGM=ICKDSF,REGION=0M 
//SYSPRINT  DD  SYSOUT=* 
//INVOL1   DD    VOL=(PRIVATE,SER=USER02),UNIT=3390,DISP=SHR 
//SYSIN    DD    * 
 INIT - 
      DDNAME(INVOL1) - 
      VERIFY(USER02) - 
      NOCHECK - 
      CONTINUE - 
      MAP - 
      NOVALIDATE - 
      NOPURGE - 
      NOBOOTSTRAP                                                                       
/* 

Make this an SMS volume

I used

//IBMUSERB JOB  (ACCOUNT),'NAME' 
//STEP1  EXEC  PGM=ADRDSSU,REGION=0M 
//SYSPRINT  DD  SYSOUT=* 
//INVOL1   DD    VOL=SER=USER02,UNIT=3390,DISP=SHR 
//SYSIN    DD    * 
 CONVERTV - 
      SMS - 
      DDNAME(INVOL1) 
/* 

The CONVERTV command is used to convert existing volumes to and from SMS management without data movement.

Using ISMF to display volumes, it said the USER* volumes were Storage Group SGBASE.

Passing parameters to Java programs

There is a quote by Alexander Pope

A little learning is a dangerous thing.
Drink deep, or taste not the Pierian Spring;
There shallow draughts intoxicate the brain,
and drinking largely sobers us again.”

I was having problems passing parameters to Java. I had a little knowledge about Java web servers on z/OS, so I was confident I knew how to configure a web browser; unfortunately I did not have enough knowledge.

I was trying to get keyrings working with a web server on z/OS, but what I specified was not being picked up. In this post I’ll focus on keyrings, but it applies to other parameters are well.

Setting the default values

You can give Java a default keyring name

-Djavax.net.ssl.keyStore=safkeyring://START1/MQRING

Which says, if the user does not specify a keystore then use this value. If the user does not create a keystore, then a default keystore is created using these parameters.

Specify the value yourself

You can create your own keystore, passing keystore name and keystore type. You could use you own option, such as -DCOLINKEYRINGNAME=safkeyring://START1/MQRING and use that when configuring the keystore to Java. In this case the -Djavax.net.ssl.keyStore=safkeyring://START1/MQRING is not used. This is the same for other parameters.

Web server configuation

The Apache web server, and IBM Liberty web servers, are configured using server.xml files, and the web server creates the keystores etc using the information in the file.

For example a web server may have content like

<Connector port="${port.http}" protocol="${http.protocol}" 
     SSLEnabled="true" 
     maxThreads="550" scheme="https" secure="true" 
     clientAuth="false" sslProtocol="TLS" 
     keystoreFile="${keystoreFile}" 
     keystorePass="${keystorePass}" 
     keystoreType="${keystoreType}" 
     sslEnabledProtocols="${ssl.enabled.protocols}" /> 

where the ${…} is an environment variable. ${keystoreFile} would use the environment variable keystoreFile or option -DkeystoreFile=”safkeyring//START1/MYRING”

You can have multiple <Connector..> each with different parameters, so using -Djavax.net.ssl.keyStore etc would not work.

You need to know which options to use, because setting the Java defaults using -Djavax.net.ssl.keyStore… may be overridden.

You also need to know which server.xml file to use! I was getting frustrated when the changes I made to the server.xml was not the server.xml being used!

Follow the instructions to install Java on z/OS and screw up production.

I downloaded an update to Java on z/OS, and noticed that the instructions would cause problems!

Java is in a directory like

/usr/lpp/java/J7.1        
/usr/lpp/java/J7.1_64     
/usr/lpp/java/J8.0        
/usr/lpp/java/J8.0_64     

I have Java V7 and V8 and 31 and 64 versions

The installation instruction say

cd to /usr/lpp/java and unpax the uploaded SDK8_64bit_SR8_FP6.PAX.Z file.

This will unpack the file into

/usr/lpp/java/J8.0_64

Overwriting your production system with the new version, so it goes live without any testing, and for a few minutes you have a mixture of files from different fixpacks!

What I would suggest is

  • Unzip the file gzip -l -d ibm-semeru-open-jre_x64_linux_11.0.20_8_openj9-0.40.0.tar.gz this upacks the file into a .tar – and delete the .gz
  • FTP it to z/OS in binary
  • Create a ZFS for the new Java.
  • mkdir  /usr/lpp/java/new
  • mount the ZFS in /usr/lpp/java/new
  • cd /usr/lpp/java/new
  • pax -ppx -rvzf SDK8_64bit_SR8_FP6.PAX.Z

This will create files under

/usr/lpp/java/new/J8.0_64

Test this by setting

export JAVA_HOME="/usr/lpp/java/new/J8.0_64"

and using it.

When you have finished your testing, you can mount the new ZFS on /usr/lpp/java/J8.0_64 .

You may want to do this as part of an IPL, because an unmount of a ZFS will terminate any thing using the file system – such as z/OSMF.

You could set up /usr/lpp/java/old/J8.0_64 and /usr/lpp/java/new/J8.0_64 and use /usr/lpp/java/J8.0_64 as an alias to one or the other.

How to debug a bash script when the easy way does not work.

I was trying to debug a program product, and to specify a Java override. The configuration used bash scripts. My problem was which script should I put my fix in?

How to debug bash scripts

From searching in the internet, the way of debugging a simple self contained bash scripts is either

  • edit the file and add the -x as in #!/bin/sh -x this give output for just this file.
  • or use sh -x … command line option to enable trace

The sh -x … command applies to the specified program name. You cannot enable the trace this way for any programs it calls.

There is no way of saying trace this command …and any others programs it calls.

Editing the script may not easy

Editing the file was a challenge, as it was on a read only file system. I had to unmount it, and mount it read/write. Easy on my one person z/OS ; not so easy on a typical z/OS image with many users. You do not want to make a change and break someone else.

I used the Unix command

df -P /usr/lpp/IBM/... 

on the file to find which file system it was on, then used the TSO commands

unmount filesystem('ABC100.ZFS') Immediate
mount filesystem('ABC100.ZFS') mode(RDWR)                   
  type(HFS) mountpoint('/usr/lpp/IBM/abcdef')

Or change the BPX* parmlib member and re-ipl which seems overkill.

I did not really want to have to edit the production scripts – but this was only at the server startup. Only one instance was affected.

The chicken and the egg problem

How do I know which file to edit to add the “-x”? After lots of investigation I found one file deep down, but I could not see who called it.

I used two commands

ps -o pid,ppid,args -p $PPID   
ps -o pid,ppid,args -u $LOGNAME 

The first says give me information about the threads parent PID.

The second says gave information about threads running with the userid of the thread

The first gave

      PID       PPID COMMAND                                                                         
 33620145   67174575 /bin/sh -c /Z24C/usr/lpp/IBM/..../bin/envvars.sh -F/parameter  

Which shows the thread was called from the envvars.sh script with the parameter -F/parameter. This was a start, but who called envvars.sh?

The second command gave

         PID       PPID COMMAND  
(1) 83951745          1 BPXBATCH    
(2) 83951791   83951745 /bin/sh /usr/lpp/IBM/.../start.sh                                     
(7) 50397360   33619999 oedit /usr/lpp/IBM/.../bin/config.final.env                           
(3) 83951836   s83951791 /bin/sh /Z24C/usr/lpp/IBM/.../bin/abcmain.sh run -config /Z24C       
(5) 67174622   33620191 /bin/sh -x /Z24C/usr/lpp/IBM/.../bin/envvars.sh -F/tmp/RSEAPI_e       
(4) 33620191   83951836 /bin/sh -c /Z24C/usr/lpp/IBM/.../bin/envvars.sh -F/tmp/RSEAPI_e       
(6) 83951881   67174622 ps -o pid,ppid,args -u STCRSE 

You have to start at the top of the process tree, and find the children of each process. The parent and child have the same colour process ids above.

  1. I know the script was started from BPXBATCH. This has a process id of 83951745 (first column of data) in red.
  2. The process with this process id as a parent (PPID) (second column of data) in red. is /bin/sh /usr/lpp/IBM/…/start.sh This is what BPXBATCH executes. If you have a system with lots of BPXBATCH instances running, you can locate the command to find which thread is of interest. This process has a process id of 83951791
  3. This process invokes /bin/sh /Z24C/usr/lpp/IBM/…/bin/abcmain.sh run -config /Z24C with a PID of 83951836.
  4. This process invokes /bin/sh -c /Z24C/usr/lpp/IBM/…/bin/envvars.sh -F/tmp/RSEAPI_e with a PID of 33620191
  5. This invokes /bin/sh -x /Z24C/usr/lpp/IBM/…/bin/envvars.sh -F/tmp/RSEAPI_e (again!)
  6. This issues the command ps -o pid,ppid,args -u STCRSE which displays the thread information. We have got to the end.
  7. Was from me editing a file in an Unix Services session, so not relevant to the investigation

Who am I ?

You can use

echo "path to me ->  ${0}     "

which gave me

path to me -> /home/colinpaice/Downloads/test.sh

What would I do to make it easier to debug?

In each shell script have code like

#if the symbol myprog_text exists
if [ -z ${myprog_test+x} ]; 
then 
   # echo "myprog_test is not set"; 
else
  # echo "specified "${myprog_test}
  # if it has a value > 0 then write the path
  if [ $myprog_test -gt 0 ]
  then 
     echo "# path to me --------------->  ${0}     "
  fi
  #if the value is 2 or more then start the trace
  if [ $myprog_test -gt 1 ]
  then 
     set -x 
  fi
fi

This checks to see if global variable myprog_test is set, if it set to a 1 or larger, it displays the path name, if it is set to 2 or larger it turns trace on, using set -x.

With this each script checks a variable product_myscript (where myscript is the name of the script), and takes the appropriate action.

To turn the traces you list all of the scripts in the directory, then use

export myprod_myscript=1

for each script (of interest). This will then give you a trace of which scripts were invoked. You can then set

Some scripts use a variable $SHLVL which give you the call depth. This would be useful,but his is not supported in shell in z/OS.

Is it as easy(!) as this?

No quite. Many services are started as a started task, where the JCL is like

//ABCAPI   EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT, 
//            PARM='PGM &HOME./tomcat.base/start.sh' 

You can put your global variables in STDENV. Note: this is a list of variables, not a shell script, so you cannot do

xxx=1
yyy=$xxx

Why doesn’t ctrl-s work in ISPF edit? – ah it does now.

I had been editing a file, saving it, and finding the changes were not being picked up. Looking back, it was obvious; I was using CTRL-S the familiar Linux command, instead of F3 on ISPF.

I fixed this by configuring X3270 (on Linux).

My file /home/colin/.x3270pro now has

...
x3270.keymap: mine
! Definition of the 'mine' keymap
x3270.keymap.mine: #override \
    Alt<Key>4:          String("\\x00a2")\n\
    Ctrl<Key>backslash: String("\\x00a2")\n\
    <Key>Escape:    Clear()\n\
    <Key>End:        FieldEnd()\n\
    Ctrl<Key>Delete:   EraseEOF()\n\
    Ctrl<Key>Right:    NextWord()\n\
    Ctrl<Key>Left:    PreviousWord()\n\
    Ctrl<Key>Up:    Home()\n\
    <Key>Control_L: Reset()\n\
    <Key>Control_R: Reset()\n\
    <Key>Prior: PF(7)\n\
    <Key>Next: PF(8)\n\
    <Btn3Down>:   PA(1)\n\
    Ctrl<Key>1:   PA(1)\n\
    Ctrl<Key>s:   MoveCursor(3,15) String("save") Enter()\n\

When I started a new X3270 session, Ctrl -S went to the command line, typed save and pressed enter. Job Done ! The numbers are 0 based, so 3 means line 4 on the screen.

This makes life so much easier!

Parsing command line values

I wanted to pass multiple parameters to a z/OS batch program and parse the data. There are several different ways of doing it – what is the best way ?

This question is complicated by

Checking options

Processing command line options can mean a two stage process. Reading the command line, and then checking to ensure a valid combination of options have been specified.

If you have an option -debug with a value in range 0 to 3. You can either check the range as the option is processed, or have a separate section of checks once all the parameters have been passed. If there is no order requirement on the parameters you need to have separate code to check the parameters. If you can require order to the parameters, you might be able to have code “if -b is specified, then check -a has already been specified

I usually prefer a separate section of code at it makes the code clearer.

Command styles

On z/OS there are two styles of commands

def xx(abc) parm1(value) xyz

or the Unix way

-def -xx abc -parm1 -1 -a –value value1 -xyz.

Where you can have

  • short options “-a” and “-1”
  • long option with two “-“, as in “–value”,
  • “option value” as is “-xx abc”
  • “option and concatenated value” as in “-xyz”; option -x, value yz

I was interested in the “Unix way”.

  • One Unix way is to have single character option names like -a -A -B -0. This is easy to program – but it means the end user needs to lookup the option name every time as the options are not usually memorable.
  • Other platforms (but not z/OS) have parsing support for long names like – -userid value.
  • You can parse a string like ro,rw,name=value, where you have keyword=value using getsubopt.
  • I wrote a simple parser, and a table driven parser for when I had many options.

Defining the parameter string toJCL.

The traditional way of defining a parameter string in batch is EXEC PGM=MYPROG,PARM=’….’ but the parameter is limited in length.

I tend to use

// SET P1=COLIN.PKIICSF.C 
// SET P2="optional"
//S1 EXEC PGM=MYPROG,PARM='parms &P1 &P2'  

You can get round the parameter length limitation using

//ISTEST   EXEC PGM=CGEN,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD * 
/ 
 -detail 0 
 -debug 0 
 -log "COLINZZZ" 
 -cert d

Where the ‘/’ on its own delimits the C run time options from my program’s options.

The values are start in column 2 of the data. If it starts in column 1, the value is concatenated to the value in the previous line.

You can use JCL and System symbols

// EXPORT SYMLIST=(*) 
// SET LOG='LOG LOG' 
//ISTEST   EXEC PGM=CGEN,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD *,SYMBOLS=EXECSYS
/ 
 -log "COLINZZZ" 
 -log "&log"
 ...

This produced -log COLINZZZ -log “LOG LOG”

Parsing the data

C main programs have two parameters, a count of the number of parameter, and an array of null terminated strings.

You can process these

int main( int argc, char *argv??(??)) 
{ 
  int iArg; 
  for (iArg = 1;iArg< argc; iArg ++   ) 
  { 
    printf(".%s.\n",argv[iArg]); 
  } 
  return 0; 
} 

Running this job

//CPARMS   EXEC  CCPROC,PROG=PARMS 
//ISTEST   EXEC PGM=PARMS,REGION=0M,PARMDD=MYPARMS 
//MYPARMS DD * 
/ 
 -debug 0 
 -log "COLIN  ZZZ" 
 -cert 
 -ae colin@gmail.com 

gave

.-debug.                   
.0.                        
.-log.                     
.COLIN  ZZZ.               
.-cert.                    
.-ae.                      
.colin@gmail.com.          

and we can see the string “COLIN ZZZ” in double quotes was passed in as a single string.

Parsing with single character options

C has a routine getopt, for processing single character options like -a… and -1… (but not -name) for example

while ((opt = getopt(argc, argv, "ab:c:")) != -1) 
   { 
       switch (opt) { 
       case 'a': 
           printf("-a received\n"); 
           break; 
       case 'b': 
           printf("-b received \n"); 
           printf("optarg %d\n",optarg); 
           if (optarg) 
             printf("-b received value %s\n",optarg); 
           else 
             printf("-b optarg is0       \n"); 
           break; 
       case 'c': 
           printf("-c received\n"); 
           printf("optarg %d\n",optarg); 
           if (optarg) 
             printf("-c received value %s\n",optarg); 
           else 
             printf("-c optarg is0       \n"); 
           break; 
       default: /* '?' */ 
           printf("Unknown n"); 
     } 
   } 

The string “ab:c:” tells the getopt function that

  • -a is expected with no option
  • -b “:” says an option is expected
  • -c “:” says an option is expected

I could only get this running in a Unix environment or in a BPXBATCH job. In batch, I did not get the values after the option.

When I used

//BPX EXEC PGM=BPXBATCH,REGION=0M,
// PARM='PGM /u/tmp/zos/parm.so -a -b 1 -cc1 '

the output included

-b received value b1
-c received value c1

This shows that “-b v1” and “-cc1” are both acceptable forms of input.

Other platforms have a getopt_long function where you can pass in long names such as –value abc.

getsubopt to parse keyword=value

You can use getsubopt to process an argument string like “ro,rw,name=colinpaice”.

If you had an argument like “ro, rw, name=colinpaice” this is three strings and you would have to use getsubopt on each string!

You have code like

int main( int argc, char *argv??(??)) 
{ 
 enum { 
       RO_OPT = 0, 
       RW_OPT, 
       NAME_OPT 
   }; 
   char *const token[] = { 
       [RO_OPT]   = "ro", 
       [RW_OPT]   = "rw", 
       [NAME_OPT] = "name", 
       NULL 
   }; 
   char *subopts; 
   char *value; 

   subopts = argv[1]; 
 while (*subopts != '\0' && !errfnd) { 
   switch (getsubopt(&subopts, token, &value)) { 
     case RO_OPT: 
       printf("RO_OPT specified \n"); 
       break; 
     case RW_OPT: 
       printf("RW_OPT specified \n"); 
       break; 
     case NAME_OPT: 
       if (value == NULL) { 
          printf("Missing value for " 
                 "suboption '%s'\n", token[NAME_OPT]); 
           continue; 
       } 
       else 
         printf("NAME_OPT value:%s\n",value);
         break; 
    default: 
         printf("Option not found %s\n",value); 
         break; 
     }  // switch 
   } // while 
 }  

Within this is code

  • enum.. this defines constants RO_OPT = 0 RW_OP = 1 etc
  • char const * token defines a mapping from keywords “ro”,”rw” etc to the constants defined above
  • getsubopt(&subopts, token, &value) processes the string, passes the mapping, and the field to receive the value

This works, but was not trivial to program

It did not support name=”colin paice” with an imbedded blank in it.

My basic command line parser(101)

I have code

for (iArg = 1;iArg< argc; iArg ++   ) 
{ 
  // -cert is a keyword with no value it is present or not
  if (strcmp(argv[iArg],"-cert") == 0) 
  { 
    function_code = GENCERT    ; 
    continue; 
  } 
  else 
  //  debug needs an option
  if (strcmp(argv[iArg],"-debug") == 0 
      && iArg +1 < argc) // we have a value 
      { 
        iArg  ++; 
        debug = atoi(argv[iArg]); 
        continue; 
      } 
  else 
  ...
  else 
    printf("Unknown parameter or problem near parameter %s\n", 
           argv[iArg]);
  }   // for outer - parameters 

This logic processes keywords with no parameters such as -cert, and keyword which have a value such as -debug.

The code if (strcmp(argv[iArg],”-debug”) == 0 && iArg +1 < argc) checks to see if the keyword has been specified, and that there is a parameter following it (that is, we have not run off the end of the parameters).

Advanced – table – ize it

For a program with a large number of parameters I used a different approach. I created a table with option name, and pointer to the fields variable.

For example

getStr lookUpStr[] = { 
    {"-debug", &debug     }, 
    {"-type",  &type       }, 
    {(char *) -1,  0} 
   }; 

You then check each parameter against the list. To add a new option – you just update the table, with the new option, and the variable.

int main( int argc, char *argv??(??)) 
{ 
   char * debug = "Not specified"; 
   char * type   = "Not specified"; 
   typedef struct getStr 
   { 
      char * name; 
      char ** value; 
   } getStr; 
   getStr lookUpStr[] = { 
       {"-debug", &debug     }, 
       {"-type",  &type       }, 
       {(char *) -1,  0} 
      }; 
  int iArg; 
  for (iArg = 1;iArg< argc; iArg ++   ) 
  { 
   int found = 0; 
   getStr * pGetStr =&lookUpStr[0];
   // iterate over the options with string values
   for (; pGetStr -> name != (char *)  -1; pGetStr ++) 
   { 
     // look for the arguement in the table
     if (strcmp(pGetStr ->name, argv[iArg]) == 0) 
     { 
       found = 1; 
       iArg ++; 
       if (iArg < argc) // if there are enough parameters
                        // so save the pointer to the data
        *( pGetStr -> value)= argv[iArg] ; 
       else 
         printf("Missing value for %s\n", argv[iArg]);       
       break;  // skip the rest of the table
     }  // if (strcmp(pGetStr ->name, argv[iArg]) == 0) 
     if (found > 0) break; 
    } // for (; pGetStr -> name != (char *)  -1; pGetStr ++) 
   
   if (found == 0) 
   // iterate over the options with int values 
   ....
  } 
  printf("Debug %s\n",debug); 
  printf("Type  %s\n",type ); 
  return 0; 
}   

This can be extended so you have

getStr lookUpStr[] = { 
    {"-debug", &debug, "char" }, 
    {"-type",  &type ,"int"       }, 
    {(char *) -1,  0, 0} 
   }; 

and have logic like

if (strcmp(pGetStr ->name, argv[iArg]) == 0) 
     { 
       found = 1; 
       iArg ++; 
       if (iArg < argc) // if there are enough parmameters
       {
       if ((strcmp(pGetStr -> type, "char") == 0 
        *( pGetStr -> value)= argv[iArg] ; 
       else 
        if ((strcmp(pGetStr -> type, "int ") == 0 )
        *( pGetStr -> value)= atoi(argv[iArg]) ;
      ...   
     }

You can go further and have a function pointer

getStr lookUpStr[] = { 
    {"-debug", &debug,myint }, 
    {"-loop", &loop  ,myint },  
    {"-type",  &type , mystring  }, 
    {"-type",  &type , myspecial  }, 
    {(char *) -1,  0, 0} 
   };f

and you have a little function for each option. The function “myspecial(argv[iarg])” looked up values {“approved”, “rejected”…} etc and returned a number representation of the data.

This takes a bit more work to set up, but over all is cleaner and clearer.