C includes ” is not the same as <

The one line description: I found the difference between #include <a.h> and #include “a.h”.

I had problems finding compiling some sample C code which included

#include <__iew_api.h>

I was getting 

WARNING CCN3296  #include file <__iew_api.h> not found.

There were a couple of problem. The length of the name __iew_api is more than 8 characters, so will not be in a PDS or PDSE. I found it in /usr/include.

In my JCL for comping C programs I had 

LSEARCH(/usr/include/)

But still it was not found. After a cup of tea and walk down to the shops I wondered if it was the include that was causing the problems.

I changed it to

#include "__iew_api.h"

and it found it. The documentation says

The file_path can be an absolute or relative path. If the double quotation marks are used, … the preprocessor adds the directory of the including file to the list of paths to be searched for the included file.
If the double angle brackets are used, … the preprocessor does not add the directory of the including file to the list of paths to be searched for the included file.

Using Rexx under z/OS Unix Services to display thread information

I wanted to display information about threads running in a big Java application, and see what was executing during set up.

Rexx can provide information on threads, and so I was able to create a script which provided process thread data. There was documentation, but it was not very clear, so I’m documenting what I learned

My program

/* rexx */ 
do k = 1 to 20 
  call procinfo 
  say "jobname  asid    ppid    pid    threadid  tcb    cmdline" 
  do i=1 to bpxw_pid.0 
    x =procinfo(bpxw_pid.i,'process') 
    if x = '' then iterate 
    if bpxw_LOGNAME.i <> "ZWESVUSR" then iterate 
    y = procinfo(bpxw_pid.i,'thread') 
    if y = '' then iterate 
    do j=1 to bpxw_threads 
      xtcb = d2x( bpxw_TCB.j) 
      say bpxw_JOBNAME  d2x(bpxw_ASID)  right(bpxw_PPID,8), 
          right(bpxw_PID,8)  bpxw_THREAD_ID.j xtcb  bpxw_CMDLINE 
    end 
  end 
  sleep(1) 
end
  • call procinfo returns a list of process ids in a stem bpxw_pid. and userids in a stem bpxw_LOGNAME
  • do i=1 to bpxw_pid.0 iterate through the list of the processes
  • x =procinfo(bpxw_pid.i,’process’) get the process information about the process id in bpxw_pid.i.
  • if x = ” then iterate if the thread no longer exists – do the next one
  • if bpxw_LOGNAME.i <> “ZWESVUSR” then iterate ignore threads from other userids
  • y = procinfo(bpxw_pid.i,’thread’) get the thread information for the process id in bpxw_pid.i.
  • if y = ” then iterate if no data is returned skip this
  • do j=1 to bpxw_threads for each thread in the process …
  • xtcb = d2x( bpxw_TCB.j) convert the thread TCB from decimal to hex.
  • say …
    • bpxw_JOBNAME this is from the process information
    • d2x(bpxw_ASID) display the ASID of the process in hex
    • bpxw_THREAD_ID.j the thread id.

Java persistent shared classes cache on z/OS

With Java shared classes cache, by default on z/OS saves the data in shared memory. You can use the snapshot command to save a copy on disk, and use the restore command after IPL to recreate it. For my zPDT system running z/OS on a Linux server this many seconds of start up time.

In more recent Java versions, the Shared Classes Cache has supported the persistent option, where shared virtual storage is mapped to a file – and so updating memory, updates the file.

I had a few problems getting this to work, and there was no documentation on the use of the persistent option.

When I enabled it, for example with 

 -Xshareclasses:name=zoweGW,cacheDirPerm=0777,cacheDir=/u/tmp/zowec/,persistent"

I got

JVMSHRC245E Error mapping shared class cache file 
JVMSHRC336E Port layer error code = -155 
JVMSHRC337E Platform error message: EDC5132I Not enough memory. 
JVMSHRC840E Failed to start up the shared cache. 
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed 
Error: Could not create the Java Virtual Machine. 
Error: A fatal exception has occurred. Program will exit.

I had to change the SMFLIMxx parmlib member to fix this

Display the current SMFLIM configuration

You can display your current SMFLIMxx configuration using

d smflim
d smflim,r

The d smflim,r gave me

...
Member and rule number SMFLIMCP 0003                   
User:                                                  
 ZWESVUSR                                              
Attributes:                                            
 EXECUTE: NOCHANGE                                     
 JOBMSG: ISSUE                                         
 MAXSHARE: 9000000                                     
...

Which shows the rule for user ZWEVSUR came from the third rule in SMFLIMCP. It sets MAXSHARE, and other parameters.

Update the member

I updated my SMFLIMCP member to be

REGION USER(COLIN) JOBMSG(ISSUE) MAXSHARE(90000)

activated it using t SMFLIM=(CP,C2) where CP2,C2 is my list of SMFLIM members. Note: The T SMFLIM command, replaces all of the definitions with what is in the list, so you need to specify the whole list, not just the changed member.

The definitions become active immediately, you do not need to logoff and logon, or resubmit a job.

When the Java job had started, it created a file C290M21F1A64P_hw_G43L00 in the specified directory.

When persistent was not specified, files were stored in the javasharedresources subdirectory.

Should I use this persistent option?

You have the choice of using the persistent option in the -Xshareclasses…persistent parameters, or not to specify it. If you do not use the persistent option you need to save the shared memory across IPLs, by using -Xshareclasses:…,snapshotCache and restoring it after an IPL using -Xshareclasses:…restoreFromSnapshot. I used this method, and added a steps to my started tasks, one to restore (if the cache exists already, it does nothing), and one at the end, to save it.

How does the performance compare?

On my zPDT system which is not meant to be used for performance evaluations, they both had similar durations, and used similar amounts of CPU, though non persistent was usually slight better.

Funny…

I also go message

JVMSHRC561E Failed to initialize the shared classes cache, there is not enough space in the file system. Available free disk space bytes = 516144128, requested bytes = 536870912.

Which was a surprise as I thought I had enough free disk space.

Creating and running a jar file

You can start a Java program by specifying the class name and the class path to use (containing .class and/or .jar files), or you can specify just a self contained .jar file, which contains all it needs. I wont describe Java Modules ( Java Platform Module System, .jmod -> stronger encapsulation, improved dependency management, better code organization, enhanced security, streamlined JRE, and potentially faster application startup times), or a Java Run Time image (.jrt, which includes only the necessary modules required for a specific application, making it smaller, safer, and capable of starting faster than a full JRE).

Create class files

The documentation for Java says you have to create a manifest so Java knows which classes to use.

I have a simple program hw.java

class HelloWorld { 
    public static void main(String[] args) { 
        System.out.println("Hello, World!"); 
    } 
}

I compiled it with

/usr/lpp/java/J21.0_64/bin/javac hw.java

This creates .class files for each class in the original source.

  • HelloWorld.class

It will start the function “main” within the class.

Create a jar file and specify the entry point

Java needs to know the entry point class of the jar file. The code below creates the jar file, specifying the entry point class name ( -e ..) and includes all of the classes specified (just HelloWorld.class).

The next line runs the jar file.

/usr/lpp/java/J21.0_64/bin/jar -c -f hw.jar -e HelloWorld HelloWorld.class
/usr/lpp/java/J21.0_64/bin/java -jar hw.jar

Under the covers it creates a manifest file META-INF/MANIFEST.MF . It is not in the default code page, so to look at it, I expanded the jar file, used oedit META-INF/. , then used the line command ea to edit it in ASCII.

Manifest-Version: 1.0 
Created-By: 21.0.4 (IBM Corporation) 
Main-Class: HelloWorld

Create a jar file and specify a manifest.

Java needs to know the entry point class name of the jar file. You can specify it in the manifest. The code below creates the jar file, using the specified manifest file, and includes all of the classes specified (just HelloWorld.class). The manifest is the one in the previous section.

The next line runs the jar file

/usr/lpp/java/J21.0_64/bin/jar -c -f hw.jar-m META*/MAN*.MF HelloWorld.class 
/usr/lpp/java/J8.0_64/J21.0_64/bin/java -jar  hw.jar

When you create a jar file and do not specify a manifest or entry point

The code below creates the jar file, using the classes specified (just HelloWorld.class), but does not include a manifest.

If you try running the jar file it complains about no manifest.

You can run the jar file in the class path (-cp) and specify the entry class as a parameter

/usr/lpp/java/J21.0_64/bin/jar -c -f hw.jar HelloWorld.class

COLIN:/u/tmp/java: >/usr/lpp/java/J21.0_64/bin/java -jar  hw.jar     
no main manifest attribute, in hw.jar 

/usr/lpp/java/J21.0_64/bin/java -cp hw.jar   HelloWorld

It’s not quite that simple…

I was looking at some Jar files, and the manifest does not have a Main-Class entry, and there is no entry point specify on the java command – so there is clearly another way of specifying the entry point.

Programming shared memory – more head banging.

I was trying to use shared memory (to look at Java Shared Classes), and it took me a day to get it working – better documentation would have helped.

I had two basic problems

  1. Using smctl to display information about the shared memory, gave the size as 0 bytes, even though the ipcs command showed me there were megabytes of data in the shared memory area.
  2. Trying to attach the shared memory gave me “invalid parameters” return code – even though the documented reasons for this error code did not apply to my program.

I tried many things, from using different userids, to running with a different storage key, running APF authorised….

I eventually got it to work by compiling my C program in 64 bit mode rather than 31 bit mode. There is no discussion about 31 bit/64 bit in the documentation. If the shared memory in 64 bit mode, you will need 64 bit addressability, so you need a 64 bit program. But there is no way of determining that the shared memory is 64 bit!

My basic program

{ 
//struct shmid_ds  buf; 
  struct shmid_ds64 buf; 
  memset(&buf   ,0,sizeof(buf)); 
  int shmid    =  8197; 
  int rc = 0; 
  long l; 
  int cmd = IPC_STAT; 
  char * fn = "COLIN"; 
  int shmflg =0; 
    shmflg =   IPC_STAT; 
//  rc =shmctl(shmid, cmd,        &buf); 
    rc =shmctl64(shmid, cmd,        &buf); 
    perror("shmctl      "    ); 
    printf("shctl rc %i\n",rc); 
    l = buf.shm_segsz;
    printf("size     %ld\n",l); 
    printHex(stdout,&buf,sizeof(buf)); 
    /////////////////////////////////////////////// 
    //  shmat 
    /////////////////////////////////////////////// 
    char * pData = NULL; 
    pData = shmat(shmid, NULL , 0          ); 
    printf("Address %ld\n",pData); 
    printHex(stdout,pData+4096*1024,1024*1024); 
    int e = errno; 
    perror("shmat "); 
    printf("Errno: %s\n",strerror(e)); 
   return 0; 
}

Originally I was using EDCCB to compile and bind this.

The EINVAL error return code was (from the documentation) for cases where the pointer in shmat was non NULL. I was passing NULL – so none of this made sense.

The reason code 0717014A was

JRInvalidAmode: An incorrect access mode was specified on the access service
Action: The access mode specified on the access service has unsupported bits turned on. Reissue the request and specify a valid access mode.

It turned out that my program was 31 bit. When I used 64 bit – it magically started working.

I compiled it with EDCQCB, and had to change a few things to be 64 bit mode.

  • shmid_ds buf -> shmid_ds64
  • shmctl -> shmctl64

When I ran it in 31 bit mode, the length of the storage returned was 0. In 64 bit mode, it gave the correct length. This looks like a way of telling what mode the shared memory is!

Notes on shared storage

Shared storage is a Unix concept, which works on Unix on z/OS.

It allows you to share data between threads, and between different address spaces. Java Shared Classes exploit this. Java classes can be stored in the shared storage, and “Ahead Of Time” compilation can be stored in the shared cache. Next time a JVM starts, it can look in the shared cache, and use the already compiled class.

Each shared storage segment has a key which you use in the various shared memory API calls.

There are no real restrictions, in some of the documentation on the web, it suggests using 1234 as a key! This has its limitations. A better way, used by Java is based on the inode of a file

You can display information about InterProcess Communication Status (ipcs) using the ipcs command.

You can display shared memory, and shared semaphores.

ipcs command lists… 

Shared Memory:                                                           
T         ID     KEY        MODE       OWNER    GROUP                    
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                    
m      73733 0x6137142d --rw------- OMVSKERN     SYS1                    
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                    
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                    
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1

The Java shared class files on my system are

767 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_semaphore_zoweGW_G43L00             
768 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_memory_zoweGW_G43L00

Inode 767 is 0x2ff, Inode 768 is 0x300.

0x61… inode…. 2d

061 because created with t = ftok(fn,idi );

Shared Memory:                                                          
T         ID     KEY        MODE       OWNER    GROUP                   
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                   
m     139269 0x6137142d --rw------- OMVSKERN     SYS1                   
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                   
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                   
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1                   
Semaphores:                                                             
T         ID     KEY        MODE       OWNER    GROUP                   
s     135172 0x8137132d --ra------- OMVSKERN     SYS1                   
s      69637 0x8137e72f --ra------- ZWESVUSR ZWEADMIN                   
s     135174 0x8100400f --ra------- ZWESVUSR ZWEADMIN                   
s      69639 0x8c10840f --ra-ra-ra- ZWESVUSR ZWEADMIN                   
s     135178 0x810e9f0f --ra------- OMVSKERN     SYS1                   
s      69643 0x8c108a0f --ra-ra-ra- ZWESVUSR ZWEADMIN

More details

You can use

ipcs -m -a

to display more information including size, and number of users.

Using Java internal trace

I was trying to understand a class load problem, and needed to use Java internal virtual machine trace to see what was happening.

You specify -Xtrace options at start up. Even with limiting options, I had over 2 millions lines of trace output – and the Java application had not fully started!

There are lots of components you can specify see here. I used

 -Xtrace:maximal={j9jcl,j9shr,j9prt,omrport},output=/u/tmp/java/trace.bin

where

  • j9jcl is Java class libraries
  • j9shr is shared classes libraries
  • j9prt is VM port libraries… the platform specific (ported) code
  • omrport is the OMR ported code. (OMR is a set of common components, such as threads and garbage collection)

I haven’t used it,but you can say do something when this trace point is triggered, stop (suspend) tracing.

-Xtrace:trigger=tpnid{abc.123,suspend}

In the example trace output

07:49:24.699384689 0x000000002167f200 omrport.657 Event J9 VM NLS message: Failed to find class org/springframework/boot/loader/launch/PropertiesLauncher in shared cache for class-loader id 0.

where the trace point is omrport.657

You can put the definitions in a file -Xtrace:properties=<filename>

I have a trace file – now what?

You can format the trace file on z/OS, but I downloaded the file (in binary) to my Linux machine, because a) I have more disk space, 2) the java formatting is faster, 3) I have better tools to look at the data. You need the save level of Java (or better) to format all the fields. If there is a mismatch, some fields will not be formatted. The traceformat command is described here.

Format the data

/home/colinpaice/Downloads/jdk-21.0.6+7/bin/traceformat -verbose trace.bin trace.txt

It produced

Writing formatted trace output to file trace.txt
Processing 245.45312Mb of binary trace data
Processed 10.0Mb (4%), burst speed: 1.9344624Mb/s, average: 1.9344624Mb/s
Processed 20.0Mb (8%), burst speed: 2.4675941Mb/s, average: 2.1687446Mb/s
Processed 30.0Mb (12%), burst speed: 3.4868424Mb/s, average: 2.4814205Mb/s
...

I wrote the blog post Finding the needle in a field of hay, to help extract the records of interest.

On the traceformat command you can specify

  • -indent: Indents trace messages at each Entry trace point and outdents trace messages at each Exit trace point. The default is not to indent the messages.
  • -threads: Filters the output for the given thread IDs only. thread id is the ID of the thread, which can be specified in decimal or hex (0x) format. Any number of thread IDs can be specified, separated by commas.

Looking at the data

I used the commands grep -n …., tail -n +999, head -n 500, less as described here. But for detailed analysis I piped the output into a file, and used an editor on the output. This meant the lines did not wrap, I could annotate (and save) the trace, and highlight interesting records.

Because some of the lines are very long, I split interesting lines so all of the data was displayed in the window, and I did not need to scroll.

Trying to use a Java jmod on midrange

I was using functions shipped in a .jmod on z/OS, and could not find how to get use them in Eclipse IDE.

The code was

...
import com.ibm.oti.shared.*; 
...

Building Java code for z/OS on other operating systems told me how to build a jar file from the .jmod

I had to use the right level of Java on midrange. See Getting the right Java installed on Linux. This level had the openj9.sharedclasses.jmod file.

Even though I built the jar file and updated the Java build class path, it could not find the package, getting message “The package com.ibm.oti.shared is not available”.

I found

  • I had to add the .jmod file to the “module” part of the build class path, and not the classpath itself.
  • I had a jar with the same name in the Eclipse application tree.

I fixed these and it worked.

Getting the right Java installed on Linux

I was programming Java Shared Classes Caching on z/OS, and tried using Eclipse IDE on Linux to edit the source. I had several problems with this.

The import files are in a .jmod file, not the .jar files I was expecting.

Compile on z/OS

To compile the Java source on z/OS I needed to add the .jmod file to the classpath.

cp="-cp /usr/lpp/java/J21.0_64/jmods/openj9.sharedclasses.jmod" 
/usr/lpp/java/J8.0_64/J21.0_64/bin/javac $cp m4.java

With this I could use

...
import com.ibm.oti.shared.*; 
...

When I used the source in the Eclipse IDE, it could not find the imports.

Wrong version of Java

I had installed Java openj9-21 on Linux, but it did not have the openj9.sharedclasses.jmod file.

I downloaded the Semeru code from here.

Often Java is installed in /usr/java/…. files. I installed the new version into one of my personal directories.

The file ibm-semeru-open-jdk_x64_linux_21.0.6_7_openj9-0.49.0.tar.gz was downloaded to my Downloads directory.

You can untar it in place – or unzip it to a different location using the -C …. option

tar -xvf ibm-semeru-open-jdk_x64_linux_21.0.6_7_openj9-0.49.0.tar.gz

This created a directory jdk-21.0.6+7 and created the files below it.

You can update the “sudo update-alternatives…” command by

sudo update-alternatives –list java
sudo update-alternatives –install /usr/bin/javac javac /home/colinpaice/Downloads/jdk-21.0.6+7/bin/javac
sudo update-alternatives –install /usr/bin/java java /home/colinpaice/Downloads/jdk-21.0.6+7/bin/java 1
sudo update-alternatives –install /usr/bin/jar jar /home/colinpaice/Downloads/jdk-21.0.6+7/bin/jar 1

Which defines which symlinks to define for the javac, java and jar commands.

You also need to update your Java_Home environment variable.

There’s more

I had to configure Eclipse to use this new level of Java. Window-> Preferences-> Installed JREs, and use the location of the directory /home/colinpaice/Downloads/jdk-21.0.6+7.

Understanding spawn and _BPX_SHAREAS

You can use spawn() to create another thread to do work. It may be able to run in the same address space as the originator, or it may run in its own address space.

It is cheaper to run in the requester’s address space, as it just creates a new TCB. If it runs in a different address space, in one of the pool of OMVS BPXAS address spaces, there is additional overhead.

I set up a shell script to call a Rexx script which did a spawn of another shell script.

I used the Rexx script to display information about the threads.

With _BPX_SHAREAS=YES – share the address space

the output was

jobname  asid    ppid    pid    threadid  tcb  cmdline
COLIN    21        1 50397218 212A80003 8BEA50 OMVS                    
COLIN    21 50397218 16842787 212A68002 8B9C90 -sh                     
COLIN2   4C 16842787 50397295 212AA8000 8FB2F8 sh kk.sh                
COLIN2   4C 50397295 33620080 212AB0000 8D6A88 ./r.rexx YES

We can see the following

  1. The top level process in OMVS (parent process id) 1 invoked a program OMVS with process id(pid) 50397218, in address space 0x21
  2. This process invoked a shell (-sh) with pid 16842787 in address space 0x21
  3. This executed a command “sh kk.sh” (my test script) with a process id in address space 0x4c, jobname COLIN2, and TCB 8FB2F8.
  4. This invoked shell script invoked a command “./r.rexx YES” in the same address space 0x4c, jobname COLIN2 with a different TCB 8D6A88. This is sharing the address space.

With _BPX_SHAREAS=NO – do not share the address space

the output was similar to the _BPX_SHAREAS=YES, but different

jobname  asid    ppid    pid    threadid  tcb    cmdline
COLIN    21        1 50397218 212A80003 8BEA50 OMVS                        
COLIN    21 50397218 16842787 212A68002 8B9C90 -sh                         
COLIN9   4B 16842787 83951726 212AA8000 8FB380 sh kk.sh                    
COLIN1   4D 83951726 16842864 212AB0000 8FB2F8 ./r.rexx NO
  • The sh k.sh ran in a different address space 0x4B, with a different jobname COLIN9.
  • Because _BPX_SHARESAS=NO, the command “./r.rexx YES” executed in a different address space 0x4d, jobname COLIN1.

Comparison between the two scenarios

  1. With _BPX_SHAREAS=YES, one address space was shared, with two (lightweight) TCBs in it.
  2. With _BPX_SHAREAS=NO, the address spaces were not shared, and one of the pool of BPXAS address spaces were used.

When do you get not shared, even when _BPX_SHAREAS=YES was specified?

There are several cases when the system will not run a program in a shared address space.

Integrity

If there is a mismatch between APF states, for example

  • the caller is APF authorised; you do not want an unauthorised program access the memory in the shared address space of the APF authorised thread.
  • the caller is not APF authorised, but you are calling an APF authorised program.

The called program may change the userid or the group.

If your program changes the userid or group it is running under, for example a web server doing work for different userids. You can set a flag (s) on a file chmod to indicate that this program may change userid or group. See set-user-ID and Set-group-ID in chmod. You can use the ls -ltr command

-rwsr-sr-x   1 OMVSKERN ZWEADMIN    1336 Feb 26 16:41 r.rexx

Where the first s is for set-user-ID. The second s is for set-group-ID.

When the file had u+s, I got the following error message

FSUM9209 cannot execute: reason code = 0b1b0473: EDC5157I An internal error has occurred.