Notes on shared storage

Shared storage is a Unix concept, which works on Unix on z/OS.

It allows you to share data between threads, and between different address spaces. Java Shared Classes exploit this. Java classes can be stored in the shared storage, and “Ahead Of Time” compilation can be stored in the shared cache. Next time a JVM starts, it can look in the shared cache, and use the already compiled class.

Each shared storage segment has a key which you use in the various shared memory API calls.

There are no real restrictions, in some of the documentation on the web, it suggests using 1234 as a key! This has its limitations. A better way, used by Java is based on the inode of a file

You can display information about InterProcess Communication Status (ipcs) using the ipcs command.

You can display shared memory, and shared semaphores.

ipcs command lists… 

Shared Memory:                                                           
T         ID     KEY        MODE       OWNER    GROUP                    
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                    
m      73733 0x6137142d --rw------- OMVSKERN     SYS1                    
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                    
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                    
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1

The Java shared class files on my system are

767 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_semaphore_zoweGW_G43L00             
768 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_memory_zoweGW_G43L00

Inode 767 is 0x2ff, Inode 768 is 0x300.

0x61… inode…. 2d

061 because created with t = ftok(fn,idi );

Shared Memory:                                                          
T         ID     KEY        MODE       OWNER    GROUP                   
m       8196 0x00000000 --rw-------   WEBSRV   WEBGRP                   
m     139269 0x6137142d --rw------- OMVSKERN     SYS1                   
m      73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN                   
m     139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN                   
m       8200 0x610ea00f --rw------- OMVSKERN     SYS1                   
Semaphores:                                                             
T         ID     KEY        MODE       OWNER    GROUP                   
s     135172 0x8137132d --ra------- OMVSKERN     SYS1                   
s      69637 0x8137e72f --ra------- ZWESVUSR ZWEADMIN                   
s     135174 0x8100400f --ra------- ZWESVUSR ZWEADMIN                   
s      69639 0x8c10840f --ra-ra-ra- ZWESVUSR ZWEADMIN                   
s     135178 0x810e9f0f --ra------- OMVSKERN     SYS1                   
s      69643 0x8c108a0f --ra-ra-ra- ZWESVUSR ZWEADMIN

More details

You can use

ipcs -m -a

to display more information including size, and number of users.

Using Java internal trace

I was trying to understand a class load problem, and needed to use Java internal virtual machine trace to see what was happening.

You specify -Xtrace options at start up. Even with limiting options, I had over 2 millions lines of trace output – and the Java application had not fully started!

There are lots of components you can specify see here. I used

 -Xtrace:maximal={j9jcl,j9shr,j9prt,omrport},output=/u/tmp/java/trace.bin

where

  • j9jcl is Java class libraries
  • j9shr is shared classes libraries
  • j9prt is VM port libraries… the platform specific (ported) code
  • omrport is the OMR ported code. (OMR is a set of common components, such as threads and garbage collection)

I haven’t used it,but you can say do something when this trace point is triggered, stop (suspend) tracing.

-Xtrace:trigger=tpnid{abc.123,suspend}

In the example trace output

07:49:24.699384689 0x000000002167f200 omrport.657 Event J9 VM NLS message: Failed to find class org/springframework/boot/loader/launch/PropertiesLauncher in shared cache for class-loader id 0.

where the trace point is omrport.657

You can put the definitions in a file -Xtrace:properties=<filename>

I have a trace file – now what?

You can format the trace file on z/OS, but I downloaded the file (in binary) to my Linux machine, because a) I have more disk space, 2) the java formatting is faster, 3) I have better tools to look at the data. You need the save level of Java (or better) to format all the fields. If there is a mismatch, some fields will not be formatted. The traceformat command is described here.

Format the data

/home/colinpaice/Downloads/jdk-21.0.6+7/bin/traceformat -verbose trace.bin trace.txt

It produced

Writing formatted trace output to file trace.txt
Processing 245.45312Mb of binary trace data
Processed 10.0Mb (4%), burst speed: 1.9344624Mb/s, average: 1.9344624Mb/s
Processed 20.0Mb (8%), burst speed: 2.4675941Mb/s, average: 2.1687446Mb/s
Processed 30.0Mb (12%), burst speed: 3.4868424Mb/s, average: 2.4814205Mb/s
...

I wrote the blog post Finding the needle in a field of hay, to help extract the records of interest.

On the traceformat command you can specify

  • -indent: Indents trace messages at each Entry trace point and outdents trace messages at each Exit trace point. The default is not to indent the messages.
  • -threads: Filters the output for the given thread IDs only. thread id is the ID of the thread, which can be specified in decimal or hex (0x) format. Any number of thread IDs can be specified, separated by commas.

Looking at the data

I used the commands grep -n …., tail -n +999, head -n 500, less as described here. But for detailed analysis I piped the output into a file, and used an editor on the output. This meant the lines did not wrap, I could annotate (and save) the trace, and highlight interesting records.

Because some of the lines are very long, I split interesting lines so all of the data was displayed in the window, and I did not need to scroll.

Trying to use a Java jmod on midrange

I was using functions shipped in a .jmod on z/OS, and could not find how to get use them in Eclipse IDE.

The code was

...
import com.ibm.oti.shared.*; 
...

Building Java code for z/OS on other operating systems told me how to build a jar file from the .jmod

I had to use the right level of Java on midrange. See Getting the right Java installed on Linux. This level had the openj9.sharedclasses.jmod file.

Even though I built the jar file and updated the Java build class path, it could not find the package, getting message “The package com.ibm.oti.shared is not available”.

I found

  • I had to add the .jmod file to the “module” part of the build class path, and not the classpath itself.
  • I had a jar with the same name in the Eclipse application tree.

I fixed these and it worked.

Getting the right Java installed on Linux

I was programming Java Shared Classes Caching on z/OS, and tried using Eclipse IDE on Linux to edit the source. I had several problems with this.

The import files are in a .jmod file, not the .jar files I was expecting.

Compile on z/OS

To compile the Java source on z/OS I needed to add the .jmod file to the classpath.

cp="-cp /usr/lpp/java/J21.0_64/jmods/openj9.sharedclasses.jmod" 
/usr/lpp/java/J8.0_64/J21.0_64/bin/javac $cp m4.java

With this I could use

...
import com.ibm.oti.shared.*; 
...

When I used the source in the Eclipse IDE, it could not find the imports.

Wrong version of Java

I had installed Java openj9-21 on Linux, but it did not have the openj9.sharedclasses.jmod file.

I downloaded the Semeru code from here.

Often Java is installed in /usr/java/…. files. I installed the new version into one of my personal directories.

The file ibm-semeru-open-jdk_x64_linux_21.0.6_7_openj9-0.49.0.tar.gz was downloaded to my Downloads directory.

You can untar it in place – or unzip it to a different location using the -C …. option

tar -xvf ibm-semeru-open-jdk_x64_linux_21.0.6_7_openj9-0.49.0.tar.gz

This created a directory jdk-21.0.6+7 and created the files below it.

You can update the “sudo update-alternatives…” command by

sudo update-alternatives –list java
sudo update-alternatives –install /usr/bin/javac javac /home/colinpaice/Downloads/jdk-21.0.6+7/bin/javac
sudo update-alternatives –install /usr/bin/java java /home/colinpaice/Downloads/jdk-21.0.6+7/bin/java 1
sudo update-alternatives –install /usr/bin/jar jar /home/colinpaice/Downloads/jdk-21.0.6+7/bin/jar 1

Which defines which symlinks to define for the javac, java and jar commands.

You also need to update your Java_Home environment variable.

There’s more

I had to configure Eclipse to use this new level of Java. Window-> Preferences-> Installed JREs, and use the location of the directory /home/colinpaice/Downloads/jdk-21.0.6+7.

Understanding spawn and _BPX_SHAREAS

You can use spawn() to create another thread to do work. It may be able to run in the same address space as the originator, or it may run in its own address space.

It is cheaper to run in the requester’s address space, as it just creates a new TCB. If it runs in a different address space, in one of the pool of OMVS BPXAS address spaces, there is additional overhead.

I set up a shell script to call a Rexx script which did a spawn of another shell script.

I used the Rexx script to display information about the threads.

With _BPX_SHAREAS=YES – share the address space

the output was

jobname  asid    ppid    pid    threadid  tcb  cmdline
COLIN    21        1 50397218 212A80003 8BEA50 OMVS                    
COLIN    21 50397218 16842787 212A68002 8B9C90 -sh                     
COLIN2   4C 16842787 50397295 212AA8000 8FB2F8 sh kk.sh                
COLIN2   4C 50397295 33620080 212AB0000 8D6A88 ./r.rexx YES

We can see the following

  1. The top level process in OMVS (parent process id) 1 invoked a program OMVS with process id(pid) 50397218, in address space 0x21
  2. This process invoked a shell (-sh) with pid 16842787 in address space 0x21
  3. This executed a command “sh kk.sh” (my test script) with a process id in address space 0x4c, jobname COLIN2, and TCB 8FB2F8.
  4. This invoked shell script invoked a command “./r.rexx YES” in the same address space 0x4c, jobname COLIN2 with a different TCB 8D6A88. This is sharing the address space.

With _BPX_SHAREAS=NO – do not share the address space

the output was similar to the _BPX_SHAREAS=YES, but different

jobname  asid    ppid    pid    threadid  tcb    cmdline
COLIN    21        1 50397218 212A80003 8BEA50 OMVS                        
COLIN    21 50397218 16842787 212A68002 8B9C90 -sh                         
COLIN9   4B 16842787 83951726 212AA8000 8FB380 sh kk.sh                    
COLIN1   4D 83951726 16842864 212AB0000 8FB2F8 ./r.rexx NO
  • The sh k.sh ran in a different address space 0x4B, with a different jobname COLIN9.
  • Because _BPX_SHARESAS=NO, the command “./r.rexx YES” executed in a different address space 0x4d, jobname COLIN1.

Comparison between the two scenarios

  1. With _BPX_SHAREAS=YES, one address space was shared, with two (lightweight) TCBs in it.
  2. With _BPX_SHAREAS=NO, the address spaces were not shared, and one of the pool of BPXAS address spaces were used.

When do you get not shared, even when _BPX_SHAREAS=YES was specified?

There are several cases when the system will not run a program in a shared address space.

Integrity

If there is a mismatch between APF states, for example

  • the caller is APF authorised; you do not want an unauthorised program access the memory in the shared address space of the APF authorised thread.
  • the caller is not APF authorised, but you are calling an APF authorised program.

The called program may change the userid or the group.

If your program changes the userid or group it is running under, for example a web server doing work for different userids. You can set a flag (s) on a file chmod to indicate that this program may change userid or group. See set-user-ID and Set-group-ID in chmod. You can use the ls -ltr command

-rwsr-sr-x   1 OMVSKERN ZWEADMIN    1336 Feb 26 16:41 r.rexx

Where the first s is for set-user-ID. The second s is for set-group-ID.

When the file had u+s, I got the following error message

FSUM9209 cannot execute: reason code = 0b1b0473: EDC5157I An internal error has occurred.

Restoring a copy of the Java Shared classes cache from disk

On z/OS the Java shared classes use shared storage. Data is added to the shared classes as the application process classes, so successive Java applications can use this shared data.

You can harden this to disk, so it is available across an IPL, using the snapshotCache command.

When restoring the snapshot, you need to be careful to get the file/semaphore/shared memory owner and permissions correct. I found the easiest way is to the the started procedure which runs the Java application to always do a restoreSnapshotCache. It is very easy to get the permissions wrong, and Java will not start.

A typical (for me) command is

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot

What happens when you restore a snapshot?

Restoring from an “normal” userid with no authority

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot
JVMSHRC721E The JVM is not configured to access the non-persistent shared cache snapshot file “/u/tmp/zowec/javasharedresources/C290M17F1A64S_zoweGW_G43L00”. To open the file, use ‘groupAccess’ sub-option.
JVMSHRC699E Failed to restore the non-persistent shared cache “zoweGW” from the snapshot

Using an “normal” userid and groupAccess gave

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot,groupAccess
JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360
JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.
JVMSHRC727E An error has occurred in creating the new non-persistent shared cache
JVMSHRC808I Compressed references shared cache “zoweGW” is destroyed. Use option -Xnocompressedrefs if you want to destroy a non-com pressed references cache.
JVMSHRC699E Failed to restore the non-persistent shared cache “zoweGW” from the snapshot

Using a userid with su (id = 0) userid, also failed to work for the same reasons

The userid ZWESVUSR is a nolog userid, so you cannot logon to TSO with it.

Become the cache owner

A superuser can issue

su ZWESVUSR

Then issue the restoreFromSnapshot command, and remember to issue “exit” to get out of su mode.

Do it as part of your application startup

I have a started task with userid ZWESVUSR and issue restoreFromSnapshot command from BPXBATCH. It could be the first step of your Java application startup.

For example, the JCL

//CCPCACH  JOB 
//* 

//RESTORE  EXEC PGM=BPXBATCH,REGION=0M,PARMDD=PARMDD 
//SYSPRINT DD SYSOUT=* 
//SYSOUT   DD SYSOUT=* 
//STDOUT   DD SYSOUT=* 
//STDERR   DD SYSOUT=* 
//PARMDD  DD *
SH  /u/tmp/zowec/restore.sh 
//* 
//

The script 

#!/bin/sh 
set -x 
/usr/lpp/java/J8.0_64/J21.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot,groupAccess

If the shared cache is active you get a message

JVMSHRC726E Non-persistent shared cache “zoweGW” already exists. It cannot be restored from the snapshot.

which can be ignored.

The restoreSnapshotCommand takes a few seconds to run.

Saving a copy of the Java Shared classes to disk

On z/OS the Java shared classes use shared virtual storage. Data is added to the shared classes as the application process classes, so successive Java applications can use this shared data.

You can harden this to disk, so it is available across an IPL, by using the Java shared classes snapshotCache command, and run a restoreSnapshortCache after the IPL (or as part of your application start up).

What happens when you snapshot a cache?

You use a command like

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,snapshotCache

This creates a file in the “cacheDir” like

 828 -rwxrwxrwx   1 ZWESVUSR ZWEADMIN 10485760 C290M17F1A64S_zoweGW_G43L00

Where the file is owned by userid that created the shared cache, and its group. User ZWESVUSR, and group:ZWEADMIN. The release of Java is in the file names… C290M17.

If you do not have authority to create such as file the snapshot request will fail.

You might want to consider putting a snapshot job step into your Java applications JCL, so you always save the latest copy.

What’s happening with my Java shared classes?

When your Java application ends it prints information like

JVMSHRC168I Total shared class bytes read=78435794. Total bytes stored=522418
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is o.
Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0.

So you can see what amount of space is used.

Displaying activity as Java starts

-Xshareclasses:verboseIO,…

For example

 -Xshareclasses:verboseIO,verbose,name=zoweGW,cacheDirPerm=0777,cacheDir=/u/tmp/zowec/"

This produces output like

Entries which were found and used

Found class java/lang/Object in shared cache for class-loader id 0. 
Found class java/lang/J9VMInternals in shared cache for class-loader id 0.

Entries which were not found…but added.

The following shows that the entry was not found in the share classes cache, but was added successfully. Next time this would give ” found and used” above.

Failed to find class java/security/Guard in shared cache for class-loader id 0. 
Stored class java/security/Guard in shared cache for class-loader id 0 with 
URL /Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules (index 0).

What classes were not found and not added?

Failed to find class org/yaml/snakeyaml/constructor/Construct in shared cache for class-loader id 0. 
Failed to find class org/yaml/snakeyaml/constructor/Construct in shared cache for class-loader id 2.

When Java option -verbose:class was enabled, there was additional information

class load: org.yaml.snakeyaml.constructor.Construct from: 
jar:nested:/u/tmp/zowep/components/gateway/bin/gateway-service.jar/!BOOT-INF/lib/snakeyaml-2.2.jar!/

The jar:nested means it is a jar within a bigger jar. The shared class support does not support this.

To exploit the shared classes function it seems to be better to have individual .jar files rather than one large jar file with lots of jar files within it.

The loader id 2 is from the spring framework loaded. This loads the nested jar files.

What classes were loaded from where?

With the Java option -verbose:class (not a shared classes cache option) this reports on classes used.

 class load: java/net/URLClassLoader$SharedClassMetaDataCache from: jrt:/java.base 
 class load: java/net/URLClassLoader$SharedClassMetaData from: jrt:/java.base

What’s in the shared cache?

-Xshareclasses:…,printStats

Use the printStats command

/usr/lpp/java/J21.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,printStats,

to display a summary of the cache

cache layer                          = 0                       
cache size                           = 83885520                
softmx bytes                         = 83885520                
free bytes                           = 31274608                
Reserved space for AOT bytes         = -1                      
Maximum space for AOT bytes          = -1                      
Reserved space for JIT data bytes    = -1                      
Maximum space for JIT data bytes     = 31457280                
Metadata bytes                       = 752058                  
Metadata % used                      = 1%                      
Class debug area size                = 6680576                 
Class debug area used bytes          = 5576890                 
Class debug area % used              = 83%                     
                                                               
ROMClass bytes                       = 35210448                
AOT bytes                            = 8456750                 
JIT data bytes                       = 1150472                 
Zip cache bytes                      = 0                       
Startup hint bytes                   = 240                     
Data bytes                           = 360368                  
                                                               
# ROMClasses                         = 14554                   
# AOT Methods                        = 3510                    
# Classpaths                         = 2                       
# URLs                               = 0                       
# Tokens                             = 0                       
# Zip caches                         = 0                       
# Startup hints                      = 2                       
# Stale classes                      = 0                       
% Stale classes                      = 0%                      
                                                               
                                                               
Cache is 62% full                                              
                                                               
Cache is accessible to current user = true

Class debug area % used. If this is close to 100% your cache is too small, you will need to destroy the cache, and make it bigger.

-Xshareclasses:…,printAllStats

The printAllStats command provides the information in printStats, and detailed information like

1: 0x000002003ADA0DB4 CLASSPATH 
 /Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules 
1: 0x000002003ADA0D80 ROMCLASS: java/lang/Object at 0x00000200364581E0. 
 Index 0 in classpath 0x000002003ADA0DB4 
 ROMMETHOD: <init> Signature: ()V Address: 0x00000200364583D8 
 ROMMETHOD: clone Signature: ()Ljava/lang/Object; Address: 0x00000200364583F4 
 ROMMETHOD: equals Signature: (Ljava/lang/Object;)Z Address: 0x000002003645841C 
 ROMMETHOD: finalize Signature: ()V Address: 0x000002003645844C 
 ROMMETHOD: getClass Signature: ()Ljava/lang/Class; Address: 0x0000020036458484 
 ROMMETHOD: hashCode Signature: ()I Address: 0x00000200364584A0 
 ROMMETHOD: notify Signature: ()V Address: 0x00000200364584C0 
 ROMMETHOD: notifyAll Signature: ()V Address: 0x00000200364584D8 
 ROMMETHOD: toString Signature: ()Ljava/lang/String; Address: 0x00000200364584F0 
 ROMMETHOD: wait Signature: ()V Address: 0x000002003645852C 
 ROMMETHOD: wait Signature: (J)V Address: 0x0000020036458554 
 ROMMETHOD: wait Signature: (JI)V

This shows for class java/lang/Object coming from /Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules there were several methods.

Not for humans, but for search engines – Java messages and codes

I hit these messages when doing some Java work. Ive added more information to the messages. I expect people will only get to this page from a web search.

EDC messages

EDC5129I No such file or directory.

Also ENOENT error code 129.

This is not strictly true. When using semget() this code means the userid does not have access to it. Perhaps EACCESS code would have been better.

Java shared classes.

JVMSHRC336E Port layer error code = …

For example JVMSHRC336E Port layer error code = -197360. -197360 is 0xfffC FD10

From the first half word you can tell the external function which caused the problem

FTOK   FFFD
SEMGET FFFC
SEMCTL FFFB
SEMOP  FFFA
SHMGET FFF9
SHMCTL FFF8
SHMAT  FFF7
SHMDT  FFF6
GETIPC FFF5

See here for the code saying…

  • EACCESS FD12 -Permission is denied.
  • EEXIST FD11 -The file exists.
  • ENOENT FD10 – No such file, directory, or IPC member exists.
  • EINVAL FD0F – The parameter is incorrect.
  • ENOMEM FD0E – Not enough space is available (I think this means virtual storage).
  • ENOSPC FD0D -No space is left on the device, or no space is available to create the IPC member ID.
  • ELOOP FD0C – A loop is encountered in symbolic links.
  • ENAMETOOLONG FD0B – The file name is too long.
  • ENOTDIR FD0A – Not a directory.
  • EPERM FD09 -The operation is not permitted.
  • ERANGE FD08 -Result is too large.
  • E2BIG FD07 – The parameter list is too long, or the message to receive was too large for the buffer.
  • EAGAIN FD06 – The resource is temporarily unavailable.
  • EFBIG FD05 – The file is too large.
  • EIDRM FD04 – Identifier removed.
  • EINTR FD03 – A function call is interrupted.
  • EMFILE FD02 – Too many files are open for this process.

So JVMSHRC336E Port layer error code = -197360 is 0xfffC FD10 which is SEMGET ENOENT .

One’s I have hit

  • -197359 is 0xfffC FD11 which is SEMGET EEXIST -The file exists.
  • -197360 is 0xfffC FD10 which is SEMGET ENOENT – No such file, directory, or IPC member exists.
  • –262894 is 0xfffb FD12 which is SEMCTL EACCESS – Permission denied.
  • -328433 is 0fffa FD0F which is SEMOP – The parameter is incorrect.
  • -393968 is 0xfff9 fd10 which is SHMGET EPERM -The operation is not permitted.
  • -459502 is 0xfff8 FD12 which SHMCTL EACCESS -Permission is denied.

JVMSHRC337E Platform error message: EDC5132I Not enough memory. (errno2=0x072B06AB)

The errno 072b06ab says


JRMmapMaxShareFail: A map request is attempted but the total share pages exceeds the MAXSHARE limit
Action: Ensure mmap usage does not exceed MAXSHARE limits.

You need to ensure that your system has the SMFLIMxx parmlib updates. This is a requirement for the ibmjava:8 container image in order to support caches mapped above the 2 GB address range. The maximum size of these caches are limited by the MAXSHARE value within the SMFLIMxx PARMLIB member.

Issue

D SMFLIM

to see what limits you have defined.

I had NO SMF LIMITS ARE IN EFFECT.

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -262894

JVMSHRC337E Platform error message: semctl : EDC5111I Permission denied. (errno2=0x070E0303)
JVMSHRC028E Permission Denied
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine

-262894 is SEMCTL EACCESS – Permission denied.

The owner of the shared file cache was not the userid trying to use it.

chown ZWESVUSR:SYS1 *

Changed the files, such as

ZWESVUSR SYS1     314572800 Feb  9 08:43 C290M17F1A64S_zoweGW_G43L00                   
ZWESVUSR SYS1          32 Feb 10 06:35 C290M17F1A64_semaphore_zoweGW_G43L00            
ZWESVUSR SYS1          40 Feb 10 06:35 C290M17F1A64_memory_zoweGW_G43L00

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360

JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.

-197360 is SEMGET ENOENT – No such file, directory, or IPC member exists.

I got these trying to restore a sharedclasses cache, when I did not have access to the file.

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot

I gave the userid access and it worked

chmod 777 /u/tmp/zowec/javasharedresources/*

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC336E Port layer error code = -459502


JVMSHRC337E Platform error message: shmctl : EDC5111I Permission denied.
JVMSHRC028E Permission Denied
JVMSHRC626I The stats of the shared cache cannot be obtained since a valid shared cache does not exist.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed

-459502 is SHMCTL EACCESS -Permission is denied.

The userid issuing the command does not have access to the resource.

The documentation says the shared class cache is created with ONLY USER read/write access by default unless the groupAccess command-line suboption is used, in which case the access is read/write for user and groups.

Note: Users with super user authority gid=0(SYS1) can issue the command with no additional authority.

To find the group list the directories containing the cache, for example if /var/zosmf/data/logs/.classCache/ was specified use ls -ltr /var/zosmf/data/logs/.classCache/javasharedresources.

For me it had owner IZUSVR group IZUADMIN.

I used the RACF command connect COLIN group(IZUADMIN) to connect the userid to the group. Even then the command failed, because groupAccess had not been defined on the -Xshareclasses… parameter. I had to delete the cache so it was recreated next time theJVM started. Then the java -Xshareclasses:cacheDir=/var/zosmf/data/logs/.classCache,name=liberty-IZUSVR,verbose,printStats worked.

JVMSHRC023E   Cache does not exist

I had

-Xshareclasses:cacheDir=/javasc,name=izusvr1cache,printStats

I had to remove the printStats.

JVMSHRC364E SH_OSCachesysv::acquireWriteLock() call to j9shsem_wait on semid … has failed with error -328433.

-328433 is SEMOP – The parameter is incorrect.

You can use the ipcs Unix commands to display the semaphore ids.

JVMSHRC005I No shared class caches available

I was using

/usr/lpp/java/J11.0_64/bin/java -Xshareclasses:cacheDir=/global/zosmf/data/logs/.classCache/,verbose,listAllCaches

to display information about shared cache usage, and kept getting the JVMSHRC005I No shared class caches available message. I experienced two reasons for this.

  1. The information in the file, was for last week’s IPL, and the the information in today’s memory was invalid.
  2. I was using the wrong level of Java. Once I used the right level of Java it worked!

restoreFromSnapshot

IBMUSER:/u/ibmuser: >cd /u/tmp/zowec
IBMUSER:/u/tmp/zowec: >/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot
JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360
JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.
JVMSHRC727E An error has occurred in creating the new non-persistent shared cache
JVMSHRC808I Compressed references shared cache “zoweGW” is destroyed. Use option -Xnocompressedrefs if you want to destroy a non-com pressed references cache.
JVMSHRC699E Failed to restore the non-persistent shared cache “zoweGW” from the snapshot

This may be connected to the the following

The following files were in the directory 

-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_semaphore_zoweGW_G43L00
-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_memory_zoweGW_G43L00

For example the above files were had owner: ZWESVUSR group: ZWEADMIN.

The userid was in group ZWEADMIN, and so does not get R/W access to the files.

Errno2

  • 0x071D0303: JRIpcDenied: Access was denied because the caller does not have the correct permission.
  • 0x053b006c: JRFileNotThere: The requested file does not exist
  • 0x0594003d: JRDirNotFound: A directory in the pathname was not found

Java Health center messages

Health center Non IBM version of Java Health Center client

I got this when connecting the Health Center on Eclipse to a Liberty server. I think the message really means, unable to connect to the port.

The TSO command tso netstat allconn did not show it was active.