Java persistent shared classes cache on z/OS

With Java shared classes cache, by default on z/OS saves the data in shared memory. You can use the snapshot command to save a copy on disk, and use the restore command after IPL to recreate it. For my zPDT system running z/OS on a Linux server this many seconds of start up time.

In more recent Java versions, the Shared Classes Cache has supported the persistent option, where shared virtual storage is mapped to a file – and so updating memory, updates the file.

I had a few problems getting this to work, and there was no documentation on the use of the persistent option.

When I enabled it, for example with

 -Xshareclasses:name=zoweGW,cacheDirPerm=0777,cacheDir=/u/tmp/zowec/,persistent" 

I got

JVMSHRC245E Error mapping shared class cache file 
JVMSHRC336E Port layer error code = -155
JVMSHRC337E Platform error message: EDC5132I Not enough memory.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I had to change the SMFLIMxx parmlib member to fix this

Display the current SMFLIM configuration

You can display your current SMFLIMxx configuration using

d smflim
d smflim,r

The d smflim,r gave me

...
Member and rule number SMFLIMCP 0003
User:
ZWESVUSR
Attributes:
EXECUTE: NOCHANGE
JOBMSG: ISSUE
MAXSHARE: 9000000
...

Which shows the rule for user ZWEVSUR came from the third rule in SMFLIMCP. It sets MAXSHARE, and other parameters.

Update the member

I updated my SMFLIMCP member to be

REGION USER(COLIN) JOBMSG(ISSUE) MAXSHARE(90000) 

activated it using t SMFLIM=(CP,C2) where CP2,C2 is my list of SMFLIM members. Note: The T SMFLIM command, replaces all of the definitions with what is in the list, so you need to specify the whole list, not just the changed member.

The definitions become active immediately, you do not need to logoff and logon, or resubmit a job.

When the Java job had started, it created a file C290M21F1A64P_hw_G43L00 in the specified directory.

When persistent was not specified, files were stored in the javasharedresources subdirectory.

Should I use this persistent option?

You have the choice of using the persistent option in the -Xshareclasses…persistent parameters, or not to specify it. If you do not use the persistent option you need to save the shared memory across IPLs, by using -Xshareclasses:…,snapshotCache and restoring it after an IPL using -Xshareclasses:…restoreFromSnapshot. I used this method, and added a steps to my started tasks, one to restore (if the cache exists already, it does nothing), and one at the end, to save it.

How does the performance compare?

On my zPDT system which is not meant to be used for performance evaluations, they both had similar durations, and used similar amounts of CPU, though non persistent was usually slight better.

Funny…

I also go message

JVMSHRC561E Failed to initialize the shared classes cache, there is not enough space in the file system. Available free disk space bytes = 516144128, requested bytes = 536870912.

Which was a surprise as I thought I had enough free disk space.

What’s happening with my Java shared classes?

When your Java application ends it prints information like

JVMSHRC168I Total shared class bytes read=78435794. Total bytes stored=522418
JVMSHRC818I Total unstored bytes due to the setting of shared cache soft max is 0. Unstored AOT bytes due to the setting of -Xscmaxaot is o.
Unstored JIT bytes due to the setting of -Xscmaxjitdata is 0.

So you can see what amount of space is used.

Displaying activity as Java starts

-Xshareclasses:verboseIO,…

For example

 -Xshareclasses:verboseIO,verbose,name=zoweGW,cacheDirPerm=0777,cacheDir=/u/tmp/zowec/" 

This produces output like

Entries which were found and used

Found class java/lang/Object in shared cache for class-loader id 0. 
Found class java/lang/J9VMInternals in shared cache for class-loader id 0.

Entries which were not found…but added.

The following shows that the entry was not found in the share classes cache, but was added successfully. Next time this would give ” found and used” above.

Failed to find class java/security/Guard in shared cache for class-loader id 0. 
Stored class java/security/Guard in shared cache for class-loader id 0 with
URL /Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules (index 0).

What classes were not found and not added?

Failed to find class org/yaml/snakeyaml/constructor/Construct in shared cache for class-loader id 0. 
Failed to find class org/yaml/snakeyaml/constructor/Construct in shared cache for class-loader id 2.

When Java option -verbose:class was enabled, there was additional information

class load: org.yaml.snakeyaml.constructor.Construct from: 
jar:nested:/u/tmp/zowep/components/gateway/bin/gateway-service.jar/!BOOT-INF/lib/snakeyaml-2.2.jar!/

The jar:nested means it is a jar within a bigger jar. The shared class support does not support this.

To exploit the shared classes function it seems to be better to have individual .jar files rather than one large jar file with lots of jar files within it.

The loader id 2 is from the spring framework loaded. This loads the nested jar files.

What classes were loaded from where?

With the Java option -verbose:class (not a shared classes cache option) this reports on classes used.

 class load: java/net/URLClassLoader$SharedClassMetaDataCache from: jrt:/java.base 
class load: java/net/URLClassLoader$SharedClassMetaData from: jrt:/java.base

What’s in the shared cache?

-Xshareclasses:…,printStats

Use the printStats command

/usr/lpp/java/J21.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,printStats,

to display a summary of the cache

cache layer                          = 0                       
cache size = 83885520
softmx bytes = 83885520
free bytes = 31274608
Reserved space for AOT bytes = -1
Maximum space for AOT bytes = -1
Reserved space for JIT data bytes = -1
Maximum space for JIT data bytes = 31457280
Metadata bytes = 752058
Metadata % used = 1%
Class debug area size = 6680576
Class debug area used bytes = 5576890
Class debug area % used = 83%

ROMClass bytes = 35210448
AOT bytes = 8456750
JIT data bytes = 1150472
Zip cache bytes = 0
Startup hint bytes = 240
Data bytes = 360368

# ROMClasses = 14554
# AOT Methods = 3510
# Classpaths = 2
# URLs = 0
# Tokens = 0
# Zip caches = 0
# Startup hints = 2
# Stale classes = 0
% Stale classes = 0%


Cache is 62% full

Cache is accessible to current user = true

Class debug area % used. If this is close to 100% your cache is too small, you will need to destroy the cache, and make it bigger.

-Xshareclasses:…,printAllStats

The printAllStats command provides the information in printStats, and detailed information like

1: 0x000002003ADA0DB4 CLASSPATH 
/Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules
1: 0x000002003ADA0D80 ROMCLASS: java/lang/Object at 0x00000200364581E0.
Index 0 in classpath 0x000002003ADA0DB4
ROMMETHOD: <init> Signature: ()V Address: 0x00000200364583D8
ROMMETHOD: clone Signature: ()Ljava/lang/Object; Address: 0x00000200364583F4
ROMMETHOD: equals Signature: (Ljava/lang/Object;)Z Address: 0x000002003645841C
ROMMETHOD: finalize Signature: ()V Address: 0x000002003645844C
ROMMETHOD: getClass Signature: ()Ljava/lang/Class; Address: 0x0000020036458484
ROMMETHOD: hashCode Signature: ()I Address: 0x00000200364584A0
ROMMETHOD: notify Signature: ()V Address: 0x00000200364584C0
ROMMETHOD: notifyAll Signature: ()V Address: 0x00000200364584D8
ROMMETHOD: toString Signature: ()Ljava/lang/String; Address: 0x00000200364584F0
ROMMETHOD: wait Signature: ()V Address: 0x000002003645852C
ROMMETHOD: wait Signature: (J)V Address: 0x0000020036458554
ROMMETHOD: wait Signature: (JI)V

This shows for class java/lang/Object coming from /Z31B/usr/lpp/java/J8.0_64/J21.0_64/lib/modules there were several methods.

Setting up Java shared cache support on z/OS

With Java shared cache support, Java stores classes and JIT information in shared memory, so it can be reused by other Java applications, or it the application is restarted.

Some other shared classes blog posts

You can use the Java command snapshotCache to save the shared buffer to disk, and restoreSnapshot to restore from disk to recreate the shared memory.

For my Zowe startup I use Java start-up option

-Xshareclasses:verboseIO,verbose,name=zoweGW,cacheDirPerm=0777,cacheDir=/u/tmp/zowec/

Before you start you need to consider

  • Sharing the shared classes cache
  • Which directory to use
  • What name to use
  • How big
  • Processes to save the cache to disk
  • Read only?

Sharing the shared classes cache

Java uses a semaphore when updating the shared classes cache. I believe that multiple Java instances can share the cache, so products which have multiple address spaces can share the same cache if they are using the same level of Java.

I believe you could use one shared classes directory for all your Java applications, Liberty, z/OSMF, zOS Connect, Zowe etc. I haven’t found any documentation to say you cannot share it, but the application’s userid needs access to the cache, and so may need the same userid, or be in the correct group to be able to access the cache, and ipc memory and semaphores. The Zowe address spaces all use the same userid, so all have access to the cache.

To further complicate it…

You can run more than one JVM within a CICS address space, they can all use the same cache.

To be able to destroy and recreate a cache (for example to make it larger ), the cache cannot be in use. For availability you may want multiple caches so you do not need to shutdown all CICS regions to be able to recreate it. As all Zowe address spaces start and stop together, they can use a shared classes cache.

Which directory to use?

If you do not specify a cachDir, it defaults to /tmp. This is OK – but some on some z/OS systems, the /tmp directory is emptied on a regular basis. This means your cache is deleted, and it needs to be recreated next time your application starts.
For Zowe I used /u/tmp/zowec.

What name to use?

If you do not specify a name it takes the userid of the application, as part of the file name.

Within my cacheDir, it creates a directory javasharedresources. Within this directory I have files

  • C290M21F1A64S_zoweGW_G43L00 with size 83886080 (84MB) this is the hardened copy of the cache
  • C290M21F1A64_semaphore_zoweGW_G43L00 with size 32
  • C290M21F1A64_memory_zoweGW_G43L00 with size 40

Where the …M21F… is the level ofJava.

When I use the printStats command, it gives the cache size as 84MB.

How big?

I specified
-Xscmx80m
-Xscmaxaot180m
-Xscmaxjitdata30m

Once you have specified the size, you cannot change it unless you delete and recreate the cache, so you need to get it right before you start. Make it too large, rather than too small. Restoring it from a snapshot will use the size when the snapshot was taken.

Processes to save the cache to disk.

As part of your application shutdown you could snapshot the cache to a file. See here (tba).

Processes to save the cache to disk.

You might want to restore the cache from disk as part of your IPL processing, or as part of application start up. If the cache already exists it is not overwritten. It takes a few seconds to snapshot or restore. See here(tba).

Clean up the shared classes

If you delete the ipc shared memory, and the ipc semaphore before you start your Java application, will rebuild the shared classes – though this will take longer as all of the classes need to be read and processed.

Read only

If Java detects that the shared classes cache is out of date, it will updates the shared classes cache.

Once you created and populated a shared classes cache, you can specify readOnly, to prevent updates to it

Introduction to Java Shared classes on z/OS

Java shared classes provides the facility for Java to store classes, and JITted classes in shared memory which persists across the restart of a Java application. You can harden this shared memory to disk, and reload it after an IPL.

If shared classes are being used, then as the JVM starts it looks for the classes in the shared cache, and if not found, loads them from the file system, processes them and stores them in the shared classes memory. With this scheme, the second start of the JVM gains from the shared classes.

See the introduction documentation and some practical documentation.

Background

To support the use of the shared memory, Java uses standard facilities for shared memory, and for semaphores (to ensure single threading of updates).

I have my shared classes in directory /u/tmp/zowec. When the shared classes are used, two files are created.

  • javasharedresources/C290M17F1A64_semaphore_zoweGW_G43L00
  • javasharedresources/C290M17F1A64_memory_zoweGW_G43L00

Listing them gives

672 -rw-r--r--   1 ZWESVUSR ZWEADMIN ... C290M17F1A64_semaphore_zoweGW_G43L00  
673 -rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_memory_zoweGW_G43L00

The files are owned by userid ZWESVUSR and group ZAWADMIN from my started task userid. The semaphore file has an inode of 672 (0x2a0), and the shared memory file has an inode of 673(0x2a1).

The shared memory and semaphores use InterProcess Communication (ipc) facilities.

The id in hex of the files is used as part of the ipc key.

The command ipcs gave

Shared Memory:                                                
T ID KEY MODE OWNER GROUP
m 204805 0x6102a131 --rw------- ZWESVUSR ZWEADMIN

Semaphores:
T ID KEY MODE OWNER GROUP
s 200708 0x8102a031 --ra------- ZWESVUSR ZWEADMIN

These resources are owned by userid:ZWESVUSR, and group:ZWEADMIN. The userid and group comes from the userid that created the shared classes cache.

If you need to “clean up” you can delete these using the ipcrm command.