I had a problem with z/OSMF. I configured it to use an exiting keyring, but it consistently refused to use it. I had messages like
[WARNING ] CWPKI0809W: There is a failure loading the defaultKeyStore keystore. If an SSL configuration references the defaultKeyStore keystore, then the SSL configuration will fail to initialize.
This blog post covers how I debugged this situation.
What seemed strange was this only occurred when an Elliptic Curve certificate was being used – and not an RSA certificate.
Even more curiouser was the documentation mentioned access to the <ringOwner>.<ringName>.LST resource in the RDATALIB class. See here. I didn’t have this defined and yet RSA certificates would work! So curiouser and curiouser (or for the people who like correct grammar, curiouser and more curiouser).
All applications needing access to certificates and private keys use the R_datalib callable service.
The bottom line
- z/OSMF has userid IZUSVR
- I had a keyring and used two certificates
- An RSA certificate, CCPKeyring.IZUDFLT, belonging to userid IZUSVR – based on the sample JCL provided by z/OSMF
- An existing Elliptic Curve certificate NISTEC224 belonging to userid COLIN. This works else where.
- Without <ringOwner>.<ringName>.LST defined the class(RDATALIB) the RSA certificate worked
- Without <ringOwner>.<ringName>.LST defined the class(RDATALIB) the Elliptic Curve certificate failed
- Once I found the problem I defined <ringOwner>.<ringName>.LST in class(RDATALIB), and gave the userid IZUSVR Update access to it – and the Elliptic curve worked
- The reasons (being wise after the event)
- R_datalib checks access on one profile in the RDATALIB class first – <ringowner>.<ringname>.LST. If there is none, it will fall back to check on two profiles in the FACILITY class – IRR.DIGTCERT.LISTRING and IRR.DIGTCERT.GENCERT. If the certificate is not owned by the accessing ID (except CERTAUTH or SITE), RDATALIB class has to be used for private key access.
- This is true for the RSA certificate, used the IRRDIGTCERT.LISTRING class(FACILITY) and had access. So this worked.
- For the Elliptic Curve, the caller’s userid (IZUSVR) is not the associated with the certificate (COLIN) so this fails, and the logic drops through to the RDATALIB checking.
- The caller’s user ID has READ or UPDATE authority to the ..LST resource in the RDATALIB class. READ access enables retrieving one’s own private key, UPDATE access enables retrieving other’s. The ring did not exist, and so this access was not given.
How did I debug this? – Using Java trace
Adding configuration to z/OSMF
I copied /global/zosmf/configuration/local_override.cfg to /global/zosmf/configuration/local_override.colin
I edited/global/zosmf/configuration/local_override.cfg and changes the JVM options line to
JVM_OPTIONS=”-Xoptionsfile=’/global/zosmf/configuration/local_override.colin'”
I edited the local_override.colin, deleted all but the JVM options line, then split the line at \n so it looks like
-Dcom.ibm.ws.classloading.tcclLockWaitTimeMillis=300000
-Xscmx150M
-Xquickstart
Add debug information to the configuraton file
I added
-Djava.security.auth.debug=pkcs11keystore
-Dlog.level=Error
The output
[err] Jan 17, 2025 8:18:52 AM com.ibm.crypto.ibmjcehybrid.provider.HybridRACFKeyStore engineLoad
TRACE: Loading keyring CCPKeyring.IZUDFLT as a JCECCARACFKS type keystore.
...
[err] Jan 17, 2025 8:19:02 AM com.ibm.crypto.hdwrCCA.provider.RACFInputStream getEntry
FINER: The private key of NISTEC224 is not available or no authority to access the private key
[err] Jan 17, 2025 8:19:02 AM com.ibm.crypto.ibmjcehybrid.provider.HybridRACFKeyStore engineLoad
TRACE: Error loading and storing certificates and key material from underlying JCECCARACFKS keyring CCPKeyring.IZUDFLT
java.io.IOException: The private key of NISTEC224 is not available or no authority to access the private key . This can be expected if the IBMJCECCA is not setup correctly or
ICSF is down. Will now attempt to load the keyring as a JCERACFKS keyring.
Which is not a very helpful message.
How did I debug this? – Using RACF trace
R_datalib is the callable service to ALL the exploiters which need access to a RACF keyring (certificates and private keys). It is r_datalib or its alias irrsdl00 with callable type number 41.
Enable the RACF trace
#SET TRACE(CALLABLE(TYPE(41))JOBNAME(IZU*))
Start GTF
S GTF.GTF,M=GTFRACF
This reported
IEF403I GTF - STARTED - TIME=08.17.03
IEF188I PROBLEM PROGRAM ATTRIBUTES ASSIGNED
AHL121I TRACE OPTION INPUT INDICATED FROM MEMBER GTFRACF OF PDS
USER.Z24C.PROCLIB
TRACE=USRP
USR=(F44)
END
AHL103I TRACE OPTIONS SELECTED --USR=(F44)
AHL906I THE OUTPUT BLOCK SIZE OF 27998 WILL BE USED FOR OUTPUT 702
DATA SETS:
SYS1.TRACE
I started z/OSMF until it failed.
Stop GTF
p GTF
AHL006I GTF ACKNOWLEDGES STOP COMMAND
AHL904I THE FOLLOWING TRACE DATASETS CONTAIN TRACE DATA :
SYS1.TRACE
Use IPCS to look at the dump, using command GTF USR(ALL). Go to the bottom of the output, use the command report view. This gives an ISPF edit session.
- x all
- f ‘RACF Reason code:’ all
- You are interested in the non zero codes. “Label” each line of interest using the line prefix command .a, .b etc.
- reset
- loc .a
- This will position you by the labelled line. Look up the RACF return and reason codes here. I had Reason Code 2c, which is decimal 44. Look for the keyring, or other information. I do not know which data tells you which sub operation r_datalib was doing, but for me it had the keyring name “CCPKeyring.IZUDFLT “. The description in the reason code documentation does not cover the situation of not having update access to the keyring, so I’ve raised a doc comment on it.

