ZWEAO503E The server is not ready to handle the request: /apicatalog/api/v1/

I got this when the ZAAS service was stopped, use the f zowe,appl=DISPLAY to display the status of the various tasks.

You can use f zowe,appl=start(ZAAS) to start it. Check the zowe sysprint data set for any error messages.

I also got the following message in the z/OSMF logs

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user adcdb has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

BPXTLS failed: rc=-1, return code=143, reason code=0x0be80000

Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required
Other platforms: impersonation is not supported

This is actually from BPX1TLS, phread_security_np. There are not many cases where 0be80000 is mentioned. This link mentions PTKTDATA profiles.

LoadBalancer does not contain an instance for the service zaas

This was a tricky one to track down.

Zowe works with z/OSMF. They communicate with certificates. This means the z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key.

Not only that , if verifyCertificates STRICT is specified in the zowe.yaml file, then Zowe will check the server’s certificate sent from z.OSMF. This includes the altname or altIP, the altIP is the IP address of the connection. An internal IP address is 127.0.0.0. An external IP address is almost anything else. On my system I have addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6. You can display them using the TSO NETSTAT HOME command.

The zOSMF certificate did not have an altip specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work.

You can also get this message when the system is partially up.

ZWEAM701E; InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name

The request to the URL ‘/zaas/api/v1/auth/login’ has failed: SignatureException: Unable to compute RS256 signature. Cause: Unrecognized RSA or RSASSA-PSS key algorithm name. caused by: InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name. messageAction:Refer to specific exception details for troubleshooting.

It looks you Zowe server needs an RSA key for use with JWT. I changed from an Elliptic key to an RSA and it worked.

Using the web browser

403 Impersonation error

• Error: response status is 403
• Download
• Impersonation error

In the logs/zssServer….log it has

BPXTLS failed: rc=-1, return code=139, reason code=0x0be803d1
Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required

This is because Zowe does not have access to the profiles.

Action:
Check the profiles are defined, and the Zowe userid has UPDATE access to them.

Header X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I actually got the message ZWEAG160E No authentication provided in the request 5 times.

The header is collecting all errors during the treatment of the request. Because the response was 503 the Gateway retries it 5 times. This is strange, but expected behaviour.

I also got an error message in the z/OSMF log /global/zosmf/data/logs/zosmfServer/logs/trace.log

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

Give the userid access to the profile. It may be easier to connect the userid to the group. In my case to group IZUUSER.

I gave the userid access and then got

IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request

{
“additionalInfo”: null,
“debug”: null,
“httpStatus”: 403,
“messageID”: “IYUCM0008W”,
“messageText”: “IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services.”,
“requestMethod”: “GET”,
“requestUri”: “/zosmf/provisioning/rest/1.0/scr”
}

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I’m not sure about this. Certificate authentication should work, but I needed to specify

–basic –user colin2:password

Python

Fatal (HANDSHAKE_FAILURE): Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

This post said

I got the Insufficient buffer remaining for AEAD cipher fragment error when trying to use the Python client (browser and curl worked normally). After lots of trial and error, it seems the error was related to the SSL certificates not having properly configured extension fields, especially keyUsage and extendedKeyUsage, and the fact that urllib3 doesn’t include /etc/ssl/certs/ca-certificates.crt by default.

Another post said “This is an OpenJDK issue with TLS 1.3 “occurring between the OpenShift default route and Mule applications with re-encypt route enabled.”

Keystore file message is confusing

I saw the following in a GW trace record

[35mZWESVUSR [0;39m [36mINFO [0;39m((o.a.t.u.n.N.certificate)) Connector
[https-jsse-nio-0.0.0.0-7554], TLS virtual host [default], certificate type [UNDEFINED] configured from keystore [/u/tmp/zowec/.keystore] using alias [CONN1.IZUDFLT] with trust store [null]

it was confusing because I was not using a keystore file. This looks like a bug in Spring, and can be ignored.

ZWEAG121E Authorization header is missing, or the request body is missing or invalid for URL ‘/zaas/api/v1/auth/login’

• messageAction: Provide valid authentication.
• messageReason: The authorization header is missing, or the request body is missing or invalid.

I got this when trying to do a certificate login – and no password.

• The certificate used did not have a mapping to a userid.
• The userid was revoked

ZWEAO500E: The service has encountered a situation it doesn’t know how to handle.

Please contact support for further assistance. More details are available in the log under the provided message instance ID.

Case 1

I got this when I changed the a logging value using the REST API. I had

“configuredLevel”: “WARN2″‘

which is invalid.

Case 2

In the log I got

java.lang.ClassCastException: com.ibm.crypto.plus.provider.ECPublicKey incompatible with java.security.interfaces.RSAPublicKey

This is because the key type in my keyring was an Elliptic curve, but the code assumesit is RSA, and uses an RSA function – which does not work with an Elliptic key.

HTTPIE: Https error: ConnectionError: HTTPSConnectionPool(host=…, port=…)

Max retries exceeded with url: … (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’)) while doing a GET request to URL: https://…

The back end was not active so the TLS hand shake failed.

I experienced the messages below when using Zowe. I’ve included additional text to further explain the messages

BPXP005I A FORK OR SPAWN ERROR WAS ENCOUNTERED. RETURN CODE 00000070 REASON CODE 0B250012

Unix return code  00000070 maps to EAGAIN (the resource is temporarily unavailable)

Unix reason code 0B250012 maps to JRMaxChild (the maximum number of processes for this user ID has been exceeded)

Increase the appropriate value MAXPROCSYS or MAXPROCUSER. This can be done dynamically using the SETOMVS command.

Use

• D OMVS,O to display maxprocuser
• setomvs MAXPROCUSER=40

BPXI039I SYSTEM LIMIT SHRLIBRGNSIZE HAS REACHED 90% OF ITS CURRENT CAPACITY OF 83886080

See Tuning the z/OS shared library region.

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -262894

JVMSHRC337E Platform error message: semctl : EDC5111I Permission denied. (errno2=0x070E0303)
JVMSHRC028E Permission Denied
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine

-262894 is SEMCTL EACCESS – Permission denied.

The owner of the shared file cache was not the userid trying to use it.

chown ZWESVUSR:SYS1 *

Changed the files, such as

ZWESVUSR SYS1     314572800 Feb  9 08:43 C290M17F1A64S_zoweGW_G43L00                   
ZWESVUSR SYS1          32 Feb 10 06:35 C290M17F1A64_semaphore_zoweGW_G43L00            
ZWESVUSR SYS1          40 Feb 10 06:35 C290M17F1A64_memory_zoweGW_G43L00               

JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360

JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.

-197360 is SEMGET ENOENT – No such file, directory, or IPC member exists.

I got these trying to restore a sharedclasses cache, when I did not have access to the file.

/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot

I gave the userid access and it worked

chmod 777 /u/tmp/zowec/javasharedresources/*

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC336E Port layer error code = -459502

JVMSHRC337E Platform error message: shmctl : EDC5111I Permission denied.
JVMSHRC028E Permission Denied
JVMSHRC626I The stats of the shared cache cannot be obtained since a valid shared cache does not exist.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed

-459502 is SHMCTL EACCESS -Permission is denied.

The userid issuing the command does not have access to the resource.

The documentation says the shared class cache is created with ONLY USER read/write access by default unless the groupAccess command-line suboption is used, in which case the access is read/write for user and groups.

Note: Users with super user authority gid=0(SYS1) can issue the command with no additional authority.

To find the group list the directories containing the cache, for example if /var/zosmf/data/logs/.classCache/ was specified use ls -ltr /var/zosmf/data/logs/.classCache/javasharedresources.

For me it had owner IZUSVR group IZUADMIN.

I used the RACF command connect COLIN group(IZUADMIN) to connect the userid to the group. Even then the command failed, because groupAccess had not been defined on the -Xshareclasses… parameter. I had to delete the cache so it was recreated next time theJVM started. Then the java -Xshareclasses:cacheDir=/var/zosmf/data/logs/.classCache,name=liberty-IZUSVR,verbose,printStats worked.

JVMSHRC023E   Cache does not exist

I had

-Xshareclasses:cacheDir=/javasc,name=izusvr1cache,printStats

I had to remove the printStats.

JVMSHRC364E SH_OSCachesysv::acquireWriteLock() call to j9shsem_wait on semid … has failed with error -328433.

-328433 is SEMOP – The parameter is incorrect.

You can use the ipcs Unix commands to display the semaphore ids.

JVMSHRC005I No shared class caches available

I was using

/usr/lpp/java/J11.0_64/bin/java -Xshareclasses:cacheDir=/global/zosmf/data/logs/.classCache/,verbose,listAllCaches

to display information about shared cache usage, and kept getting the JVMSHRC005I No shared class caches available message. I experienced two reasons for this.

1. The information in the file, was for last week’s IPL, and the the information in today’s memory was invalid.
2. I was using the wrong level of Java. Once I used the right level of Java it worked!

restoreFromSnapshot

IBMUSER:/u/ibmuser: >cd /u/tmp/zowec
IBMUSER:/u/tmp/zowec: >/usr/lpp/java/J17.0_64/bin/java -Xshareclasses:cacheDir=/u/tmp/zowec,name=zoweGW,restoreFromSnapshot
JVMSHRC020E An error has occurred while opening semaphore
JVMSHRC336E Port layer error code = -197360
JVMSHRC337E Platform error message: semget : EDC5129I No such file or directory.
JVMSHRC727E An error has occurred in creating the new non-persistent shared cache
JVMSHRC808I Compressed references shared cache “zoweGW” is destroyed. Use option -Xnocompressedrefs if you want to destroy a non-com pressed references cache.
JVMSHRC699E Failed to restore the non-persistent shared cache “zoweGW” from the snapshot

This may be connected to the the following

The following files were in the directory

-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_semaphore_zoweGW_G43L00
-rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_memory_zoweGW_G43L00

For example the above files were had owner: ZWESVUSR group: ZWEADMIN.

The userid was in group ZWEADMIN, and so does not get R/W access to the files.

Errno2

• 0x071D0303: JRIpcDenied: Access was denied because the caller does not have the correct permission.
• 0x053b006c: JRFileNotThere: The requested file does not exist
• 0x0594003d: JRDirNotFound: A directory in the pathname was not found