Zowe: Problem determination

This is a work in progress as I learn more about debugging Zowe

Startup and processing the zowe.yaml file.

Troubleshooting Zowe Launcher. Creates a trace of the parsing of the Zowe.yaml file.

It is verbose;only use when necessary.

For example

It displays the contents of the zowe.yaml file

MapStart                                                 
zowe:
MapStart
runtimeDirectory:
Scalar: (len=13)/u/tmp/zowep/
logDirectory:
Scalar: (len=17)/u/tmp/zowec/logs
workspaceDirectory:
Scalar: (len=22)/u/tmp/zowec/workspace
extensionDirectory:
Scalar: (len=23)/u/tmp/zowec/extensions

and

path:                                                             
validate JSON value->type=4 specTypeMask=0x4
typeCode=2 shifted=0x4 mask=0x4
validateJSONObject required=[]
accessPath (top is blank):
validate object pname=zowe

Component loggers

I found this doc, I haven’t used it

Customizing Zowe API Mediation Layer logging for Specifies one of the following services: zaas, gateway, discovery, api-catalog, caching-service.

ZSS https trace

In the zowe.yaml

zss: 
enabled: true
port: 7557
crossMemoryServerName: ZWESIS_STD
agent:
jwt:
fallback: true
64bit: true
https:
trace: true

produces a trace in the logs directory

zssServer-2025-05-29-08-39.log.tlstrace

use gsktrace zssServer-2025-05-29-08-39.log.tlstrace >gsktrace.txt

What are my Java options?

See Specifying Java overrides

Zowe tracing.

See Enabling tracing for app-server and zss.

Zowe: Specifying Java overrides

As part of my configuring Zowe, and using Java shared classes, I’ve needed to change the Java parameters in Zowe. I know there is a plan to provide this support – but this is what I’ve done.

Find the start up script.

If you use the Unix command

find zowe/components -name startup.sh

This will list the scripts for the components.

I edited zowe/components/gateway/bin/start.sh and inserted at the top

 COL="/u/tmp/zowec/" 

TAG="GW"
fn="${COL}/java.options.${TAG}"
if [ -f "${fn}" ]; then
set -x
export
exec 1>>${COL}start.log1${TAG} 2>>${COL}start.log2${TAG}
COLIN="-Xoptionsfile=${fn} -XshowSettings "
else
COLIN=""
fi

and at the bottom of the file

    -Dloader.path=${GATEWAY_LOADER_PATH} \ 
${COLIN} \
-jar ${JAR_FILE} &

This code

  • checks for a file java.options.GW in the Zowe instance directory.
  • If this file exists
    • Create log files in the directory
    • Create an override variable (COLIN), specifying the file name, and telling Java to print out the settings.
  • If it doesn’t exist, set the variable (COLIN) to blank
  • When the Java command is issued, substitute the variable COLIN – which is either blank, or the options file and display Java options command.

Create the java options file.

You can use

touch java.options.GW

or edit the the and add content.

The output

It produced output like

VM settings: 
Min. Heap Size: 32.00M
Max. Heap Size: 512.00M
Using VM: IBM J9 VM

Property settings:
XcompilationThreads3 =
Xhealthcenter =
Xjit:count = 0
...
JRE 21 z/OS s390x-64-Bit Compressed References 20241024_16 (JIT enabled, AOT enabled)
OpenJ9 - f45de8e9eb0
OMR - 55ddfd47ab0
IBM - 3c87141
JCL - df334d2be4a based on jdk-21.0.4+7
java.home = /Z31B/usr/lpp/java/J21.0_64
java.io.tmpdir = /tmp
....
xcom.ibm.java.diagnostics.healthcenter.headless.output.directory = /u/tmp/zowec

Locale settings:
default locale = English (United States)
default display locale = English (United States)
...
Security settings summary:
See "java -X" for verbose security settings options
Security provider static configuration: (in order of preference)
Provider name: OpenJCEPlus
Provider name: IBMZSecurity
Provider name: SUN
Provider name: SunRsaSign
...
May 29, 2025 9:30:25 AM sun.security.ssl.SSLLogger log
WARNING: No AlgorithmParameters for sect163k1
May 29, 2025 9:30:27 AM sun.security.ssl.SSLLogger log
WARNING: No AlgorithmParameters for sect163r1
...
Security TLS configuration (SunJSSE provider):
Enabled Protocols:
TLSv1.3
TLSv1.2

Something for the weekend. Which is heavier and ounce of gold or an ounce of paper?

I remember being asked this question at my grandfather’s knee. Of course the popular answer, which is wrong, is they both weight the same. Yes, they have different weights, an ounce of gold is heavier than an ounce of paper.

An ounce of gold weights 31 grams, and an ounce of paper weight 28 grams. Gold is measured in ounces troy, and most other things, including paper, are measured in ounces avoirdupois.

A similar question is why do cars in Britain have a better miles per gallon than in the US – it is not because Americans have bigger cars. A US gallon is 3.7 litres and a UK gallon is 4.5 litres, Americans have smaller litres.

Jump up and down: Do not give userids access to resources!

I am doing some work connecting subsystems together. The documentation for all the products involved, describe giving a userid permission to access to a resource.

This is not best practice. Like many things it will be obvious once you understand it. You need to look at the bigger pictures.
The documentation says for userids who want to use “this”, give them access to the profile “…”. What’s wrong with that?

  • If you have a department of 1000 people, giving them all access to the resource will be tedious
  • There are likely to be several resources people need access to, so connecting these 1000 userids to multiple resources will be even more tedious.
  • Someone joins your department – so you need to connect their userid to the long list of groups.
  • Someone leaves your department – you cannot trivially ask what resources can this userid access – you have to look at the access list for each resource and remove the userid from the group.

You most probably have groups set up already. Rather than give the userid access, give the group access to the resource.

If someone joins your group – you connect their userid to the groups – and they have access. If someone leaves your department, remove them from the groups – and they no longer have access to the resources.

You my want to support a new product… That’s easy – give the group(s) access to the resources – and the people will auto-magically get access.

As I said it is obvious once you see it.

Do not give userids access to resources – give groups access, and connect the userid to the group. I’ll go and raise documentation comments on the products’ documentation.

What are good tools for working with a backend server?

I’ve used a web browser, cURL, openssl client, and Python for getting to a back end REST server. I’ve recently discovered HTTPie. For example

https GET https://127.0.0.1:7554/application/loggers –cert colinpaice.pem –cert-key colinpaice.key.pem –verify ca.pem2

This invokes some Python code which does all the work. It produced

The documentation is here.

Keep a session

By default each request is a single shot, nothing is retained (such as a Java Web Token) so you need to enter your credentials every time.

However, HTTPie also supports persistent sessions via the --session=SESSION_NAME_OR_PATH option. In a session, custom HTTP headers (except for the ones starting with Content- or If-), authentication, and cookies (manually specified or sent by the server) persist between requests to the same host.

Named sessions

You can create one or more named session per host. For example, this is how you can create a new session named user1

http --session=user1 -a user1:password .... 

From now on, you can refer to the session by its name (user1). When you choose to use the session again, all previously specified authentication or HTTP headers will automatically be set.

Plugins

There are plugins available see here. For example authentication

Not for humans – AT-TLS and security messages

EZD1287I TTLS Error RC: 6 Initial Handshake

6 Key label is not found

My server certificate was in the keyring, but it had expired. I renewed it, and got past this.

EZD1287I TTLS 403 No certificate received from partner.

403 No certificate received from partner.

In my curl request I did not have –cert ./colinpaice.pem:password –key ./colinpaice.key.pem, when the server expected a certificate (gpmserve had CLIENT_CERT(ACCEPT)).

My z/OS had HandshakeRole ServerWithClientAuth, but the client did not provide a certificate.

EZD1287I TTLS Error RC: 428 Initial Handshake

The private key cannot be obtained from the certificate.

The server userid needs access to the keyring. If the private key belongs to the server’s userid, then the server’s userid needs read access to the keyring. If the private key belongs to a different userid, the server’s userid needs update access to the keyring. See here for more information.

EZD1287I TTLS Error RC: 435 Initial Handshake

435 Certification authority is unknown.

I got this having replaced the CA certificate. Deleting a certificate removes it from any keyring. When you recreate the CA, you need to add it to every keyring it was in. Before deleting a certificate it is worth listing it to see where it is used. I added it to my keyring and it worked!

EZD1287I TTLS Error RC: 510 No acceptable key labels found

510 No acceptable key labels found

My server certificate was in the keyring, but it did not have the capabilities to support the handshake. For example client only supported RSA, but server was ECC.

pthread_security_np, pthread_security_app_np

This function call can be used to map a certificate to a userid.

I got

ESRCH (143): The user ID provided as input is not defined to the security product or does not have an OMVS segment defined.

errno2 0be8044c. Code 044c is JRNoCertforUser, There is no userid defined for this certificate. Action: Ensure the userid is known to the SAF service.


You also get this message if the mapping from certificate to userid is missing. For example

RACDCERT DELMAP(LABEL('IBMUSER1Label))ID(IBMUSER)
RACDCERT MAP ID(IBMUSER) -
WITHLABEL('IBMUSER1Label') -
SDNFILTER('CN=colinpaice.O=cpwebuser.C=GB')
RACDCERT LISTMAP ID(IBMUSER)
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

R_ticketserv (IRRSPK00): Parse or extract 8, 16, 28

I was using the service to generate a pass token, and got SAF 8 RACF 16 RS 28.

Unable to generate PassTicket. Verify that the secured signon (PassTicket) function and
application ID is configured properly by referring to Using PassTickets in z/OS Security Server RACF Security Administrator’s Guide.

Action

I needed to define

RALTER  PTKTDATA MYAPPL  SSIGNON(KEYMASKED(7E4304D681920260)) - 
APPLDATA('NO REPLAY PROTECTION')

Where MYAPPL is the application name.

R_ticketserv (IRRSPK00): Parse or extract 8, 8, 16

With userid = “ADCBD” and APPL = ” MVSS0W1″ I got SAF 0 RACF 0 RS 0.

With userid = “ADCBD” and APPL = ” TSOS0W1″ I got SAF 8 RACF 8 RS 16. Not authorized to use this service.
I used

RDEFINE PTKTDATA   IRRPTAUTH.TSOS0W1.*  UACC(NONE) 

and it worked

EZD1287I TTLS Error RC: 467 Initial Handshake

Running from Python – a request worked. Running from Chrome gave me the above message.
I solved it by making the connection TLS 1.3 ON and TLS 1.2 OFF

The doc (Cryptographic Services System Secure Sockets Layer Programming)

says

467 Signature algorithm not in signature algorithm pairs list.
Explanation
A signature algorithm that is used to sign a local or peer certificate is not included in the signature algorithm pairs list or the certificate signature algorithm pairs list. The server certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the client during the TLS handshake. The client certificate chain must use signature algorithms included in the signature algorithm pairs or the certificate signature algorithm pairs that are presented by the server during the TLS handshake. If remote partner specifies the signature algorithm pairs and the certificate signature algorithm pairs, the specified certificate signature algorithm pairs take precedence over the signature algorithm pairs. For TLS V1.2 handshakes, the GSK_TLS_CERT_SIG_ALG_PAIRS setting is only used on the client
side to indicate the signature algorithms that the client supports in the server’s certificate chain.
User response
Ensure that the signatures of the local and peer certificates in the certificate chain use signature algorithms that are present in the signature algorithm pairs list or the certificate signature algorithm pairs list that is presented by the session partner. If the certificate chain is correct, configure the client or server or both to specify all necessary signature algorithms pairs in the GSK_TLS_SIG_ALG_PAIRS or SK_TLS_CERT_SIG_ALG_PAIRS
settings to allow use of the certificate chain. If GSK_TLS_CERT_SIG_ALG_PAIRS is specified, it takes precedence while checking the signature algorithms used in the certificate chain.

Zowe: setup: configuring end user userids

Connect the userid to the right groups.

It is better to give permissions to groups, rather than to individual userids.

If someone changes jobs, within your organisation, you just remove the userid of the person leaving from the group, and they lose all access.

If you have given access to a userid, you have to go through all resources (for example APPLs) and remove access.

When a new person joins, you just connect them to the group, and they should have access.

Zowe groups

If you are going to use Zowe, then user userid needs access to the Zowe group.

  • If class(APPL) OMVSAPPL is defined, the Zowe group needs read access to it
  • If users will be using z/OMSF they will need read access to the class(APPL) IZUDFLT (or what ever is specified)
  • If certificate logon is being used, the certificate will need to map to a userid

Access z/OSMF

If userid will be using z/OSMF they need access to the z/OSMF group.

In the z/OSMF logs, I got

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

I connected userid COLIN2 to the z/OSMF group IZUUSER. I think the change is available immediately.

Access to OMVSAPPL

An application can switch userid by using the pthread_security_np, and passing in a userid and password or a certificate (which maps to a userid).

The application can control which userid can be used by means of an resource in the APPL class.

If the resource was called MYAPPL, then pthread_security_np would use “MYAPPL”, and the userid being switched to much gave read access to MYAPPL.

The default (and only supported) resource is called OMVSAPPL.

If OMVSAPPL in class(APPL) is defined, the userid must have read access to it. z/OSMF also uses OMVSAPPL.
If OMVSAPPL in class(APPL) is not defined, no checks are done.

With Zowe you cannot specify the resource name.

Certificate logon

You can use a digital certificate from a web browser ( curl, or other tools) to authenticate to z/OS.  You need to map the certificate to a userid.

A certificate coming in can have a Distinguished Name like CN=adcdd.O=cpwebuser.C=GB  (Note the ‘.’not ‘,’ between elements).

Your userid needs to have SPECIAL define to be able to use the RACDCERT command (SPECIAL, not just GROUP-SPECIAL).

You will need a definition like (see here for the command)

RACDCERT MAP ID(ADCDD ) - 
    SDNFILTER('CN=adcdd.O=cpwebuser.C=GB') - 
    WITHLABEL('adcdd')

or a general definition for those certificate with  O=cpwebuser.C=GB, ignoring the CN part

RACDCERT MAP ID(ADCDB ) - 
   SDNFILTER('O=cpwebuser.C=GB') - 
   WITHLABEL('cpwerbusergb') 

or using the Issuing Distinguished Name (the Certificate Authority)

IDNFILTER(‘CN=TESTCA.OU=SSSCA.C=GB)

Using a generic

SDNFILTER(‘CN=a*.O=cpwebuser.C=GB’)

does not work.

If you attempt to use a certificate which is not mapped you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(COLIN)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0163)  SUBJECT(CN=adcdd.O=cpwebuser.C=GB) ISSUER(CN=SSCA8.OU=CA.O=SSS.C=GB).

It is worth defining these using JCL, because if you try to add it, and it already exists then you get a message saying it exists already.  If you know the userid, you can list the maps associated with it.   If you do not know the userid, there is no practical way of finding out – you have to logon with the certificate, and display the userid from the web browser, or extract the list of all users, and use LISTMAP on all of them.

Zowe: Planning: How do I protect what Zowe users can do on z/OS?

As part of my planning for Zowe, I wanted to know how I could control what Zowe users can do on z/OS. I could not find any definitions for security profiles, so how do I do it?

It took a few days thinking about this to realise I was looking a the problem the wrong way. The correct way of looking at it, is that Zowe is a transport system for getting requests from a user’s work station to z/OS. This is similar to a 3270 emulator connection to z/OS. You control what the userid can do, and do not try to control what the 3270 emulator can do.

When a userid logs on to TSO through a 3270 emulator, z/OS knows the userid of the address space, and can control access to what resources the userid can access.

When a userid logs on to Zowe there are two paths that can be taken:

  • Zowe can create a TSO address space for the userid, using the CEA facilities. The z/OS® CEA TSO/E address space manager provides services to programmatically start and manage TSO/E address spaces and provides a communications mechanism for use between the caller and the programs running in these managed address spaces.
  • A thread within Zowe can use the pthread_security and change the userid of that thread. It can use your certificate, or a userid and password to validate the user. At the end of the request it resets the userid back.

From a permissions perspective, it does not matter if request came into a TSO address space or as a result of the pthread_security request. The userid is extracted and normal SAF processing is used to manage access to a resource.

But…

Your system may have IPSEC rules which police traffic into and out of TCP/IP on z/OS, for example allow traffic from these external IP address during 0900 to 1700 Monday to Friday, and deny access at any other time.

You might need to have similar rules for connectivity through Zowe. There are several ports used by Zowe and z/OSMF. You need to review what controls you need for these ports to stop unwanted traffic from accessing your system.

Zowe: Which messages to automate

In the Zowe.yaml file is a section sysMessages. This is a list of the messages that are displayed on syslog. You can add messages to these.

The provided list is

  • ZWEAM000I %s started in %s seconds, giving the component and the duration of the startup
  • ZWED0031I The appServer is ready at ipAddress, Plugins successfully loaded: percentage% (successful/total)
  • ZWEL0001I component %s started. A start request was issued for the component
  • ZWEL0002I component %s stopped. A request was made to stop the component
  • ZWEL0006I starting components. A request was made to start the component specified as enable: true in the zowe.yaml file.
  • ZWEL0008I stopping components. A request was made to stop Zowe, and Zowe is shutting down.
  • ZWEL0018I Zowe instance prepared successfully. The pre-starting checks have been completed. This message is followed by ZWEL0006I starting components.
  • ZWEL0021I component %s stopped
  • ZWEL0022I Zowe Launcher stopped. This is the last message that Zowe emits.
  • ZWES1013I ZSS Server has started. Version ‘%s’ ‘%s’
  • ZWES1601I ZSS Server is ready to accept JWT with|without fallback to legacy tokens

Zowe: messages on z/OS when using Zowe

ZWEAO503E The server is not ready to handle the request: /apicatalog/api/v1/

I got this when the ZAAS service was stopped, use the f zowe,appl=DISPLAY to display the status of the various tasks.

You can use f zowe,appl=start(ZAAS) to start it. Check the zowe sysprint data set for any error messages.

I also got the following message in the z/OSMF logs

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user adcdb has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

BPXTLS failed: rc=-1, return code=143, reason code=0x0be80000


Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required
Other platforms: impersonation is not supported

This is actually from BPX1TLS, phread_security_np. There are not many cases where 0be80000 is mentioned. This link mentions PTKTDATA profiles.

LoadBalancer does not contain an instance for the service zaas

This was a tricky one to track down.

Zowe works with z/OSMF. They communicate with certificates. This means the z/OSMF trust store keyring needs the CA of the Zowe server certificate, and the Zowe trust store keyring needs the CA of the z/OSMF server key.

Not only that , if verifyCertificates STRICT is specified in the zowe.yaml file, then Zowe will check the server’s certificate sent from z.OSMF. This includes the altname or altIP, the altIP is the IP address of the connection. An internal IP address is 127.0.0.0. An external IP address is almost anything else. On my system I have addresses 127.0.0.1 and 10.1.1.2 and 10.1.2.6. You can display them using the TSO NETSTAT HOME command.

The zOSMF certificate did not have an altip specified, and so failed the Zowe checks. I had to set the Zowe option verifyCertificates NOSTRICT for it to work.

You can also get this message when the system is partially up.

ZWEAM701E; InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name

The request to the URL ‘/zaas/api/v1/auth/login’ has failed: SignatureException: Unable to compute RS256 signature. Cause: Unrecognized RSA or RSASSA-PSS key algorithm name. caused by: InvalidKeyException: Unrecognized RSA or RSASSA-PSS key algorithm name. messageAction:Refer to specific exception details for troubleshooting.

It looks you Zowe server needs an RSA key for use with JWT. I changed from an Elliptic key to an RSA and it worked.

Using the web browser

403 Impersonation error

  • Error: response status is 403
  • Download
  • Impersonation error

In the logs/zssServer….log it has

BPXTLS failed: rc=-1, return code=139, reason code=0x0be803d1
Error: service has no impersonation; make sure process user has sufficient authority:
z/OS: program control flag must be set, UPDATE access to BPX.SERVER and BPX.DAEMON SAF resources is required

This is because Zowe does not have access to the profiles.

Action:
Check the profiles are defined, and the Zowe userid has UPDATE access to them.

Header X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I actually got the message ZWEAG160E No authentication provided in the request 5 times.

The header is collecting all errors during the treatment of the request. Because the response was 503 the Gateway retries it 5 times. This is strange, but expected behaviour.

I also got an error message in the z/OSMF log /global/zosmf/data/logs/zosmfServer/logs/trace.log

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

Give the userid access to the profile. It may be easier to connect the userid to the group. In my case to group IZUUSER.

I gave the userid access and then got

IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request

{
“additionalInfo”: null,
“debug”: null,
“httpStatus”: 403,
“messageID”: “IYUCM0008W”,
“messageText”: “IYUCM0008W: The user ID colin is not authorized to resource type Navigation Task, resource name Software Services.”,
“requestMethod”: “GET”,
“requestUri”: “/zosmf/provisioning/rest/1.0/scr”
}

X-Zowe-Auth-Failure: ZWEAG160E No authentication provided in the request.

I’m not sure about this. Certificate authentication should work, but I needed to specify

–basic –user colin2:password

Python

Fatal (HANDSHAKE_FAILURE): Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

This post said

I got the Insufficient buffer remaining for AEAD cipher fragment error when trying to use the Python client (browser and curl worked normally). After lots of trial and error, it seems the error was related to the SSL certificates not having properly configured extension fields, especially keyUsage and extendedKeyUsage, and the fact that urllib3 doesn’t include /etc/ssl/certs/ca-certificates.crt by default.

Another post said “This is an OpenJDK issue with TLS 1.3 “occurring between the OpenShift default route and Mule applications with re-encypt route enabled.”

Keystore file message is confusing

I saw the following in a GW trace record

[35mZWESVUSR [0;39m [36mINFO [0;39m((o.a.t.u.n.N.certificate)) Connector
[https-jsse-nio-0.0.0.0-7554], TLS virtual host [default], certificate type [UNDEFINED] configured from keystore [/u/tmp/zowec/.keystore] using alias [CONN1.IZUDFLT] with trust store [null]

it was confusing because I was not using a keystore file. This looks like a bug in Spring, and can be ignored.

ZWEAG121E Authorization header is missing, or the request body is missing or invalid for URL ‘/zaas/api/v1/auth/login’

  • messageAction: Provide valid authentication.
  • messageReason: The authorization header is missing, or the request body is missing or invalid.

I got this when trying to do a certificate login – and no password.

  • The certificate used did not have a mapping to a userid.
  • The userid was revoked

ZWEAO500E: The service has encountered a situation it doesn’t know how to handle.

Please contact support for further assistance. More details are available in the log under the provided message instance ID.

Case 1

I got this when I changed the a logging value using the REST API. I had

“configuredLevel”: “WARN2″‘

which is invalid.

Case 2

In the log I got

java.lang.ClassCastException: com.ibm.crypto.plus.provider.ECPublicKey incompatible with java.security.interfaces.RSAPublicKey

This is because the key type in my keyring was an Elliptic curve, but the code assumesit is RSA, and uses an RSA function – which does not work with an Elliptic key.

HTTPIE: Https error: ConnectionError: HTTPSConnectionPool(host=…, port=…)

Max retries exceeded with url: … (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’)) while doing a GET request to URL: https://…

The back end was not active so the TLS hand shake failed.