Category: Trace

Zowe cli: Tracing the data

I spent a long time trying to find what data from a Zowe CLI commands was being sent to the backend. I did traces on the back end – but they did not show me the data. I eventually found traces for the work station end.

Trace the tls setup

export NODE_DEBUG='tls,https'

The tls trace gave me

TLS 18487: client _init handle? true
TLS 18487: client initRead handle? true buffered? false
TLS 18487: client _start handle? true connecting? false requestOCSP? false
TLS 18487: client emit session
TLS 18487: client onhandshakedone
TLS 18487: client _finishInit handle? true alpn false servername false
TLS 18487: client emit secureConnect. rejectUnauthorized: false, authorizationError: SELF_SIGNED_CERT_IN_CHAIN

The https trace gave me the output below. Certificates were used, and these were in the traced data

HTTPS 18509: createConnection [Object: null prototype] {
  headers: {
    'Content-Type': 'application/json',
    'ibm-mq-rest-csrf-token': 'true'
  },
  hostname: '10.1.1.2',
  method: 'POST',
  path: null,
  port: 9443,
  rejectUnauthorized: false,
  timeout: 5000,
  cert: <Buffer ... 4021 more bytes>,
  key: <Buffer 2d  ... 1654 more bytes>,
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] {
      keepAlive: true,
      scheduling: 'lifo',
      timeout: 5000,
      noDelay: true,
      path: null
    },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      '10.1.1.2:9443:::Certificate:\n
    Data:\n
        Version: 3 (0x2)\n        Serial Number: 683 (0x2ab)\n
        Signature Algorithm: ecdsa-with-SHA256
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----\n
:::-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----\n::false:::::::::::::': []
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: true,
...
  },
  host: '10.1.1.2',
  keepAlive: true,
  scheduling: 'lifo',
  noDelay: true,
  servername: '',
  _agentKey: '10.1.1.2:9443:::Certificate:\n' +
    '    Data:\n' +
    '        Version: 3 (0x2)\n' +
    '        Serial Number: 683 (0x2ab)\n' +
    '        Signature Algorithm: ecdsa-with-SHA256\n'
  ...

    '-----BEGIN CERTIFICATE-----\n' +
  ...
    '-----END CERTIFICATE-----\n' +
    ':::-----BEGIN PRIVATE KEY-----\n' +
    ...
    '-----END PRIVATE KEY-----\n' +
    '::false:::::::::::::',
  encoding: null,
  keepAliveInitialDelay: 1000
}

Note: it traces setting up the handshake, it does not trace the application data flow.

Trace the application data

The section above shows the data for the TLS handshake obtained with NODE_DEBUG
To get the application data you need the Zowe CLI trace.

See Setting Cli Trace levels.

ZOWE_APP_LOG_LEVELZowe CLI logging levelLog4JS log levels (OFF, TRACE, DEBUG, INFO, WARN, ERROR, FATAL)WARN
ZOWE_IMPERATIVE_LOG_LEVELImperative CLI Framework logging levelLog4JS log levels (OFF, TRACE, DEBUG, INFO, WARN, ERROR, FATAL)WARN

The output is in the directory pointed to by ZOWE_CLI_HOME – which defaults to ~/.zowe/logs on Linux.

ZOWE_APP_LOG_LEVEL

The ZOWE_APP_LOG_LEVEL trace is boring. It is in home/colinpaice/.zowe/logs/zowe.log

[2025/08/02 15:05:37.754] [TRACE] [main.js:40] Init was successful
[2025/08/02 15:05:37.837] [DEBUG] [MQSessionUtils.js:22] Creating an MQ session from arguments
[2025/08/02 15:05:37.845] [INFO] [ConfigAutoStore.js:135] Skipping update of profile properties. Check that config file exists and autoStore is true.

ZOWE_IMPERATIVE_LOG_LEVEL

The ZOWE_IMPERATIVE_LOG_LEVEL=debug trace in logs/imperative.log has

[2025/08/02 15:12:02.162] [TRACE] [AppSettings.js:39] Attempting to load settings file: /home/colinpaice/tmp/settings/imperative.json
[2025/08/02 15:12:02.163] [TRACE] [AppSettings.js:56] Settings were loaded
[2025/08/02 15:12:02.163] [TRACE] [AppSettings.js:57] Loaded Settings:
[2025/08/02 15:12:02.163] [TRACE] [AppSettings.js:58] { overrides: { CredentialManager: '@zowe/cli' } }
[2025/08/02 15:12:02.163] [DEBUG] [ConfigManagementFacility.js:55] ConfigManagementFacility.init() - Start
[2025/08/02 15:12:02.163] [DEBUG] [UpdateImpConfig.js:38] Adding definition = 'config'
[2025/08/02 15:12:02.162] [TRACE] [AppSettings.js:39] Attempting to load settings file: /home/colinpaice/tmp/settings/imperative.json
[2025/08/02 15:12:02.163] [TRACE] [AppSettings.js:56] Settings were loaded
[2025/08/02 15:12:02.163] [TRACE] [AppSettings.js:57] Loaded Settings:
...

Lots of profile information and the schema…

The command entered

[2025/08/02 15:12:02.410] [DEBUG] [CommandYargs.js:75] Defining command: mqsc
[2025/08/02 15:12:02.411] [DEBUG] [CommandYargs.js:174] Building positional string from: mqsc
[2025/08/02 15:12:02.412] [DEBUG] [CommandYargs.js:180] Positional String:  [qmgr] [cmd]
[2025/08/02 15:12:02.415] [DEBUG] [CommandYargs.js:94] Defining command builder for: mqsc
[2025/08/02 15:12:02.421] [DEBUG] [CommandYargs.js:105] Handler invoked for: mqsc
[2025/08/02 15:12:02.422] [DEBUG] [CommandYargs.js:118] Executing Handlers: mqsc
[2025/08/02 15:12:02.423] [DEBUG] [CommandYargs.js:136] Executing Handlers (1 total)
[2025/08/02 15:12:02.433] [INFO] [CommandProcessor.js:254] Invoking command "mqsc"...
[2025/08/02 15:12:02.439] [INFO] [CommandProcessor.js:255] Command issued:

zowe --mq-p mq --cert-key-file ./colinpaice.key.pem --cert-file ./colinpaice.pem --host 10.1.1.2 --port 9443 mq run mqsc CSQ9 DIS QMGR ALL


[2025/08/02 15:12:02.439] [TRACE] [CommandProcessor.js:256] Invoke parameters:
{
  arguments: {
    _: [ 'mq', 'run', 'mqsc' ],
    'mq-p': 'mq',
    'mq-profile': 'mq',
    mqP: 'mq',
    mqProfile: 'mq',
    'cert-key-file': './colinpaice.key.pem',
    certKeyFile: './colinpaice.key.pem',
    'cert-file': './colinpaice.pem',
    certFile: './colinpaice.pem',
    host: '10.1.1.2',
    H: '10.1.1.2',
    port: '9443',
    P: '9443',
    version: undefined,
    V: undefined,
    'available-commands': undefined,
    ac: undefined,
    availableCommands: undefined,
    'response-format-json': undefined,
    rfj: undefined,
    responseFormatJson: undefined,
    help: undefined,
    h: undefined,
    'help-web': undefined,
    hw: undefined,
    helpWeb: undefined,
    'help-examples': undefined,
    helpExamples: undefined,
    'reject-unauthorized': undefined,
    ru: undefined,
    rejectUnauthorized: undefined,
    'show-inputs-only': undefined,
    showInputsOnly: undefined,
    '$0': 'zowe',
    qmgr: 'CSQ9',
    cmd: 'DIS QMGR ALL'
  },
  silent: false,
  responseFormat: 'default'
}

More stuff

Send the request


[2025/08/02 15:12:02.499] [INFO] [AbstractRestClient.js:339] Setting socket connection timeout ms: 60000
[2025/08/02 15:12:02.500] [TRACE] [AbstractRestClient.js:538] Using PEM Certificate authentication
[2025/08/02 15:12:02.501] [TRACE] [AbstractRestClient.js:807] appendInputHeaders called with options on rest client {"headers":{},"hostname":"10.1.1.2","method":"POST","path":"/ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc","port":9443,"rejectUnauthorized":false,
"timeout":60000,
"cert":{"type":"Buffer","data":[67,101,114,...,10]},
"key":{"type":"Buffer","data":[45,45,45,45,45,66,69,71,73,78,32,80,82,...5,10]}
}

MQRestClient
[2025/08/02 15:12:02.501] [TRACE] [AbstractRestClient.js:440] Rest request: POST 10.1.1.2:9443/ibmmq/rest/v1/admin/action/qmgr/CSQ9/mqsc 
[2025/08/02 15:12:02.541] [DEBUG] [AbstractRestClient.js:173] will write data for request
[2025/08/02 15:12:02.542] [DEBUG] [AbstractRestClient.js:178] writing JSON for request
[2025/08/02 15:12:02.542] [TRACE] [AbstractRestClient.js:179] JSON body: {"type":"runCommand","parameters":{"command":"DIS QMGR ALL"}}
[2025/08/02 15:12:02.916] [DEBUG] [AbstractRestClient.js:575] Content length of response is: 3299
[2025/08/02 15:12:02.919] [TRACE] [AbstractRestClient.js:626] Data chunk received...
[2025/08/02 15:12:02.924] [DEBUG] [AbstractRestClient.js:668] onEnd() called for rest client MQRestClient
[2025/08/02 15:12:02.934] [INFO] [CommandProcessor.js:476] Handler for command "mqsc" succeeded.
[2025/08/02 15:12:02.937] [INFO] [CommandProcessor.js:824] Command "mqsc" completed with success flag: "true"
[2025/08/02 15:12:02.938] [TRACE] [CommandProcessor.js:825] Command "mqsc" finished.

and finally the response


{
  success: true,
  exitCode: 0,
  message: '',
  stdout: <Buffer 52 75 0a ... 1915 more bytes>,
  stderr: <Buffer >,
  data: {
    commandResponse: [
      {
        completionCode: 0,
        reasonCode: 0,
        text: [
          'CSQN205I   COUNT=       3, RETURN=00000000, REASON=00000000',
          'CSQM409I %CSQ9 QMNAME(CSQ9                                            ) DESCR(CSQ9, IBM MQ for z/OS - V9.0.1                                  ) 
...
          "CSQ9022I %CSQ9 CSQMDRTS ' DIS QMGR' NORMAL COMPLETION"
        ]
      }
    ],
    overallReasonCode: 0,
    overallCompletionCode: 0
  },
  error: undefined
}

Zowe: Changing trace on z/OS

Trace within Zowe is a bit confusing. Different components, such as Gateway and Application-Server seems to have different ways of tracing.

As I understand it, you set a global switch for a components trace, restart Zowe, then use a REST API to set or reset subcomponents.

It took me a day or so to be able to change the trace of Zowe components!

There is Zowe documentation on Troubleshooting Zowe API Mediation Layer. Which is incomplete, and missing useful documentation.

The syntax of the logging commands is here

List the existing traces

The documentation says 

GET scheme://hostname:port/application/loggers

This didn’t work for me because I was using port 7554. After I tried all of the Zowe ports, the following worked

"https://10.1.1.2:7558/application/loggers"

Where 7558 is the port for zaas.

This returned one long string, with many logical lines of output. For example:

{
    "levels": [
        "OFF",
        "ERROR",
        "WARN",
        "INFO",
        "DEBUG",
        "TRACE"
    ],
    "loggers": {
        "ROOT": {
            "configuredLevel": "INFO",
            "effectiveLevel": "INFO"
        },
        "_org": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "_org.springframework": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
...
       
    },
    "groups": {
        "web": {
            "configuredLevel": null,
            "members": [
                "org.springframework.core.codec",
                "org.springframework.http",
                "org.springframework.web",
                "org.springframework.boot.actuate.endpoint.web",
                "org.springframework.boot.web.servlet.ServletContextInitializerBeans"
            ]
        },
        "sql": {
            "configuredLevel": null,
            "members": [
                "org.springframework.jdbc.core",
                "org.hibernate.SQL",
                "org.jooq.tools.LoggerListener"
            ]
        }
    }
}
  • 434 lines for org.springframework….
  • 36 lines for com.netflix….
  • 286 line for io….
  • 63 lines for org.apache…
  • 57 lines for org.hibernate
  • 68 lines for org.ehcache
  • 109 lines for org.zowe.apiml.

Selecting one record type

I used a Bash script

#!/bin/bash 
url="https://10.1.1.2:7558/application/loggers/org.zowe.apiml.zaas.security"
certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
https GET ${url} $certs  $verify

gave

{
    "configuredLevel": "null",
    "effectiveLevel": "DEBUG"
}

This means org.zowe.apiml.zaas.security was configured as null, but one of org.zowe, org.zowe.apiml, or org.zowe.apiml.zaas was set to DEBUG, and all of the definitions below it were set to DEBUG.

If I set the value of org.zowe.apiml.zaas.security to WARN, then set the value of org.zowe.apiml.zaas to DEBUG, the value of org.zowe.apiml.zaas.security stayed at WARN.

{
    "configuredLevel": "WARN",
    "effectiveLevel": "WARN"
}

Updating a value

It took me a while to get a script to update the value.

url="https://10.1.1.2:7558/application/loggers/org.zowe.apiml.zaas.security"
certs="--cert colinpaice.pem --cert-key colinpaice.key.pem"
verify="--verify no"
echo -n '{"configuredLevel": "WARN"}'  |  https POST ${url}  $certs  $verify
echo "================="
https GET ${url} $certs  $verify

This pipes the update {“configuredLevel”: “WARN”} into the https command. This returned no data. If it fails it may return a message.

Changing the hierarchy

Logically changing a value for a.b.c to ZZZZ works as follows

for every element under a.b.c
   if the element's value is null 
   then set element's value = ZZZZ
   else do nothing