System SSL error codes

When using System SSL, I got a return code which I could not find.

I eventually found in the documentation

SSL error codes are defined internally to be ‘134CCxxx’, where xxx is the hexadecimal representation of the SSL error code. The range is ‘134CC001’ through ‘134CCFFF’. For example, SSL error code 414 would be ‘134CC19E’.

Which may be accurate – but is not very helpful.

The data below is the value in hex, in decimal, and the description

0x134CC000 323796992 GSK_OK
0x134CC001 323796993 GSK_INVALID_HANDLE
0x134CC002 323796994 GSK_API_NOT_AVAILABLE
0x134CC003 323796995 GSK_INTERNAL_ERROR
0x134CC004 323796996 GSK_INSUFFICIENT_STORAGE
0x134CC005 323796997 GSK_INVALID_STATE
0x134CC006 323796998 GSK_KEY_LABEL_NOT_FOUND
0x134CC007 323796999 GSK_CERTIFICATE_NOT_AVAILABLE
0x134CC008 323797000 GSK_ERR_CERT_VALIDATION
0x134CC009 323797001 GSK_ERR_CRYPTO
0x134CC00A 323797002 GSK_ERR_ASN
0x134CC00B 323797003 GSK_ERR_LDAP
0x134CC00C 323797004 GSK_ERR_UNKNOWN_ERROR
0x134CC00D 323797005 GSK_ERR_STRUCTURE_TOO_SMALL
0x134CC00E 323797006 GSK_ERR_CIPHER_SUITES_NOT_SUPPLIED
0x134CC065 323797093 GSK_OPEN_CIPHER_ERROR
0x134CC066 323797094 GSK_KEYFILE_IO_ERR
0x134CC067 323797095 GSK_KEYFILE_INVALID_FORMAT
0x134CC068 323797096 GSK_KEYFILE_DUPLICATE_KEY_ERR
0x134CC069 323797097 GSK_KEYFILE_DUPLICATE_LABEL_ERR
0x134CC06A 323797098 GSK_BAD_FORMAT_OR_INVALID_PASSWORD
0x134CC06B 323797099 GSK_KEYFILE_CERTIFICATE_EXPIRED
0x134CC06C 323797100 GSK_ERR_LOAD_GSKLIB
0x134CC06D 323797101 GSK_KEYFILE_NO_CA_CERTIFICATES
0x134CC0C9 323797193 GSK_NO_KEYFILE_PASSWORD
0x134CC0CA 323797194 GSK_KEYRING_OPEN_ERROR
0x134CC0CB 323797195 GSK_RSA_TEMP_KEY_PAIR
0x134CC0CC 323797196 GSK_KEYFILE_PASSWORD_EXPIRED
0x134CC12D 323797293 GSK_CLOSE_FAILED
0x134CC12E 323797294 GSK_CONNECTION_ACTIVE
0x134CC191 323797393 GSK_ERR_BAD_DATE
0x134CC192 323797394 GSK_ERR_NO_CIPHERS
0x134CC193 323797395 GSK_ERR_NO_CERTIFICATE
0x134CC194 323797396 GSK_ERR_BAD_CERTIFICATE
0x134CC195 323797397 GSK_ERR_UNSUPPORTED_CERTIFICATE_TYPE
0x134CC196 323797398 GSK_ERR_IO
0x134CC197 323797399 GSK_ERR_BAD_KEYFILE_LABEL
0x134CC198 323797400 GSK_ERR_BAD_KEYFILE_PASSWORD
0x134CC199 323797401 GSK_ERR_BAD_KEY_LEN_FOR_EXPORT
0x134CC19A 323797402 GSK_ERR_BAD_MESSAGE
0x134CC19B 323797403 GSK_ERR_BAD_MAC
0x134CC19C 323797404 GSK_ERR_UNSUPPORTED
0x134CC19D 323797405 GSK_ERR_BAD_CERT_SIG
0x134CC19E 323797406 GSK_ERR_BAD_CERT
0x134CC19F 323797407 GSK_ERR_BAD_PEER
0x134CC1A0 323797408 GSK_ERR_PERMISSION_DENIED
0x134CC1A1 323797409 GSK_ERR_SELF_SIGNED
0x134CC1A2 323797410 GSK_ERR_NO_READ_FUNCTION
0x134CC1A3 323797411 GSK_ERR_NO_WRITE_FUNCTION
0x134CC1A4 323797412 GSK_ERR_SOCKET_CLOSED
0x134CC1A5 323797413 GSK_ERR_BAD_V2_CIPHER
0x134CC1A6 323797414 GSK_ERR_BAD_V3_CIPHER
0x134CC1A7 323797415 GSK_ERR_BAD_SEC_TYPE
0x134CC1A8 323797416 GSK_ERR_BAD_SEC_TYPE_COMBINATION
0x134CC1A9 323797417 GSK_ERR_HANDLE_CREATION_FAILED
0x134CC1AA 323797418 GSK_ERR_INITIALIZATION_FAILED
0x134CC1AB 323797419 GSK_ERR_LDAP_NOT_AVAILABLE
0x134CC1AC 323797420 GSK_ERR_NO_PRIVATE_KEY
0x134CC1AD 323797421 GSK_ERR_INVALID_V2_HEADER
0x134CC1AE 323797422 GSK_ERR_CERTIFICATE_EXPIRED
0x134CC1AF 323797423 GSK_ERR_CERTIFICATE_REVOKED
0x134CC1B0 323797424 GSK_ERR_NO_NEGOTIATION
0x134CC1B1 323797425 GSK_ERR_EXPORT_RESTRICTION
0x134CC1B2 323797426 GSK_ERR_INCOMPATIBLE_KEY
0x134CC1B3 323797427 GSK_ERR_UNKNOWN_CA
0x134CC1B4 323797428 GSK_ERR_BAD_CRL
0x134CC1B5 323797429 GSK_ERR_CONNECTION_CLOSED
0x134CC1B6 323797430 GSK_ERR_INTERNAL_ERROR_ALERT
0x134CC1B8 323797432 GSK_ERR_INCORRECT_KEY_USAGE
0x134CC1B9 323797433 GSK_ERR_CLIENT_AUTH_ALERT
0x134CC1BA 323797434 GSK_ERR_MULTIPLE_LABEL
0x134CC1BB 323797435 GSK_ERR_MULTIPLE_DEFAULT
0x134CC1BC 323797436 GSK_ERR_RNG
0x134CC1BD 323797437 GSK_ERR_DB_NOT_FIPS
0x134CC1BE 323797438 GSK_ERR_TLS_EXTENSION_MISMATCH
0x134CC1BF 323797439 GSK_ERR_REQUIRED_TLS_EXTENSION
0x134CC1C0 323797440 GSK_ERR_UNRECOGNIZED_NAME
0x134CC1C1 323797441 GSK_ERR_INVALID_FRAGMENT_LENGTH
0x134CC1C2 323797442 GSK_ERR_BAD_MSG_LEN
0x134CC1C3 323797443 GSK_ERR_ECURVE_NOT_SUPPORTED
0x134CC1C4 323797444 GSK_ERR_EC_PARAMETERS_NOT_SUPPLIED
0x134CC1C5 323797445 GSK_ERR_SIGNATURE_NOT_SUPPLIED
0x134CC1C6 323797446 GSK_ERR_BAD_EC_PARAMS
0x134CC1C7 323797447 GSK_ERR_ICSF_NOT_AVAILABLE
0x134CC1C8 323797448 GSK_ERR_ICSF_SERVICE_FAILURE
0x134CC1C9 323797449 GSK_ERR_ICSF_NOT_FIPS
0x134CC1CA 323797450 GSK_ERR_BAD_V3_EXPANDED_CIPHER
0x134CC1CB 323797451 GSK_ERR_ECURVE_NOT_FIPS_APPROVED
0x134CC1CC 323797452 GSK_ERR_RENEGOTIATION_INDICATION
0x134CC1CD 323797453 GSK_ERR_EC_PARAMETER_FORMAT
0x134CC1CE 323797454 GSK_ERR_EC_POINT_FORMAT
0x134CC1CF 323797455 GSK_ERR_CRYPTO_HARDWARE_NOT_AVAILABLE
0x134CC1D0 323797456 GSK_ERR_BAD_ECURVE_LIST
0x134CC1D1 323797457 GSK_ERR_ICSF_FIPS_DISABLED
0x134CC1D2 323797458 GSK_ERR_BAD_SIG_ALG_PAIR
0x134CC1D3 323797459 GSK_ERR_MISSING_SIGNATURE_ALGORITHM
0x134CC1D4 323797460 GSK_ERR_MISSING_KEY_ALGORITHM
0x134CC1D5 323797461 GSK_ERR_INCORRECT_KEY_ATTRIBUTE
0x134CC1D6 323797462 GSK_ERR_NON_SUITE_B_CERTIFICATE
0x134CC1D7 323797463 GSK_ERR_SECURE_LABEL_OPERATION_UNSUPPORTED
0x134CC1D8 323797464 GSK_ERR_ICSF_CLEAR_KEY_SUPPORT_NOT_AVAILABLE
0x134CC1D9 323797465 GSK_ERR_OCSP_SIG_REQUIRED
0x134CC1DA 323797466 GSK_ERR_BAD_HTTP_RESPONSE
0x134CC1DB 323797467 GSK_ERR_BAD_OCSP_RESPONSE
0x134CC1DC 323797468 GSK_ERR_NO_CACHE_SESSION
0x134CC1DD 323797469 GSK_ERR_CLIENT_AND_SERVER_SID_DO_NOT_MATCH
0x134CC1DE 323797470 GSK_ERR_PEER_ATTRIBUTES_CONFLICT
0x134CC1DF 323797471 GSK_ERR_BAD_SID_VALUE
0x134CC1E0 323797472 GSK_ERR_BAD_PEER_ID
0x134CC1E1 323797473 GSK_ERR_OCSP_RESPONDER_ERROR
0x134CC1E2 323797474 GSK_ERR_OCSP_EXPIRED
0x134CC1E3 323797475 GSK_ERR_OCSP_REQ_ERROR
0x134CC1E4 323797476 GSK_ERR_MAX_RESPONSE_SIZE_EXCEEDED
0x134CC1E5 323797477 GSK_ERR_HTTP_IO_ERROR
0x134CC1E6 323797478 GSK_ERR_OCSP_NONCE_CHECK_FAILED
0x134CC1E7 323797479 GSK_ERR_OCSP_RESPONSE_TIMEOUT
0x134CC1E8 323797480 GSK_ERR_REVINFO_NOT_YET_VALID
0x134CC1E9 323797481 GSK_ERR_HOSTNAME_NOT_VALID
0x134CC1EA 323797482 GSK_ERR_P12_BAD_ENCODING
0x134CC1EB 323797483 GSK_ERR_REQ_CERT_BC_EXT_MISSING
0x134CC1EC 323797484 GSK_ERR_MAX_REV_EXT_LOC_VALUES_EXCEEDED
0x134CC1ED 323797485 GSK_ERR_HTTP_RESPONSE_TIMEOUT
0x134CC1EE 323797486 GSK_ERR_LDAP_RESPONSE_TIMEOUT
0x134CC1EF 323797487 GSK_ERR_OCSP_TRY_LATER
0x134CC1F0 323797488 GSK_ERR_OCSP_RESPONSE_SIGALG_NOT_VALID
0x134CC1F1 323797489 GSK_ERR_OCSP_REQUEST_SIGALG_NOT_VALID
0x134CC1F2 323797490 GSK_ERR_OCSP_RESPONSE_NOT_FOUND
0x134CC1F3 323797491 GSK_ERR_OCSP_RESPONSE_DUPLICATE_FOUND
0x134CC1F4 323797492 GSK_ERR_3DES_KEY_PARTS_NOT_UNIQUE
0x134CC1F5 323797493 GSK_INVALID_BUFFER_SIZE
0x134CC1F6 323797494 GSK_WOULD_BLOCK
0x134CC1F7 323797495 GSK_WOULD_BLOCK_READ
0x134CC1F8 323797496 GSK_WOULD_BLOCK_WRITE
0x134Cb1F9 323797497 GSK_ERR_RECORD_OVERFLOW
0x134CC1FA 323797498 GSK_ERR_TASK_MODE_REQUIRED
0x134CC1FB 323797499 GSK_ERR_BAD_CERT_X509_VERSION
0x134CC1FC 323797500 GSK_ERR_KEY_IS_SMALLER_THAN_MINIMUM
0x134CC1FD 323797501 GSK_ERR_BAD_KEY_LABEL_LIST
0x134CC1FE 323797502 GSK_ERR_NO_KEY_LABEL_LIST_MATCH
0x134CC1FF 323797503 GSK_ERR_OCSP_NOT_ENABLED
0x134CC200 323797504 GSK_ERR_BAD_CERTIFICATE_STATUS_RESPONSE
0x134CC201 323797505 GSK_ERR_INAPPROPRIATE_PROTOCOL_FALLBACK
0x134CC202 323797506 GSK_ERR_MISSING_TLS_EXTENSION
0x134CC203 323797507 GSK_ERR_BAD_KEY_SHARE_LIST
0x134CC204 323797508 GSK_ERR_NO_COMMON_KEY_SHARES
0x134CC205 323797509 GSK_ERR_MISMATCH_ECCURVE_KEY_SHARES
0x134CC206 323797510 GSK_ERR_ALERT_NOT_YET_SUPPORTED
0x134CC207 323797511 GSK_ERR_REQUIRED_CIPHERS_NOT_SPECIFIED
0x134CC208 323797512 GSK_ERR_REQUIRED_4CHAR_CIPHERS
0x134CC209 323797513 GSK_ERR_FALLBACK_ATTEMPT_DETECTED
0x134CC20A 323797514 GSK_ERR_INVALID_SIGALG_USED
0x134CC20B 323797515 GSK_ERR_KEY_SHARES_LARGER_THAN_ECCURVE
0x134CC20C 323797516 GSK_ERR_REMOTE_MISSING_TLS_EXTENSION
0x134CC20D 323797517 GSK_ERR_REMOTE_BAD_KEY_SHARE_LIST
0x134CC20E 323797518 GSK_ERR_MISSING_REQUEST_SIGALGS
0x134CC20F 323797519 GSK_ERR_BAD_LOCAL_CERT_X509_VERSION
0x134CC210 323797520 GSK_ERR_LOCAL_KEY_IS_SMALLER_THAN_MIN
0x134CC211 323797521 GSK_ERR_INVALID_TLS_KEY_ALGORITHM
0x134CC212 323797522 GSK_ERR_TLS_EXTENSION_NOT_ALLOWED
0x134CC213 323797523 GSK_ERR_INVALID_COMPRESSION_FIELD
0x134CC214 323797524 GSK_ERR_SESSION_ID_NOT_VALID
0x134CC215 323797525 GSK_ERR_REMOTE_UNSUPPORTED_CERTIFICATE
0x134CC216 323797526 GSK_ERR_BAD_PSK_IDENTITY_VALUE
0x134CC217 323797527 GSK_ERR_BAD_PSK_EXCHANGE_MODES
0x134CC218 323797528 GSK_ERR_SESSION_TICKET_EXPIRED
0x134CC219 323797529 GSK_ERR_BAD_SESSION_TICKET_ATTRIBUTES
0x134CC21A 323797530 GSK_ERR_BAD_RESUMPTION_CIPHER
0x134CC21B 323797531 GSK_ERR_BAD_INPUT_BUFFER
0x134CC21C 323797532 GSK_ERR_PROTOCOL_NOT_SUPPORTED_WITH_FIPS
0x134CC21D 323797533 GSK_ERR_REMOTE_BAD_CERTIFICATE
0x134CC21E 323797534 GSK_ERR_REMOTE_HANDSHAKE_FAILURE
0x134CC21F 323797535 GSK_ERR_REMOTE_BAD_MESSAGE
0x134CC220 323797536 GSK_ERR_REQUIRED_TLS_EXT_MASTER_SECRET
0x134CC221 323797537 GSK_ERR_MISMATCH_TLS_EXT_MASTER_SECRET
0x134CC222 323797538 GSK_ERR_INCORRECT_EXT_KEY_USAGE
0x134CC223 323797539 GSK_ERR_SYSPLEX_SESSION_TICKET_CACHE
0x134CC224 323797540 GSK_ERR_SERVER_REF_ID_NO_MATCH
0x134CC225 323797541 GSK_ERR_P12_BAD_PBKDF2_PARAMETER
0x134CC226 323797542 GSK_ERR_CRL_SIGALG_INVALID
0x134CC227 323797543 GSK_ERR_CRL_SIGALG_NOT_FOUND
0x134CC259 323797593 GSK_ERR_NOT_SSLV3
0x134CC25A 323797594 GSK_MISC_INVALID_ID
0x134CC25B 323797595 GSK_INVALID_FUNCTION
0x134CC25C 323797596 GSK_ERR_CIPHER_RESET_REQUIRED
0x134CC25D 323797597 GSK_ERR_NOT_SUPPORTED_BY_PROTOCOL
0x134CC2BD 323797693 GSK_ATTRIBUTE_INVALID_ID
0x134CC2BE 323797694 GSK_ATTRIBUTE_INVALID_LENGTH
0x134CC2BF 323797695 GSK_ATTRIBUTE_INVALID_ENUMERATION
0x134CC2C0 323797696 GSK_ATTRIBUTE_INVALID_SID_CACHE
0x134CC2C1 323797697 GSK_ATTRIBUTE_INVALID_NUMERIC_VALUE
0x134CC2C2 323797698 GSK_ATTRIBUTE_INVALID_PARAMETER
0x134CC2C3 323797699 GSK_ATTRIBUTE_INVALID_TLS_EXTENSION
0x134CC2C4 323797700 GSK_ATTRIBUTE_INVALID_TLS_EXT_DATA

So my value of 0x134CC000 .. means GSK_OK.

Lots of little problem including ping not working

It all started with a simple problem. I just wanted to use rsync on Linux to copy some files from z/OS to a backup disk. In later discussions using rsync may not be the best approach because it does not support the extended attributes and code page support used on z/OS. But that is a different problem.

The problems were

rsync was not installed

rsync uses SSH to connect to the system. I could ssh to the system, so that was OK.

rsync -a colin@172.26.1.2:/usr/tmp/zowet/colin/* /run/media/colin/ADCD110/rsync/zOS
** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html
bash: line 1: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) [Receiver=3.4.1]

This did

  • logon to colin@172.26.1.2
  • :
  • /usr/tmp/zowet/colin/* files from this directory
  • to /run/media/colin/ADCD110/rsync/zOS

The response bash: line 1: rsync: command not found meant that rsync was not installed on the remote end.

Install rsync failed

I found it was available with the zopen install rsync command

The command

zopen install rsync                   
- Querying repo for latest package information.
curl: (6) Could not resolve host: raw.githubusercontent.com

So there was a problem finding raw.githubusercontent.com.

Ping failed., so ….

Investigate the network

Because it was a routing type problem I looked into this configuration. In my resolver started task I had

//* 
//* TCPIP RESOLVER - COLINS
//*
//RESOLVER PROC PARMS=CTRACE(CTIRES00)
//*
//EZBREINI EXEC PGM=EZBREINI,REGION=0M,TIME=1440,
// PARM=('&PARMS',
// 'ENVAR("RESOLVER_TRACE=/var/log/resolver"/')
//SETUP DD DISP=SHR,DSN=COLIN.TCPPARMS(GBLRESOL),FREE=CLOSE
//SYSTCPT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//*

The setup file had

  DEFAULTTCPIPDATA('COLIN.TCPPARMS(GBLTDATA)') 
GLOBALTCPIPDATA('COLIN.TCPPARMS(RESOLVE)')
# GLOBALTCPIPDATA(/etc/resolv.conf)
;
# -----------------------------------------------------------------
# Default zPDT Linux Base to z/OS Tunnel (Stand-Alone)
# -----------------------------------------------------------------
;
# GLOBALIPNODES(/etc/hosts)
GLOBALIPNODES('COLIN.TCPPARMS(HOSTS)')

The HOSTS file had

185.199.110.133        raw.githubusercontent.com 
185.199.110.133 release-assets.githubusercontent.com
169.63.188.167 downloads.pyaitoolkit.ibm.net

I tried refreshing it,

f resolver,refresh,setup=COLIN.TCPPARMS(GBLRESOL)

But that didn’t help.

A wireshark trace

I did a wireshark trace. It showed the ping request coming from z/OS, but there was no response.
This felt familiar, and found a blog post (from me) zopen: installing packages which pointed me to How to get out to the internet. SNAT, DNAT and MASQUERADE

This had

iface=wlxcc641aee92c5
sudo iptables -t nat -A POSTROUTING -s 10.1.1.2 -o $iface -j MASQUERADE

Unfortunately, my wireless router had changed, and I did not have my current router in the list – so it was failing.

The solution for the network problem

I added my wireless router (with the correct name) and I was able to issue the command zopen install rsync

Retry the rsync command – not in path

I retried the rsync command – and it still failed.

When I log interactively using SSH it invokes .profile.

When I logon non interactively it uses .bashrc.

In my .bashrc script, I had not set the path. I added

export PATH=/usr/zopen/usr/local/bin/:$PATH 
export >> bashlog

I looked at the bashlog file and it had

declare -x PATH="/usr/zopen/usr/local/bin/:/bin" 

So the PATH now included zopen stuff.

The non interactive script was writing to “the console”.

In the script I was writing out debug information.

I used the following to solve the problem

echo "in shell" b>> bashlog

I could log on and see what environment variables were used.

And finally

And the rsync from Linux worked!

What a faff to get the rsync command to work, and later on find that rsync is not the best option for backing up files on z/OS!

Some people just do not think!

I had to fly somewhere, and I checked in a bag. When I got to my destination my bag didn’t arrive. The screen said flight ABC123, all bags delivered – agent=someAgent.

What then followed was, on the whole, a disappointing demonstration of lack of thinking.

Where is the agent?

I had to ask the person at the exit where the agent’s desk was. “Down there on the right”. This was true – but a better statement would have been – we are in zone A. The desk is in zone C – about 400 yards away. I found it after a long walk around the airport.

I eventually found the agent, who gave me a piece of paper saying phone this number, or email them on….

To me, the obvious thing would be to have a piece of paper in the baggage collection area saying –

If your agent is … then phone…. or email… with details of your bag…
If your agent is…

I checked into the hotel, and tried phoning the number. After half an hour I gave up, because no one answered.

I emailed them, then went for a beer in the bar. An hour later I had a reply “You bag was in the baggage room”.

The first problem is that the baggage came out on a gravity slope – with rollers. Heavy suitcases rolled down no problem. Light bags didn’t move – and needed a shove from a heavy bag. I am guessing my bag was last off, and because it was light – it stuck at the top of the slope – and didn’t move. The thought of going behind the rubber shutters to see if my bag was stuck, never occurred to me!

The second problem was… OK my bag is at the airport – how do I get it. I tried phoning – and after a half hour gave up. I sent a second email asking what I should do, I didn’t get a reply.
I went to the agent – who suggested I phone the number AGAIN. (This was 10 pm)

I bumped into some airport staff – wearing their distinctive uniforms. One guy said go and stand by Costa, and I’ll find you. 10 minutes later he had found my bag and I was reunited.

I raised a complaint to the airport – I look forward to my reply… in a month. ( In the complaint I said how helpful the member of staff was).

So although it had a happy ending. I thought why my experience was so bad. Had people who designed the aiport, or worked for the agent, never flown before – or lost thier luggage.

What I would have done?

  • Have a “lost baggage computer terminal” in the baggage area. You enter your bag tag, phone number, and email address.
  • Have instructions written on the wall on what to do if your bag has not arrived
  • Provide useful information, “We have found your bag. Please go to … or find a member of staff who can then arrange to get it for you”
  • The someAgent’s desk is in zone C.

Using Linux rsync to back up files.

I am a great believer in having multiple backups of my data. I’ve been trying out the rsync facility available on Linux (and other Unix platforms). I think of rsync as remote synchronise with my current files.

My shell script

it="--itemize-changes"
dest="/media/colin/ADCD110/rsync/"
rsync $it -a /home/colin $dest --exclude-from='exclude-file.txt'

Where

  • –itemize-changes lists the files being backed up
  • -a This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission)
    • -r recursive
    • -l When symlinks are encountered, recreate the symlink on the destination.
    • -t preserve modification times
    • -g preserve group information
  • /home/colin from this directory
  • copied to /media/colin/ADCD110/rsync/, this external disk drive and directory
  • –exclude-from=’exclude-file.txt’ exclude files from the file

and exclude-file.txt contains

colin/.docker/*
colin/Downloads/*
colin/snap/*
colin/.cache/*
colin/grafana-enterprise_13.0*
colin/snap/*
colin/snap
colin/.config/Code/logs/*
: Especially this one
colin/.docker/desktop/vms/0/data/Docker.raw

The output

When I ran the command I got output like

>f.st...... colin/.local/share/gnome-shell/application_state
.d..t...... colin/.local/share/nautilus/tags/

Where

  • > means that a file is being transferred
  • f this is a file
  • d this is a directory
  • s the file size is different
  • t the modification time is different

which explains why the file was copied.

A new file (not in the backup) has

>f+++++++++ colin/new

What happens if I rerun it?

Usually no, of few files are copied. Once I start working, then the next time it runs it will backup any changed files, which are not on the exlude-files list.

It must be very easy to write bad documentation – because there is so much around!

There was a chat about the joy of writing technical documentation (and blog posts), one person said they loved doing it, and someone else said it was often looked down as an easy job, for those that cannot code.

I see technical writing is a skill that not every one has. For example it requires empathy – knowing how someone else feels. It is a bit like breathing. Every one can do it. As a singer you need to learn how to breathe to get the music to all the audience.

There are two view of communicating

  • What the audience want’s to hear
  • What I want to tell the audience

I went to a presentation about Greek architecture. Most of the presentation was of pictures of the speaker and his friends sitting in cafes in Greece. In one or two pictures was some architecture. This is an example of “What I want to tell the audience”. The audience wanted to hear about architecture – and having no cafes would have been a bonus.

When you write, you need to consider why people are reading the documentation

  • Is it for a narrative story? “Begin at the beginning, and go on till you come to the end: then stop” said the King of Hearts in Lewis Carroll’s Alice’s Adventures in Wonderland.
  • Is it to find the answer to a question? This might involve searching for text in the document, and the reading will unlikely to be sequential.

You might want to write down a list of questions that you think a reader may have when reading your documentation. You then use these questions as headings. There is a difference between section headings:

  1. Renewing a digital certificate
  2. My certificate has expired. What do I need to do?

The second one is likely to be the question in the reader’s mind, and get a response “Yes – that’s it”.

What knowledge do you expect the reader to have?

If you need to include background information, put it in a section called background. The experienced person can then skip. If you mix background and useful content, then if you speed read the document you might miss the nugget.

People learn in different ways

  • Some people learn best reading
  • Some people learn best by listening
  • Some people learn best by doing

If you can, give examples of instructions the reader can follow themselves.

Some examples of bad documentation

I hired a car. It was a modern hybrid, with a computer screen, and half the normal buttons and controls. The instruction manual for the car was hundreds of small pages long. The first 20 or so pages were about the benefits of a hybrid car. Then followed instructions for fitting a car seat. I could not find the instructions on how to use the handbrake – which was not in the index. I think I eventually found it as “Automatic brake”.

We had great problems parking the car, because it kept complaining “Key not in car” and refusing to lock the doors.

I suggested to the car hire company that they provide one laminated piece of A4 paper with big print with

  • How to adjust the driver’s seat
  • How to adjust the wing mirrors
  • How to start the car
  • How to stop the car and leave it.

Configuring a product

About 40 years ago I had to test installing CICS on VSE.

The instruction were along the lines of

  • define this data set,
  • define that data set,
  • define this VSAM file,
  • define this VTAM APPLID…
  • Now start it up. “It will give a successfully started message”

There were more than 40 things you had to define.

When it failed, as it usually did, you got a message like “££BOPEN failed” and no information about what have failed to open. You had to go through every step and check the definitions.

I worked with the developer and a very good writer and we came up with a script

  1. Define the minimal CICS JCL set every thing possible to no. For example DUMP=NO. There were about 15 lines of JCL.
  2. Start CICS – it it fails check the definitions
  3. Stop CICS
  4. Define the DUMP data sets
  5. Change the option DUMP=NO to DUMP=YES
  6. Start CICS – it it fails check the dump data set definitions
  7. Define something else, change the option, start … stop…

If it failed, then it was obvious what caused the problem. It was the definition you just added.

This was shipped as the install documentation and customers loved it.

A few years later I was involved in the product again, and all of the useful stuff had been stripped out, and the instructions were like the original set of instructions. The tester had his set of “testing JCL” which he ran to make sure it worked.

The tester did not try to type the definitions in with all of the typical user errors. You need to have people with the appropriate level of skill to test the product, and read the documentation. Appropriate might be skilled in the platform – but low knowledge about the product. Having people skilled in the product doing the basics is wrong. We had managers do the install – that was great because they did not understand the technical level.

You need to approach testing and documentation from the user’s perspective – not from your perspective – this is the empathy I mentioned above.

Don’t repeat … point to it, and add value

I was researching how to do something, and there were many articles about how to do it. Some were short, some were long, and they all covered the basics – but not many articles covered the next step after the basics.

It would be better for every one if some of the people had written

  • “This topic is well documented here… so I will not repeat it. Following on from this, the next, more complex thing to do is …”.

There are many articles on how to create a digital certificate on Linux. My follow-on questions include

  • How to I use a second certificate, or how to I use a second certificate authority certificate in the same scenario?
  • If I want to invalidate a certificate – how do I do it?
  • If my certificate expires what do I need to do.

These questions are not usually covered.

Signposting

If the whole content fits on one page, you may not want signposts. If it is a long document (more than a page or two) to need to show the document structure.

You need a table of contents, with links to the text to help people understand the document layout, and where to go. At the top of a blog post may be text like

  • Setting up the Certificate Authority
  • Setting up the certificate
  • …
  • If I want to invalidate a certificate – how do I do it?
  • If my certificate expires what do I need to do?

Give the high level view and then go into more detail

For a complex project start high and work your way into the complexity. For one very complex presentation I started at the high level, then went through it again in more detail, and then a third time in great detail. At each detailed step the audience knew where they were going, because they had already been along the path.

I went to a presentation on an introduction to a certain data protocol. By slide 3 we were looking at all of the fields in the data header. I did not know what the protocol was for… so knowing all the details of fields was wasted on me when I didn’t know what the protocol did.

Someone may not know what you know.

Ensure that you explain terms, or the first time you use a term which may not be generally known; give a link to a definition. Those that know what the term means, will not go to the link. Those that do not know – can go and find out.

I have been working on a presentation about TLS (Transmission Link Security). I got to a point where I had to describe the different sorts of hashing available. I realised I needed a page explaining what Hashing data entails before describing TLS.

Before describing the very complex description of the TLS hand shake. I described how a conversation between someone and the Milk Management team in the EU might work.

  • You: (phoning Brussels) Hello I speak English and French, I would like to talk to someone in the Milk department
  • Them: Hello – we will speak English, my name is… Who are you…

Because the concepts were similar, explaining the TLS handshake was much easier.

Keep it simple

Explain what is needed and do not include information which may only confuse, or may never be used.

Someone explained a protocol to me it went along the lines of

  • At offset 0 is an eye catcher ABCD
  • At offset 4 is the length of the control block
  • At offset 8 is the function 0 = start, 2 = query, 3 = terminate
  • At offset 12 is the date…

The reply has

  • At offset 0 is the eyecatcher
  • ..
  • At offset 8 is the return code – 0 means it worked successfully

At the introduction level I do not need to know the offsets or the values. Several values are not of interest until I start trying to program the protocol,

My description of the protocol was like

There are several fields sent to start the session. The important one is

  • …
  • function. This has the value start, query or terminate
  • …

The response has several fields. The important one is

  • …
  • return code – 0 if it worked else a code you can look up HERE to see why it failed.

Afterthoughts

  • I saw this article as a sequential read through – perhaps as a basis for a talk at a conference next year?
  • It is a long article. I should have used sign posts, or section headers at the top.

Is it my new glasses, or have my x3270 sessions gone out of focus?

I’m running on Ubuntu Resolute Racoon, and after a recent update, my x3270 sessions connecting to z/OS all look out of focus, or like my glasses are dirty.

I think that something has changed in the fonts that are used, or my screen sizes, so when data is displayed it is having to fake the font.

I define a 3270 using a command like

x3270 -model 5 -title mstcon mstcon@localhost:3270  &

where mstcon is the short name.

I needed to be able to change the size of the 3270 screen, so the fonts were clear.

The root problem was the laptop’s screen was defined as scale 125%.  Changing this to 100% and restart the 3270 sessions fixed it.

Third attempt at fixing it.

The easiest way of fixing the problem was to use

export DPI=240
x3270a -model 5 -title mstcon mstcon@localhost:3270 &

This may all be connected with X11 being removed from Ubuntu 26.04, and Ubuntu using Wayland instead. The XWayland code making the system look like X11 may have a couple of hickups.

The information below is from my journey to getting a readable screen.

What is the size of my screen?

You can use the wmctrl command to display information about the screen. wmctrl interacts with the X11 window manager.

wmctrl -l -G

This lists the geometry of all the screens

0x00600169 -1 396  478  2496 1520 ColinNew mstcon
0x00c00169 -1 794 702 2496 1582 ColinNew colin
0x00e00169 -1 794 764 2496 1520 ColinNew tso

Where

  • ColinNew is the name of my userid
  • mstcon is the name of my x3270 window
  • 396 478 is the position of the top left corner of the window,
  • 2496 1520 is the width and height of the window

You can change the configuration using

 wmctrl -r  colin  -e  1,744,618,2496,1582 

You can grab the edges of the x3270 window, and make adjust the size till it looks right, then use the wmctrl -l -G to capture the geometry – and update your wmctrl … -e command.

Or you can do it with X3270

x3270 ... -geom 2496x1582+711+618 ...

Where -geom is -geometry widthxheight+x+y .

The font size jumps!

There are a set of fonts, as you change the size of the screen, for example make it half an inch narrower, at some point it may reconfigure, and suddenly use a smaller font (with lots of space around the content). You should size the window so it is wide enough and high enough for the font size, without wasting space around the content.

My glasses?

I also had a new pair of glasses, and have to position my head to be able to get the windows in focus. Because I keep my 3270 sessions at the top of my screens, I have to tilt my head back slightly!

Update

I did the following commands


sudo apt install fonts-3270
sudo apt install xfonts-x3270-misc
sudo sudo fc-cache -f -v

And x3270 suddenly got better with readable fonts etc..

Resetting the IBMUSER password on the ZD&T z/OS 3.2 stock image

I was sent this useful document from Dan Kalmar.

The IBM documentation on the initial logon to the ZD&T image in the “standard stock image” may not always work. The information below gives an alternative way of getting in.

Environment

  • Emulator ZD&T Personal Edition 14.2.8 (z1091 1.12.59.17), Ubuntu host
  • Image IBM z/OS 3.2 Dev & Test Stock Image for ZD&T PE 14.2.20 (built 2026-09-08)
  • IPL ipl DE27 parm DE28A0M (SYSRES Z32VS1 at DE27, IODF/LOAD on OPEVS1 at DE28)

The problem

The stock image ships with a randomized IBMUSER password. IBM’s documented procedure for Personal sEdition is to edit the password-injection file on the ZDTDPY volume, include ZDTDPY in the device map and IPL with load parm NV:

pdsUtil ZDTDPY ZDT.PASSFILE/PASS0 --extract (edit line 1 of PASS0.txt, 8 chars)
pdsUtil ZDTDPY ZDT.PASSFILE/PASS0 --overlay
ipl DE27 parm DE28NV

On this system that did not work. ZDTDPY was online with the edited PASS0, the IPL used DE28NV, yet TSO IBMUSER still failed with:

IKJ56474I USERID OR PASSWORD IS INCORRECT OR NOT AUTHORIZED

No console message referred to ZDTDPY, so the injection step did not appear to run. RACF commands from the console are refused as well:

IRRV003I (<) YOU ARE NOT ALLOWED TO ISSUE THE LISTUSER COMMAND
AS AN OPERATOR COMMAND

so the password cannot be reset from the console, and the usual ADCD defaults (IBMUSER / ADCDMST) are rejected too. (Whether the 14.2.20 image needs the matching 14.2.20 emulator for the injection to run is an open question with IBM support.)

Caution: do not keep retrying logons — RACF will revoke the userid after a few failures.

The way in

The image ships two pieces that together allow a job to run with authority, without anyone logging on:

  • SYS1.PROCLIB(SUBMIT) on OPEVS1 — a started task that copies a member to the internal reader. It is
    started with S SUBMIT,DS=dsname,M=member.
  • IBMUSER.JCL(SSHKYCP) on STGVS1 — a sample job (shipped to install an SSH key for IBMUSER). Its JOB card has no USER=, so the submitted job runs under the SUBMIT started-task user, which has enough authority to update IBMUSER.

The members can be read and rewritten from Linux with pdsUtil, the same way PASS0 is. So we replace the SSHKYCP member with a job that issues an ALTUSER, then let the shipped SUBMIT task run it at IPL time.


Key constraint: pdsUtil –overlay is an update-in-place — the replacement file must have exactly the same number of records as the original member. SSHKYCP is 26 records, so the replacement job is padded with comment lines to 26.

Procedure

All Linux commands run as the ZD&T user (ibmsys1). The volumes directory below is this system’s (/media/mybook6/z32/volumes) — substitute your own.

Step 1. Shut down z/OS and stop the emulator

awsstop

Step 2. Back up the volume that holds IBMUSER.JCL (STGVS1 here)

cd /media/mybook6/z32/volumes
cp STGVS1 ../STGVS1.bak

Step 3. Extract the SSHKYCP member and keep an untouched copy

mkdir -p ~/z32check && cd ~/z32check
pdsUtil /media/mybook6/z32/volumes/STGVS1 IBMUSER.JCL/SSHKYCP --extract
cp SSHKYCP.txt SSHKYCP.orig.txt

–extract writes SSHKYCP.txt (ASCII) into the current directory. Note the record count reported (26).

Step 4. Replace SSHKYCP.txt with the reset job, padded to 26 records

The job below runs ALTUSER under TMP/E (IKJEFT01). Keep it at exactly 26 lines. Put your own 8-character password (uppercase + digits) in place of NEWPASS1 on the ALTUSER line:


//PWRESET JOB CLASS=A,MSGCLASS=X
//* Reset IBMUSER password. Runs under the SUBMIT STC user (no USER=).
//* Start with: S SUBMIT,DS=IBMUSER.JCL,M=SSHKYCP
//*
//* --- padding so the member stays 26 records (pdsUtil --overlay rule) ---
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//*
//S1 EXEC PGM=IKJEFT01
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
ALTUSER IBMUSER PASSWORD(NEWPASS1) NOEXPIRED RESUME
LISTUSER IBMUSER
/*

Confirm the count before writing it back:

wc -l ~/z32check/SSHKYCP.txt # must print 26

Step 5. Write the member back to the volume

cd ~/z32check
pdsUtil /media/mybook6/z32/volumes/STGVS1 IBMUSER.JCL/SSHKYCP --overlay


Success looks like: AWSPDS055I 26 records overlayed PDS member ‘SSHKYCP’. A record-count mismatch instead gives AWSPDS054E … RC=-10 — re-check the line count.


Step 6. IPL and run the job from the master console

cd /media/mybook6/z32 && sh runzpdt_z32


Once BASE Z/OS SUBSYSTEMS HAVE BEEN STARTED appears, on the master console:

S SUBMIT,DS=IBMUSER.JCL,M=SSHKYCP


Watch for $HASP395 PWRESET ENDED (and IEF404I PWRESET – ENDED). If no ICH/IRR error appears, the ALTUSER was accepted.


Step 7. Log on and verify

Dial a TSO terminal (the stock image uses devices 0062–0065 for VTAM; 0060/0061 are MCS consoles) and log on:

TSO IBMUSER (then the new password at IKJ56476I ENTER PASSWORD)

The PWRESET job output is held in class X — its LISTUSER confirms IBMUSER’s status.

Step 8. Restore the original member

Once in TSO, put the shipped SSHKYCP back so the image matches IBM’s.

Edit IBMUSER.JCL(SSHKYCP) in ISPF and paste the original contents (SSHKYCP.orig.txt), or re-run –overlay with the saved original while z/OS is down. Then change the password the normal way and delete the volume backup.

Why this works


The submitted job inherits the SUBMIT started task’s RACF identity (no USER= on the JOB card), and on this image that identity has enough authority to issue ALTUSER against IBMUSER. Nothing bypasses RACF: the job runs as an already-authorized started-task user, which is why it can set a password that console RACF commands are blocked from setting. Everything up to Step 6 is done offline with pdsUtil, the same utility IBM’s own PASS0 procedure uses.

Level independence and suspected root cause

This workaround is independent of the emulator and z/OS release. pdsUtil, SYS1.PROCLIB(SUBMIT), a no-USER= job inheriting the started task’s RACF identity, and ALTUSER are all stable across ZD&T and z/OS levels, so the method works whether or not the PASS0 injection mechanism is present on the image.


The likely reason IBM’s documented PASS0 procedure produced nothing here: the code that reads ZDT.PASSFILE(PASS0) at IPL may require a later emulator than 14.2.8. The image is built for ZD&T 14.2.20; this host runs the 14.2.8 emulator. That would explain why ZDTDPY was online with the edited PASS0 and the IPL used DE28NV as documented, yet no console message referred to the injection and the password was unchanged. This is a hypothesis, not confirmed by IBM — the open question is whether the 14.2.20 stock image requires the matching 14.2.20 emulator for PASS0 injection (and for other functions). If so, applying the 14.2.20 fix pack is the proper fix, and the procedure above is the bridge until then.

Note

This is shared as a working note — verify each step against your own entitlement and IBM documentation. Device numbers, volume serials and paths are specific to this ZD&T system.


How to export and import a Grafana dashboard

Like many things, it is easy once you know how. The instructions on the internet usually missed one tiny critical piece of information. |←

Export a dashboard

In Grafana, select a dashboard. At the bottom of the display select |← (circled in the image). (This was the missing bit of information.)

  • This gives you icons
    • Download
    • Content outline
    • ->|
  • Select Download. This gives

You can either Download the file, or copy to the clip board. Because Grafana is running in a browser, the downloaded file is in your normal Downloads directory. One download had a name dashboard-1791876486315.json

Importing a dashboard

In Grafana, select Dashboards

Select the +^ on the top line, and select import dashboard.

You can drag and drop a file to the area, or select from a file list.

Capturing and replaying Opentelemetry data

I wanted to create some files so that people could get Opentelemetry, and displaying data working, but also to provide sample data. There are several sites offering configuration files, but not the data to display.

This has an additional challenge. With tools like Jaeger and Grafana, you typically display data “in the last 5 minutes”, or “in the last hour”. If I provide data, then it may be viewed a year later. However with a small bit of Python code I fixed this.

The Opentelemetry collector to capture data

This takes the data from the various sources in the network, and collects the information.

The receiver

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

The exporter

exporters:
debug:
verbosity: normal # basic: basic, normal, detailed

otlp_http:
endpoint: "http://jaeger2:4318" # was 4318
tls:
insecure: true
file/in:
path: /file.in

The magic

service:
pipelines:
traces:
receivers: [otlp]
# processors: [transform/timestamps] # I did some transformation of the data
exporters: [debug,file/in,otlp_http]

The flow

  • The data came from z/O into the oltp receiver (grpc: endpoint: 0.0.0.0:4317)
  • The service: traces: receivers…, exporters, says pipe the data from otlp and put it to
    • debug – output to the terminal
    • file/in – this maps to a file definition file.in
    • otlp_http – for Jaeger to use

The docker command

I ran the opentelemetry collector in Docker, and this needs some statements to map from internal names (such as file.in), to the real file.


touch file.in.json
truncate -s 0 file.in.json
chmod 777 file.in.json
docker run --rm --name ozf \
--volume "$(pwd)/o2.yaml":/otel-config.yaml \
-v "$(pwd)/file.in.json":"/file.in" \
--env COLLECTOR_OTLP_ENABLED=true \
--publish 4317:4317 \
--publish 4318:4318 \
--network otel-jaeger-network \
otel/opentelemetry-collector-contrib:latest --config otel-config.yaml

The command executed is

otel/opentelemetry-collector-contrib:latest --config otel-config.yaml 

Docker will download the code if needed. It will take the latest level of code.

The truncate file -s o resets the file to empty.

The configuration is in otel-config.yaml. The option –volume “$(pwd)/o2.yaml”:/otel-config.yaml maps otel-config.yaml to the file “$(pwd)/o2.yaml”

The yaml configuration file refers to /file.in. This maps to $(pwd)/file.in.json on my file system.

You can then issue commands like

cat file.in.json | jq |less

to look at the data.

The Opentelemetry collector to replay data

As well as receiving data over http, there are other sources, one of which is oltp_json_file which reads opentelemetry data from a file

receivers:
otlp_json_file:
include:
- "/input.json"
start_at: beginning

The docker command

docker run --rm  --name ozf \
--volume "$(pwd)/o1.yaml":"/otel-config.yaml" \
-v "$(pwd)/otel.in.json":"/input.json" \
--env COLLECTOR_OTLP_ENABLED=true \
--publish 4317:4317 \
--publish 4318:4318 \
--network otel-jaeger-network \
otel/opentelemetry-collector-contrib:latest --config otel-config.yaml

The data is read from file file otel.in.json in the current directory.

Transforming the date times

The JSON data has start and end times. The display tools display “recent data”, such as the “last 5 minutes”. I wrote some python code to find the earliest time stamp, calculate the data from the time stamp to now, and adjust all the timestamps.

'''
Python script to adjust the datetimes in an opentelemety json file, so it
looks as it the data has just been created.

This has been tested on Ubuntu

Format

python3 fixtime filename1 <filename2>

Where
filename1 is the name of the input file
filename2 is the name of the output file. The default is otelts.json

'''
import csv, json
import sys
import json
from datetime import datetime
import json
import time
earliestTime = 0
if len(sys.argv) < 1:
print("fixtime adjusts the otel datetime stampts in otel data to look like they've just been created")
print("The format is fixtime <inputfile> <outputfile>")
print("Where the defaults are inputfile is file.in.json and the outputfile is otel.in.json ")
sys.exit(0)
if len(sys.argv) < 2:
inputFile = "file.in.json"
else:
inputFile = sys.argv[1]
if len(sys.argv) < 3:
outputFile = "otel.in.json"
else:
outputFile = sys.argv[2]
timenow = time.time_ns()
lineno = 0
with open(inputFile, 'r') as file, open(outputFile, 'w') as fout:

for line in file:
data = json.loads(line)
# get the first time - so we can offset all of the time values
if earliestTime == 0: # get the first time found in a span
for d,dvalue in data.items(): # a dict with one item, resourceSpans
for resourceSpan in dvalue : # it is a list - do each one
scopeSpans= resourceSpan["scopeSpans"]
for scopeSpan in scopeSpans: #"scopeSpans": [ is a list of scopes
spans = scopeSpan["spans"] # a dict
# print(spans)
for s in spans: # each span in the list Only element
# s["scopeSpan"] = scopeSpan
earliestTime = s["startTimeUnixNano"]
delta = timenow- int(earliestTime)
break
break
break

# now process all of the data and add the delta to the time stampb
for d,dvalue in data.items(): # a dict with one item resourceSpans
for resourceSpan in dvalue : # it is a list - do each one
scopeSpans= resourceSpan["scopeSpans"]
for scopeSpan in scopeSpans: #"scopeSpans": [ is a list of scopes
spans = scopeSpan["spans"] # a dict
# print(spans)
for s in spans: # each span in the list Only element
s["startTimeUnixNano"] = int(s["startTimeUnixNano"]) + delta
s["endTimeUnixNano"] = int(s["endTimeUnixNano"])
# and now write out the updated data
jsonstr = json.dumps(data)
fout.write(jsonstr+"\n")

sys.exit(0)

How does it all fit together?

  • I ran the otel collector in one terminal, and Jaeger in another.
  • I stopped both of them
  • Use python3 fixtime json.file.in otel.in.json This created file otel.in.json.
  • Started otel
  • Start jaeger
  • Look at the data in a web browser pointing to the Jaeger address and port.

Where is my MQ Opentelemetry output from z/SOS?

With Opentelemetry, as work flows through a system, it sends “I am here” data back to a collector, which can process it, and pass it on to other tools to display the information on graphical dashboards.

I spent several days trying to work out why I was not getting any telemetry data from my queue managers.

Basic setup

Check SMF

The telemetry data is written to SMF. You should check it is being collected. The command D SMF gives output like

IFA714I 08.45.27 SMF STATUS          FRAME LAST   F      E   SYS=VS01       
LOGSTREAM NAME BUFFERS STATUS
A-IFASMF.DEFAULT 0 CONNECTED
A-IFASMF.COLIN 0 CONNECTED
A-IFASMF.INMEM 1289792 IN-MEMORY
A-IFASMF.MQOTEL 393304 IN-MEMORY
A-IFASMF.T1159 0 IN-MEMORY

This has some data in the buffers, so it looks like data is being produced.

You can check the SMF option using the command D SMF,O

This gives a lot of information including

INMEM(IFASMF.T1159,TYPE(1159),RESSIZMAX(0128M)) -- PARMLIB     
INMEM(IFASMF.MQOTEL,TYPE(1158),RESSIZMAX(0128M)) -- PARMLIB
INMEM(IFASMF.INMEM,TYPE(30),RESSIZMAX(0128M)) -- PARMLIB

This shows the MQ SMF records, type 1158 are mapped to name IFASMF.MQOTEL. It may be different on your system.

Check the data gatherer is active

This is a Java program which runs on z/OS and writes to an HTTP connection. I run mine as a started task.

This program does not report any statistics, if it is using CPU, then is may be processing records.

Check the network traffic

I use TLS encryption on my connection to the Opentelemetry collector server running on Linux. Using tools like wireshark on the connection allow you to see the overall traffic, but not the content of the traffic. For example

Shows there is traffic to and from port 4317 on my Linxu box.

  • The records with length of 8258 contains my Otel traffic
  • There is also TCP/IP Keep Alive flows every 15 seconds or so.

On the Linux Opentelemetry collector

During setup, I had the collector write to debug, and to a file, so I could see the traffic coming down.

But no data is being produced.

There are two switches that need to be enabled.

The Opentelemetry state is sent as a message property within MQ messages.
There is a property traceparent, which identifies the high level piece of work. The data is of the format

  • trace identifier ‘-‘ span identifier ‘-‘ flags

Where flags is a two byte character string such as 01.

Format of the flags

  • If the rightmost bit of the flags is 0, then this signals do not collect any data.
  • If the rightmost bit of the flags is 1, then do more checks.
    • If the queue has an attribute OTELTRAC(ON) (either directly, or as specified at the QMGR level), then emit the otel data
    • Else do nothing.

The flag in the traceparent is called SAMPLED, which I found confusing. I didn’t want to sample – I wanted OTEL information for all records!

I had the flags specified as 00 and did not get any output. When I changed it to 01 I got the OTEL data – it was as easy as that.