Ahh TCPIP redirect solved my routing problem

I was trying to go from z/OS running on zD&T system on a Linux server to the external internet. It was very frustrating in that a ping to a site would not work, I made an adjustment, ping still didn’t work, I made another adjustment, and then it worked. I then undid the adjustments and it worked! I optimised this by doing 3 pings – the first two failed, then it worked.
If I re-ipled – it worked. If I shutdown, and restarted the Linux server – it failed the same way.

My configuration

I had the default for IPV4 going to address 192.168.1.22. This was the value of the connection if I used FIND_IO or ip addr.

What I saw was

  1. Source 192.168.1.25 -> 151…. this did not work ( no response )
  2. Source 192.168.1.25 -> 151…. this did not work ( no response)
  3. Source 192.168.1.22 Redirect ICMP request gateway 192.168.1.254.
    • src 192.168.1.25 -> 151… worked

What was happening was my request with IP address 192.168.1.25 was being routed (because of my routing definitions) to 192.168.1.22. I don’t know if the request ever got out of my Wireless router, or the site I was pinging was unable to send a response back. After the second of these incidents the “router” with address 192.168.1.22 send a redirect message to my original node saying instead of sending me the traffic – send it directly to address 192.168.1.254 which is my wireless router’s address.

I changed the routing to be 192.168.1.254 – and next time I restarted my server, and reipled z/OS, I could ping and it worked every time.

Lesson learned

I learned that it is important to get the right definitions.

I also learned that making a change, and when you undo the change, you do not always get back to the original state.

Processing lines in ASCII files in ISPF edit macros

I was trying to make it easier to read the trace lines from Liberty. These lines can be hundreds of characters long – and it needs many scrolls right and left to display it.
If I extracted and displayed the line – it displayed garbage because the line was in ASCII, and if you just displayed the line, it displays the ASCII values.

This raised several challenges.

  1. How do do you convert from ASCII to EBCDIC in an ISPF Rexx macro.
  2. How do you nicely display a long line on one screen.

How do do you convert from ASCII to EBCDIC in an ISPF Rexx macro?

I could not find any easy code to copy, so I had to write code to create the ASCII to EBCDIC mapping in Rexx.

/* REXX */ 
/*
exec to display long (Liberty) logs by flowing text
*/
ADDRESS ISPEXEC
'ISREDIT MACRO'
"ISREDIT autosave off "
ascii0 = "000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F"
ebcdic0 = "................................"
ascii1= "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F40"
ebcdic1 = " !..$.&'()*+,-./0123456789:;<=>?@"
ascii2 ="4142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F60"
ebcdic2= "ABCDEFGHIJKLMNOPQRSTUVWXYZ(\)._."
ascii3 ="6162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F"
ebcdic3= "abcdefghijklmnopqrstuvwxyz{|}~."

ascii = ascii0 || ascii1 || ascii2||ascii3
ebcdic = ebcdic0 || ebcdic1|| ebcdic2||ebcdic3

ascii = x2c(ascii)
/*get the current line number, and extract it */
"ISREDIT (l) = LINENUM .ZCSR "
"ISREDIT ( d ) = LINE " l
/* convert it to ebcdic */
out = translate(d,ebcdic,ascii)

The upper part of the code just creates the translation tables. For example ASCII 0x21i is EBCDIC “!”.

The code

"ISREDIT         (l) = LINENUM .ZCSR " 
"ISREDIT ( d ) = LINE " l
/* convert it to ebcdic from ascii */
out = translate(d,ebcdic,ascii)
say out

reads the current (.ZCSR) line of the file into variable d, converts to EBCDIC and displays it.

How do you nicely display a long line on one screen?

I used

out = translate(d,ebcdic,ascii) 
/* and display it */
out = strip(out)
do i = 1 to length(out) by 72
say substr(out,i,72 )
end

which cuts the message into 72 byte chunks.

I enhanced this to allow me to enter a number of lines on the macro

/* REXX */ 
/*
exec to display long (Liberty) logs by flowing text
*/
ADDRESS ISPEXEC
'ISREDIT MACRO (lines) '
"ISREDIT autosave off "
"ISREDIT (fcol,lcol) = DISPLAY_COLS" /* get width of the screen */
width = lcol - fcol
ascii0 = "000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F"
ebcdic0 = "................................"
ascii1= "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F40"
ebcdic1 = " !..$.&'()*+,-./0123456789:;<=>?@"
ascii2 ="4142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F60"
ebcdic2= "ABCDEFGHIJKLMNOPQRSTUVWXYZ(\)._."
ascii3 ="6162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F"
ebcdic3= "abcdefghijklmnopqrstuvwxyz{|}~."

ascii = ascii0 || ascii1 || ascii2||ascii3
ebcdic = ebcdic0 || ebcdic1|| ebcdic2||ebcdic3

ascii = x2c(ascii)
/*get the current line number, and extract it */
"ISREDIT (last) = LINENUM .ZLAST"
"ISREDIT (l) = LINENUM .ZCSR "
if lines = "" then
lines = 1
do j = l to (l + lines - 1)
if (j > last) then leave
"ISREDIT ( d ) = LINE " j
if rc <> 0 then leave
/* convert it to ebcdic */
out = translate(d,ebcdic,ascii)
/* and display it */
out = strip(out)
/* display full width, based on screen size */

do i = 1 to length(out) by width
say substr(out,i,width)
end
say " "
end
exit

so if my macro is called zz, I can issue the command zz 3 and it displays 3 lines in the file

The couple of hours it took me to write this have made my life so much easier.

Zowe: What does this message mean?

You can use your favourite search engine to look for the message. If it is not found, you can search within the open source.

Thanks to Martin Zeitham for the following.

Each message has prefix, which indicates the server or component. For example ZWEL* is Launcher etc.
There is a command ./zwe diagnose -e messageID to get more details:

./zwe diagnose -e ZWED0020I

gave

This code corresponds to the errors related to the Zowe Desktop and the App Server.

To find the description of this error code, refer to the:

Zowe documentation for Application framework
https://docs.zowe.org/stable/troubleshoot/app-framework/appserver-error-codes

You may also explore reports from other users experiencing the same error by searching
https://github.com/search?q=org%3Azowe+ZWED0020I&type=discussions

I found it quicker to look in the Zowe bin/commands/diagnose/index.js file, which has

And note the ability to see any discussion on the message ZWED0020I in the git repository using

https://github.com/search?q=org%3Azowe+ZWED0020I&type=discussions

Something for the weekend. Which is heavier and ounce of gold or an ounce of paper?

I remember being asked this question at my grandfather’s knee. Of course the popular answer, which is wrong, is they both weight the same. Yes, they have different weights, an ounce of gold is heavier than an ounce of paper.

An ounce of gold weights 31 grams, and an ounce of paper weight 28 grams. Gold is measured in ounces troy, and most other things, including paper, are measured in ounces avoirdupois.

A similar question is why do cars in Britain have a better miles per gallon than in the US – it is not because Americans have bigger cars. A US gallon is 3.7 litres and a UK gallon is 4.5 litres, Americans have smaller litres.

What are good tools for working with a backend server?

I’ve used a web browser, cURL, openssl client, and Python for getting to a back end REST server. I’ve recently discovered HTTPie. For example

https GET https://127.0.0.1:7554/application/loggers –cert colinpaice.pem –cert-key colinpaice.key.pem –verify ca.pem2

This invokes some Python code which does all the work. It produced

The documentation is here.

Keep a session

By default each request is a single shot, nothing is retained (such as a Java Web Token) so you need to enter your credentials every time.

However, HTTPie also supports persistent sessions via the --session=SESSION_NAME_OR_PATH option. In a session, custom HTTP headers (except for the ones starting with Content- or If-), authentication, and cookies (manually specified or sent by the server) persist between requests to the same host.

Named sessions

You can create one or more named session per host. For example, this is how you can create a new session named user1

http --session=user1 -a user1:password .... 

From now on, you can refer to the session by its name (user1). When you choose to use the session again, all previously specified authentication or HTTP headers will automatically be set.

Plugins

There are plugins available see here. For example authentication

Zowe: setup: configuring end user userids

Connect the userid to the right groups.

It is better to give permissions to groups, rather than to individual userids.

If someone changes jobs, within your organisation, you just remove the userid of the person leaving from the group, and they lose all access.

If you have given access to a userid, you have to go through all resources (for example APPLs) and remove access.

When a new person joins, you just connect them to the group, and they should have access.

Zowe groups

If you are going to use Zowe, then user userid needs access to the Zowe group.

  • If class(APPL) OMVSAPPL is defined, the Zowe group needs read access to it
  • If users will be using z/OMSF they will need read access to the class(APPL) IZUDFLT (or what ever is specified)
  • If certificate logon is being used, the certificate will need to map to a userid

Access z/OSMF

If userid will be using z/OSMF they need access to the z/OSMF group.

In the z/OSMF logs, I got

CWWKS2907E: SAF Service IRRSIA00_CREATE did not succeed because user colin2 has insufficient authority to access APPL-ID IZUDFLT. SAF return code 0x00000008. RACF return code 0x00000008. RACF reason code 0x00000020.

I connected userid COLIN2 to the z/OSMF group IZUUSER. I think the change is available immediately.

Access to OMVSAPPL

An application can switch userid by using the pthread_security_np, and passing in a userid and password or a certificate (which maps to a userid).

The application can control which userid can be used by means of an resource in the APPL class.

If the resource was called MYAPPL, then pthread_security_np would use “MYAPPL”, and the userid being switched to much gave read access to MYAPPL.

The default (and only supported) resource is called OMVSAPPL.

If OMVSAPPL in class(APPL) is defined, the userid must have read access to it. z/OSMF also uses OMVSAPPL.
If OMVSAPPL in class(APPL) is not defined, no checks are done.

With Zowe you cannot specify the resource name.

Certificate logon

You can use a digital certificate from a web browser ( curl, or other tools) to authenticate to z/OS.  You need to map the certificate to a userid.

A certificate coming in can have a Distinguished Name like CN=adcdd.O=cpwebuser.C=GB  (Note the ‘.’not ‘,’ between elements).

Your userid needs to have SPECIAL define to be able to use the RACDCERT command (SPECIAL, not just GROUP-SPECIAL).

You will need a definition like (see here for the command)

RACDCERT MAP ID(ADCDD ) - 
    SDNFILTER('CN=adcdd.O=cpwebuser.C=GB') - 
    WITHLABEL('adcdd')

or a general definition for those certificate with  O=cpwebuser.C=GB, ignoring the CN part

RACDCERT MAP ID(ADCDB ) - 
   SDNFILTER('O=cpwebuser.C=GB') - 
   WITHLABEL('cpwerbusergb') 

or using the Issuing Distinguished Name (the Certificate Authority)

IDNFILTER(‘CN=TESTCA.OU=SSSCA.C=GB)

Using a generic

SDNFILTER(‘CN=a*.O=cpwebuser.C=GB’)

does not work.

If you attempt to use a certificate which is not mapped you get

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(COLIN)
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(0163)  SUBJECT(CN=adcdd.O=cpwebuser.C=GB) ISSUER(CN=SSCA8.OU=CA.O=SSS.C=GB).

It is worth defining these using JCL, because if you try to add it, and it already exists then you get a message saying it exists already.  If you know the userid, you can list the maps associated with it.   If you do not know the userid, there is no practical way of finding out – you have to logon with the certificate, and display the userid from the web browser, or extract the list of all users, and use LISTMAP on all of them.

Creating a CBT file

The CBT package is a collection of useful programs which enhance z/OS or make it easier to use. For example the PDS utility is like ISPF 3.4 on steroids. These programs have been collected for many years. Some are written in assembler (from before the time when C or COBOL were generally available), some are written in Rexx, many are new.

Some customers will accept tools from CBT, when they would not accept programs from Github.

This blog post is a guide to creating a package for inclusion in the CBT.

There is some documentation here. And there is a good article Packaging z/OS Open Source (and other) Software for Electronic Distribution by Lionel B Dyck.

The basic package is a PDS. It has a number to identify it. My package (zWireshark) was allocated the number 1063.

I created a PDS COLIN.FILEnnnn.

You should create the following members

@FILnnn

This is a description of the what the package does.

$CHANGES

This contains a change history

***********************************************************************
* *
* C H A N G E L O G *
* ------------------- *
* *
* DATE DESCRIPTION *
* ---------- ------------------------------------------------------ *
* *
* 2025/04/17 V1.0 First version on CBT *

$README

Introduction and instructions on how to use the package.

$RECEIVE

This has the JCL to unpack the package

//COLINR JOB (CCMVS),RECEIVE,                             
// NOTIFY=&SYSUID,
// CLASS=B,MSGCLASS=X,COND=(1,LT)
//*
//* CREATE NECESSARY PARTITIONED DATASETS
//* FOR ZWIRESHARK PACKAGE.
//*
//* (RENAME DATASETS AS PER YOUR INSTALLATION)
//*
//RECEIVE EXEC PGM=IKJEFT01
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
RECEIVE INDS('COLIN.CBT509.FILEnnnn(XMITJCLC)')
DSN('COLIN.ZWIRESHA.JCL')
RECEIVE INDS('COLIN.CBT509.FILEnnnn(XMITLOAD)')
DSN('COLIN.ZWIRESHA.LOADLIB')
/*
//

Your package content

You need to add the files for your package. The files will be record format FB with record length 80. If your file is not in this format you can use the TSO TRANSMIT (XMIT) command to make a portable member from your dataset. See MAKEXMIT below.

MAKEXMIT

This has the JCL I used to create the members of the package

//COLINX   JOB 1,MSGCLASS=H                                    
//S1 EXEC PGM=IKJEFT01,REGION=0M
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
xmit a.a dsn('colin.ZWIRESHA.LOAD') OUTFILE(XMITL)
xmit a.a dsn('COLIN.C.ZWIRESHA') OUTFILE(XMITC)
/*
//XMITL DD DISP=SHR,DSN=COLIN.CBT509.FILEnnnn(XMITLOAD)
//XMITC DD DISP=SHR,DSN=COLIN.CBT509.FILEnnnn(XMITJCLC)
/*
/*

XMITJCLC

I used the MAKEXMIT member to convert the JCL and C file into a portable XMIT file with format FB LRECL 80 in the PDS. This member is the XMITted file

XMITLOAD

I used the MAKEXMIT member to convert the load library into a portable XMIT file with format FB LRECL 80 in the PDS. This member is the XMITted file.

Create the shippable object

In TSO

xmit a.a  dsn('COLIN.FILEnnnn')  OUTFILE('COLIN.FILEnnnn.XMIT')

FTP the COLIN.FILEnnnn.XMIT to my workstation in binary.

Send the file to CBT.

Chaining through control blocks with C. The terrible, the right and the difficult.

The terrible

I have some terrible old C code for chaining through z/OS control blocks, which I wrote when I was first learning C.

 #define FLTCVT     16L 
#define CVTASCBH 564L
#define ASCBJBNI 172L
#define ASCBJBNS 176L
char *plStor = (char*)FLTCVT;
char *plCVT = (char*)*(long*)plStor;
char *plASCB = (char*)*(long*)(plCVT+CVTASCBH);
char *pJBNI = (char*)*(long*)(plASCB+ASCBJBNI);
char *pJBNS = (char*)*(long*)(plASCB+ASCBJBNS);

if( pJBNI != 0 ) // pointer to job name
printf("JobnameI %8.8s\n",pJBNI);

if( pJBNS != 0 ) // pointer to started task
printf("JobnameS %8.8s\n",pJBNS);

This prints out the job name. This has worked, and is not the world’s best C code. *(long*)plStor says take the value of (*) plStor, treating it as a pointer to a long ((long *)).

It needs to have “char *plCVT” so when I add an offset, the offset units is char. If I had had long * plCVT I would have to use offset CVTASCBH/4 instead of CVTASCBH to give the correct offset. The offset of CVTASCBH chars is 564. The offset of CVTASCBH longs is 2256.

Chaining – the right way

In the code above I specified offsets. This is not best practice. It is better to use header files as it makes the code easier to understand – and future proof.

#include <cvt.h> 
#include <ihaascb.h>
struct cvtmap * cvt = *((struct cvtmap **)0x10);
struct ascb * pASCB = (struct ascb *)(pCVT -> cvtascbh);
char * pName = pASCB -> ascbjbns;
printf("name %8.8s\n",pName);

The “*((struct cvtmap **)0x10)” code does not feel very elegant, but that’s C for you. Thanks to David Crayford for improving my C code. Who also said

It’s good practice to always code for 64 bit, even if you compile 31bit.

The above code is right – but it can be done in one instruction (which may not be as clear).

pName = 
((struct ascb *) ((struct cvtmap *) *((struct cvtmap **)0x10) ) -> cvtascbh) -> ascbjbns;
111111111111111 11111111111111111 111111111111111111
2222222222222222222222222
333333333333333333333333333333333333333333333
444444444444444444444444444444444444444444444444444444444444444444444444444

Where the 1,2,3 are the extent of the parenthesis. This is harder to understand than the previous example.

You can do

typedef struct ascb  *  ASCB; 
typedef struct cvtmap * CVT;
pName = ((ASCB) ((CVT) *((struct cvtmap **)0x10) ) -> cvtascbh)-> ascbjbns;

or

#define zASCB ( struct ascb  *) 
#define zCVT ( struct cvtmap *)
pName = (zASCB (zCVT *((struct cvtmap **)0x10) ) ) > cvtascbh)-> ascbjbns;

But I do not think these are as clear as the first examples (too many brackets for one thing).

Some clever code – which you should not use

I saw some “clever” code chaining along control blocks.

char * p = (char * )((int * __ptr32 * __ptr32 * )0)[4][165][53]; 

This is a good example of something which takes an expert seconds to write, but takes people not familiar with this a long time to understand.

The interpretation (thanks Bobby)

  • Consider address 0 as pointing to an array of 4-byte elements (the last ptr32), take the [4]th element, so at 16 bytes after 0, which is where CVT lives,
  • take that address and also consider that pointing to an array of 4-byte elements (the first ptr32), take the [141]th element, so (141* 4= )564 bytes beyond the start of the CVT, which is indeed CVTASCBH,
  • take that address and also consider that pointing to an array of integer elements, also 4 bytes, take the 44th element, so (44 * 4= )176 bytes beyond the start of the ASCB, which points to the jobname.

The generated code looks like

*    ppp = (char * )((int  * __ptr32 * __ptr32 * )0)[4][141][44]; 
5810 0010 L r1,16
5810 1234 L r1,(*)int*(,r1,564) 4 * 141
5800 10B0 L r0,(*)int(,r1,176) 4* 44
5000 D0A8 ST r0,ppp(,r13,168)

Changing it slightly

 *  char*   pppq= (char * )((char * __ptr32 * __ptr32 * )0)[4][141][44]; 
5810 0010 L r1,16
5810 1234 L r1,(*)uchar*(,r1,564) 4 * 141
E300 102C LLC r0,(*)uchar(,r1,44) 1 * 44 because it is a character
5000 D0AC ST r0,pppq(,r13,172)

This loads the single character at offset 44 (not 4 * 44 as it was for long *) and stores the single character. The offsets were of length 4.

If you know what you are doing the above code is compact and concise. For anyone else it could take hours to understand it. (I had to ask!)

If you make a small change it may not behave as you expect!

64 bit programs

The examples above were for 31 bit programs, referencing 31 bit addresses. You need to allow for 64 bit programs. You can specify an address is a 31 bit address by using the C qualifier __ptr32.

When compiled with 64 bit addressing the output is

ppp = (char * )((int  *        *         * )0)[4][141][44]; 
LG r6,32
LG r6,(*)int*(,r6,1128)
LGF r0,(*)int(,r6,176)
STG r0,ppp(,r4,2240)

Where some offsets are now 64 bit – and the CVT at offset 16 is now at offset 32 ( 4 * 8 byte longs) and so wrong.

You can force it to treat an address as 31 bit using

char * ppp = (char * )((int * __ptr32 * __ptr32   )0)[4][141]    ; 
LLGF r6,16
LLGF r6,(*)uchar*(,r6,564)
LLGC r0,(*)uchar(,r6,44)
STG r0,pppq(,r4,2240)

and it now treats the offsets as length 4.

You can use __ptr64 to say this is a 64 bit address. __ptr32 and __ptr64 both work in both 31 and 64 bit programs.

Notes on shared storage

Shared storage is a Unix concept, which works on Unix on z/OS.

It allows you to share data between threads, and between different address spaces. Java Shared Classes exploit this. Java classes can be stored in the shared storage, and “Ahead Of Time” compilation can be stored in the shared cache. Next time a JVM starts, it can look in the shared cache, and use the already compiled class.

Each shared storage segment has a key which you use in the various shared memory API calls.

There are no real restrictions, in some of the documentation on the web, it suggests using 1234 as a key! This has its limitations. A better way, used by Java is based on the inode of a file

You can display information about InterProcess Communication Status (ipcs) using the ipcs command.

You can display shared memory, and shared semaphores.

ipcs command lists…

Shared Memory:                                                           
T ID KEY MODE OWNER GROUP
m 8196 0x00000000 --rw------- WEBSRV WEBGRP
m 73733 0x6137142d --rw------- OMVSKERN SYS1
m 73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN
m 139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN
m 8200 0x610ea00f --rw------- OMVSKERN SYS1

The Java shared class files on my system are

767 -rw-r--r--   1 ZWESVUSR ZWEADMIN  ... C290M17F1A64_semaphore_zoweGW_G43L00             
768 -rw-r--r-- 1 ZWESVUSR ZWEADMIN ... C290M17F1A64_memory_zoweGW_G43L00

Inode 767 is 0x2ff, Inode 768 is 0x300.

0x61… inode…. 2d

061 because created with t = ftok(fn,idi );

Shared Memory:                                                          
T ID KEY MODE OWNER GROUP
m 8196 0x00000000 --rw------- WEBSRV WEBGRP
m 139269 0x6137142d --rw------- OMVSKERN SYS1
m 73734 0x6137e82f --rw------- ZWESVUSR ZWEADMIN
m 139271 0x6100410f --rw------- ZWESVUSR ZWEADMIN
m 8200 0x610ea00f --rw------- OMVSKERN SYS1
Semaphores:
T ID KEY MODE OWNER GROUP
s 135172 0x8137132d --ra------- OMVSKERN SYS1
s 69637 0x8137e72f --ra------- ZWESVUSR ZWEADMIN
s 135174 0x8100400f --ra------- ZWESVUSR ZWEADMIN
s 69639 0x8c10840f --ra-ra-ra- ZWESVUSR ZWEADMIN
s 135178 0x810e9f0f --ra------- OMVSKERN SYS1
s 69643 0x8c108a0f --ra-ra-ra- ZWESVUSR ZWEADMIN

More details

You can use

ipcs -m -a

to display more information including size, and number of users.

Finding the needle in a field of hay.

As part of a debugging a problem I collected some traces. One was over 22 million lines of code! How do you extract information from it?

My trace file was trace.txt and I was looking for entries with java/lang/Object in them.

How big is the field?

The command (wc – lower case l)

wc -l trace.txt

gave me

22662732 trace.txt

22.6 million lines of output

Which part of the field?

grep -n java/lang/Object trace.txt |less

The -n option numbers the source lines.

The grep command gave me output starting with

601428:13:39:41.475442621 0x000000002174c200 j9vm.351 Entry >loader 0x0 class java/lang/Object attemptDynamicClassLoad entry

The first number is the position in the file, 601428.

Searching for “exit” (/exit) gave me

904914:13:39:44.830209269 0x000000002174c200 j9vm.319

line number 904919 in the file.

Extract some of the records

I want to ignore the first 601428-1 lines – then take up to and including line 904914 is a line count of 303486.

tail -n +601428 trace.txt |head -n 303487 |less -N

The plus sign in tail -n + says -ignore the given number of lines.

This extracts the lines of interest and displays them with line numbers ( -N option). You can use >file1 to pipe it into a file, and then use less -N file.

I then iterated, now searching for j9shr – the Java Share Classes component, until I found the couple of interesting lines.

Extract data from a record.

You can extract field, or substrings using awk. For example piping data into

awk '{print substr($0,1,100)}'

Will display only the first 100 characters in a line.

awk '{print substr($0,7,6 )    substr($0,39,140)}'

Displays part of the time stamp – start column 7 for 6 characters, and from column 39 to 140.

You can also use cut to cut give specific fields, or field ranges, or column ranges. For example

cut --fields 1,3-5 -d " "
cut --bytes 1-8,60-100

Where -d is the delimiter.

Clever stuff

The Java trace formatting gives you the option to indent the descriptions, if function p calls function q, function q is indented by 2 additional characters.

I used this, and a bit of Python code to calculate how long each request was active for and display this on the “exit” line of the function. I also saved how many times a function was used, and total time in the function.

The results were interesting – what I thought would be “slow” functions, were actually not slow.