Author: Colin Paice

I retired from IBM where I worked on MQ on z/OS, and did customer stuff. I retired, and keep my hand in with MQ, by playing with it!

Write instructions for your target audience – not for yourself.

Over the last couple of weeks, I’ve been asked questions about installing two products on z/OS. I looked at the installation documentation, and it was written the way I would write it for myself – it was not written for other people to follow.

I sent some comments to one of the developers, and as the comments mainly apply to the other products as well, I thought I would write them down – for when another product comes along.

I’ve been doing some documentation of for AT-TLS which allows you to give applications TLS support, without changing the application, so I’ll focus on a product using TCP/IP.

What is the environment?

The environment can range from one person running z/OS on a laptop, to running a Parallel Sysplex where you have multiple z/OS instances running as a Single System Image; and taking it further, you can have multiple sites.

What levels of software

Within a Sysplex you can have different levels of software, for example one image at z/OS 2.4 and another image at z/OS 2.5 You tend to upgrade one system to the next release, then when this has been demonstrated to be stable, migrate the other systems in turn.

Within one z/OS image you can have multiple levels of products, for example MQ 9.2.3 and MQ 9.1. People may have multiple levels so they test the newer level, and when it looks stable, they switch to the newer level and later remove the older level. If the newer level does not work in production – they can easily switch back to the previous level.

Each version may have specific requirements.

  • If your product has an SVC, you may need an SVC for each version, unless the higher level SVC supports the lower level code.
  • If your product uses a TCP/IP port, you will need a port for each instance.

You need to ensure your product can run in this environment, with more than one version installed on an image.

How do things run?

Often z/OS images and programs run for many months. For example IPLing every three months to get the latest fixes on. Your product instance may run for 3 months before restarting. If you write message to the joblog, or have output going to the JES2 spool, you want to be able to purge old output without shutting down your instance. You can specify options to “spin” off output and make the file purge-able.

Your instance may need to be able to refresh its parameters. For example, if a key in a keyring changes, you need to close and reopen the keyring. This implies a refresh command, or the keyring is opened for each request.

Who is responsible for the system?

For me – I am the only person using the system and I am responsible for every thing.

For big systems there will be functions allocated to different departments:

  • Installation of software (getting the libraries and files to the z/OS image)
  • The z/OS systems team – creating and updating the base z/OS system
  • The Security team – this may be split into platform security(RACF), and network security
  • Data management – responsible for data, backup (and restore), migration of unused data sets to tape, ensuring there is enough disk space available.
  • Communications team – responsible for TCPIP connectivity, DNS, firewalls etc.
  • Database team – responsible for DB2 and other products
  • Liberty and z/OSMF etc built on top of Liberty.
  • MQ – responsible for MQ, and MQ to MQ connectivity.

Some responsibilities could be done by different teams, for example creating the security profile when creating a started task. This is a “security” task – but the z/OS systems programmer will usually do it.

How are systems changes managed?

Changes are usually made on a test system and migrated into production. I’ve seen a rule “nothing goes into production which has not been tested”. Some implications of this are

  • No changes are typed into production. A file can be copied into production, and a file may have symbolic substitution, such as SYSTEM=&SYSNAME. You can use cut and paste, but no typing. This eliminates problems like 0 being misread as O, and 1,i,l looking similar.
  • Changes are automated.
  • Every change needs a back-out process – and this back-out has been tested.
    • Delete is a 2 phase operation. Today you do a rename rather than a delete; next week you do the actual delete. If there is a problem with the change you can just rename it back again. Some objects have non obvious attributes, and if you recreate an object, it may be different, and not work the same way as it used to.

There are usually change review meetings. You have to write a change request, outlining

  • the change description
  • the impact on the existing system
  • the back-out plan
  • dependencies
  • which areas are affected.

You might have one change request for all areas (z/OS, security, networking), or a change request for each area, one for z/OS, one for security, one for networking.

Affected areas have to approve changes in their area.

How to write installation instructions

You need to be aware of differences between installing a product first time, and successive times. For example creating a security definition. It is easy to re-test an install, and not realise you already have security profiles set up. A pristine new image is great for testing installation because it is clean, and you have to do everything.

Instructions like

  • Task 1 – create sys1.proclib member
  • Task 2 – define security profile
  • Task 3 – allocate disk storage
  • Task 4 – define another security profile
  • Task 5 – update parmlib

may make sense when one person is doing the work, but not if there are many teams.

It is better to have a summary by role like

  • z/OS systems programmer
    • create proclib member
    • update parmlib
  • Security team
    • Define security profile 1
    • Define security profile 2
  • Storage management team
    • Allocate disk space

and have links to the specific topics. This way it is very clear what a team’s responsibilities are, and you can raise one change request per team.

This summary also gives a good road map so you can see the scale of the installation task.

It is also good to indicate if this needs to be done once only per z/OS image, or for every instance. For example

  • APF authorise the load libraries – once per z/OS image
  • Create a JCL procedure in SYS1.PROCLIB – once per instance

Some tasks for the different roles

z/OS system programmers

  • Create alias for MYPROD.* to a user catalog
  • APF authorise MYPROD…. datasets
  • Create PARMLIB entries
  • Update LNKLST and LPA
  • Update PROCLIB concatenation with product JCL
  • Create security profiles for any started tasks; which userid should be used?
  • WLM classification of the started task or job.
  • Schedule delete of any old log files older than a specified criteria
  • When multiple instances per LPAR, decide whether to use S MYSTASK1, S MYSTASK2, or S MYSTASK.T1, S MYSTASK.T2
  • Do you need to specify JESLOG SPIN to allows JES2 logs to be spun regulary, or when they are greater than a certain size, or any DD SYSOUT with SPIN?
  • ISPF
    • Add any ISPF Panels etc into logon procedures, or provide CLIST to do it.
    • Update your ISPF “extras” panel to add product to page.
  • Try to avoid SVCs. There are better ways, for example using authorized services.
  • Propagate the changes to all systems in the Sysplex.
  • What CF structures are needed. Do they have any specific characteristics, such as duplexed?
  • How much (e)CSA is needed, for each product instance.
  • Does your product need any Storage Class Memory (SCM).

Security team

  • Create groups as needed eg MYPRODSYS, MYPRODRO, and make requester’s userid group special, so they can add and remove userids to and from the groups.
  • Create a userid for the started task. Create the userid with NOPASSWORD, to prevent people logging on with the userid and password.
  • Protect the MYPROD.* datasets, for example members of group MYPRODSYS can update the datasets, members of group MYPRODRO only have read-only access.
  • Create any other profiles.
  • Create any certificate or keyrings, and give users access to them.
  • Set up profiles for who can issue operator commands against the jobs or procedures.
  • Does the product require an “applid”. For example users much have access to a specific APPL to be able to use the facilities. An application can use pthread_security_applid_np, to change the userid a thread is running on – but they must have access to an applid. The default applid is OMVSAPPL.
  • Do users needing to use this product need anything specific? Such as id(0), needing a Unix Segment, or access to any protected resources? See below for id(0).
  • If a client authenticates to the server, the server needs access to BPX.SERVER in the RACF FACILITY.
  • The started task userid may need access to BPX.DAEMON.
  • If a userid needs access to another user’s keyring, the requestor needs read access to user.ring.LST in CLASS(RDATALIB) or access to IRR.DIGTCERT.LISTRING.
  • If a userid needs access to a private key in a keyring the requester needs If a userid needs access to another user’s keyring, the requester needs control access to user.ring.LST in CLASS(RDATALIB).
  • You might need to program control data sets, for example RDEF PROGRAM * ADDMEM(‘SYS1.LINKLIB’//NOPADCHK) UACC(READ) .
  • Users may need access to ICSF class CSFSERV and CSFKEYS.
  • Use of CLASS(SURROGAT) BPX.SRV.<userid> to allow one userid to be a surrogate for another userid.
  • Use of CLASS(FACILITY) BPX.CONSOLE to remove the generation of BPXM023I messages on the syslog.

Storage team

  • How much disk space is needed once the product has been installed, for data sets, and Unix file systems. This includes product libraries and instance data, and logs which can grow without limit.
  • How much temporary space is needed during the install.
  • Where do Unix files for the product go? for example /opt/ or /var….
  • Where do instance files go. For example on image local disks, or sysplex shared disks. You have an instance on every member of the Sysplex – where you do put the instance files?
  • How much data will be produced in normal running – for example traces or logs.
  • When can the data be pruned?
  • Does the product need its own ZFS for instance data, to keep it isolated and so cannot impact other products.
  • Are any additional Storage Classes etc needing to be defined? These determine if and when datasets are migrated to tape, or get deleted.
  • Are any other definitions needed. For example for datasets MYPROD.LOG*, they need to go on the fastest disks, MYPROD.SAMPLES* can go on any disks, and could be migrated.

Database team

  • What databases, tables,indexes etc are required?
  • How much disk space is needed.
  • What volume of updates per second. Can the existing DB2 instances sustain the additional throughput?
  • What security and protection is needed at the table level and at the field level.
  • What groups are permitted to access which fields?
  • What auditing is needed?
  • Is encryption needed?

MQ

  • Do you need to uses MQ Shared Queue between queue managers?
  • How much data will be logged per second?
  • What is the space needed for the message storage, disk space, buffer pool and Coupling Facility?
  • Product specific definitions.
  • Security protection of any product specific definitions.

Networking

  • Which port(s) to use?
    • Do you need to control access to ports with the SAF resource on the PORT entry, and permit access to profile EZB.PORTACCESS.sysname.tcpname.resname
    • Use of SHAREPORT and SHAREPORTWLM
  • Use of Sysplex Distributor to route work coming in to a Sysplex to any available system?
  • Update the port list – so only specific job can use it
  • RACF profile for port?
  • Which cipher specs
  • Which level of TLS
  • Which certificates
  • Any AT-TLS profile?
  • Any firewall changes?
  • Any class of service?
  • Any changes to syslogd profile?
  • Are there any additional sites that will be accessed, and so need adding to the “allow” list.

Automation

  • If the started tasks, or jobs need to be started at IPL, create the definitions. Do they have any pre-reqs, for example requiring DB2 to be active.
  • If the jobs are shutdown during the day, should they be automatically restarted?
  • Add automation to shut down any jobs or started tasks, when the system is shutdown
  • Which product messages need to be managed – such as events requiring operator action, or events reported to the site wide monitoring team.

Operations

  • Play book for product, how to start, and stop it
  • Are there any other commands?

Monitoring

  • Any SMF data needed to be collected.
  • Any other monitoring.
  • How much additional CPU will be needed – at first, and in the long term.

Making your product secure

Many sites are ultra careful about keeping their system secure. The philosophy is give a user access for what they need to do – but no more. For example

  • They will not be comfortable installing a non IBM SVC into their system. An SVC can be used from any address space, so if there is any weakness in the SVC it could be exploiter.
  • Using id(0) (superuser) in Unix Services is not allowed. The userid needs to be given specific permission. If the code “changes userid” then services like pthread_security_applid_np() should be used; where the applid is part of the configuration. Alternatives include __login_applid. End users of this facility will need read access to the specific applid.

TLS and SSL

If you are using TLS there are other considerations

  • Any certificate you generate needs a long validity date, and JCL to recreate it when it expires.
  • If you create a Certificate Authority you need to document how to export it and distribute it to other platforms
  • Browsers and application may verify the host name, so you need to generate a certificate with a valid name. The external z/OS name may be different from the internal name.
  • You should support TLS V1.2 and TLS 1.3 Other TLS and SSL versions are deprecated.
  • It is good practice to have one keyring with the server certificate with its private key, and a “common” trust store keyring which has the Certificate Authorities for all the sites connecting to the z/OS image. If you connect to a new site, you update the common keyring, and all applications pick up the new CA. If you have one keyring just for your instance, you need to maintain multiple keyrings when a new certificate is added, one for each application.

Migrating from cc to xlc is like playing twister

I needed to compile a file in Unix System Services; I took an old make file, changed cc to xlc expecting it to compile and had lots of problems.

It feels like the documentation was well written in the days of the cc and c89 complier, and has a different beast inserted into it.

As started to write this blog post, I learned even more about compiling in Unix Services on z/OS!

Make file using cc 

cparmsa= -Wc,"SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE 
cparmsb= ,SO,SHOW,LIST(),XREF,ILP32,DLL,SKIPS(HIDE)" 
syslib= -I'/usr/include' -I'/usr/include/sys'  -I"//'TCPIP.SEZACMAC'" -I"//'TCPIP.SEZANMAC'" 
all: main 
parts =  tcps.o 
main: $(parts)
  cc -o tcps  $(parts) 
                                                                                                                            
%.o: %.c 
 cc  -c -o $@   $(syslib) $(cparmsa)$(cparmsb)    -V          $< 
 
clean: 
 rm  *.o

The generated compile statement is

cc -c -o tcps.o -I’/usr/include’ -I’/usr/include/sys’ -I”//’TCPIP.SEZACMAC'” -I”//’TCPIP.SEZANMAC'” -Wc,”SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE,SO, SHOW,LIST(),XREF,ILP32,DLL,SKIPS(HIDE)” -V tcps.c

Note the following

  • the -V option generates the listing. “-V produces all reports for the compiler, and binder, or prelinker, and directs them to stdout“. If you do not have -V you do not get a listing.
  • -Wc,list() says generate a list with a name like tcps.lst based on the file name being compiled. If you use list(x.lst) it does not produce any output! This is contrary to what the documentation says. (Possible bug on compiler when specifying nooe”
  • SHOW lists the included files
  • SKIPS(HIDE) omits the stuff which is not used – see below.

Make using xlc

I think the xlc compiler has bits from z/OS and bits from AIX (sensibly sharing code!). On AIX some parameters are passed using -q. You might use -qSHOWINC or -qNOSHOWINC instead of -Wc,SHOWINC

cparmsx= -Wc,"SO,SHOW,LIST(lst31),XREF,ILP32,DLL,SSCOM, 
cparmsy= DEBUG,DEF(MVS),DEF(_OE_SOCKETS),UNDEF(_OPEN_DEFAULT),NOOE" 
cparms3= -qshowinc -qso=./lst.yy  -qskips=hide -V 
syslib= -I'/usr/include' -I'/usr/include/sys'  -I"//'TCPIP.SEZACMAC'" -I"//'TCPIP.SEZANMAC'" 
all: main 
parts =  tcps.o 
main: $(parts) 
  cc -o tcps  $(parts) 
                                                                                                      
%.o: %.c 
 xlc -c -o $@   $(syslib) $(cparmsx)$(cparmsy) $(cparms3)     $< 
                                                                                                      
clean: 
 rm  *.o

This generates a statement

xlc -c -o tcps.o -I’/usr/include’ -I’/usr/include/sys’ -I”//’TCPIP.SEZACMAC'” -I”//’TCPIP.SEZANMAC'” -Wc,”SO,SHOW,LIST(lst31),XREF, ILP32,DLL, SSCOM,DEBUG,DEF(MVS),DEF(_OE_SOCKETS), UNDEF(_OPEN_DEFAULT),NOOE” -qshowinc -qso=./lst.yy -qskips=hide tcps.c

Note the -q options. You need -qso=…. to get a listing.

Any -V option is ignored, and LIST(…) is not used.

Note: There is a buglet in the compiler, specifying nooe does not always produce a listing. The above xlc statement gets round this problem.

SKIPS(SHOW|HIDE)

The SKIPS(HIDE) shows you what is used, and surpresses text which is not used. I found this useful trying to find the combination of #define … to get the program to compile.

For example with SKIPS(SHOW)

170 |#if 0x42040000 >= 0X220A0000                               | 672     4      
171 |    #if defined (_NO_PROTO) &&  !defined(__cplusplus)      | 673     4      
172 |        #define __new210(ret,func,parms) ret func ()       | 674     4      
173 |    #else                                                  | 675     4      
174 |        #define __new210(ret,func,parms) ret func parms    | 676     4      
175 |    #endif                                                 | 677     4      
176 |#elif !defined(__cplusplus) && !defined(_NO_NEW_FUNC_CHECK)| 678     4      
177 |       #define __new210(ret,func,parms) \                  | 679     4      
178 |        extern struct __notSupportedBeforeV2R10__ func     | 680     4      
179 |   #else                                                   | 681     4      
180 |     #define __new210(ret,func,parms)                      | 682     4      
181 |#endif                                                     | 683     4

With SKIPS(HIDE) the bold lines are not displayed, 

170 |#if 0x42040000 >= 0X220A0000                              | 629     4 
171 |    #if defined (_NO_PROTO) &&  !defined(__cplusplus)     | 630     4 
172 |        #define __new210(ret,func,parms) ret func ()      | 631     4 
173 |     else                                                 | 632     4 
175 |    #endif                                                | 633     4 
176 |                                                          | 634     4 
179 |   #else                                                  | 635     4 
181 |#endif                                                    | 636     4 
182 |#endif                                                    | 637     4

This shows

  • 170 The line number within the included file
  • 629 The line number within the file
  • 4 is the 4th included file. In the “I N C L U D E S” section it says 4 /usr/include/features.h
  • rows 174 is missing … this is the #else text which was not included
  • rows 177, 178,180 are omitted.

This makes is much easier to browse through the includes to find why you have duplicate definitions and other problems.

Tracing AT-TLS on z/OS

AT-TLS on z/OS provides TLS support for applications by magically inserting itself into an application using TCP/IP, without changing the application.

You can collect a trace of AT-TLS starting up, but I was interested in tracing the handshake.

  • If syslogd (system wide program for collecting log data) is active, then trace will be written to the Unix file system.
  • if syslogd is not active then the data is written to syslog.
  • You can configure it so errors get written to syslog and syslogs.

My server COLATTLS started task is a program acting as a TCP/IP program, with Program Control, so my application gets to interface with AT-TLS, extract information and control the connection.

AT-TLS definitions

In my AT-TLS definitions I had

TLSConnectionAction              TNCA 
{ 
  TTLSCipherParmsRef              TLS13TLS12 
  TTLSSignatureParmsRef           TNESigParms 
  TTLSConnectionAdvancedParmsRef  TNCOonAdvParms 
  CtraceClearText                 Off 
  Trace                          255 
}

This trace statement traces everything. See below for a description of what is traced.

Using syslogd

Syslogd is a daemon for applications, they write data to syslogd, and you configure syslogd to define where the output goes to.

My syslog JCL started task procedure is: 

//SYSLOGD PROC 
//* Read parms from /etc/syslog.conf 
//CONFHFS EXEC PGM=SYSLOGD,REGION=4096K,TIME=NOLIMIT, 
//         PARM='ENVAR("_CEE_ENVFILE_S=DD:STDENV")/-c -i       ' 
//STDENV   DD DUMMY 
//SYSPRINT DD SYSOUT=* 
//SYSIN    DD DUMMY 
//SYSERR   DD SYSOUT=* 
//SYSOUT   DD SYSOUT=* 
//CEEDUMP  DD SYSOUT=*

This reads its control statements from /etc/syslog.conf (the default). See Configuring the syslog daemon. My file has 

*.INETD*.*.*       /var/log/inetd 
auth.* /var/log/auth 
mail.* /var/log//mail -F 640 -D 770 
local1.err       /var/log/local1 
*.err            /var/log/errors 
*.CPAGENT.*.*       /var/log/CPAGENT.%Y.%m.%d  
*.TTLS*.*.*          /var/log/TTLS.%Y.%m.%d  
*.Pagent.*.*        /var/log/Pagent.%Y.%m.%d  
*.TCPIP.*.debug     /var/log/TCPIPdebug.%Y.%m.%d  
*.TCPIP.*.warning   /var/log/TCPIP.%Y.%m.%d  
*.TCPIP.*.err       /var/log/TCPIPerr.%Y.%m.%d  
*.TCPIP.*.info      /var/log/TCPIPinfo.%Y.%m.%d  
*.SYSLOGD*.*.*      /var/log/syslogd.%Y.%m.%d  
*.TN3270*.*.*       /var/log/tn3270 
*.SSHD*.*.*         /var/log/SSHD

The output for *.TCPIP.*.debug goes to a file like /var/log/TCPIPdebug.2023.04.03

The configuration says, for example,

  • the output from TCPIP, with priority code debug or less goes to file /var/log/TCPIPdebug…
  • the output from TCPIP, with priority code info or less goes to file /var/log/TCPINFO…
  • the output from TN3270 goes to /var/log/tn3270 – for all priorities.

Because “debug” is debug or lower, the file will also contain the “info” messages. Some messages are written to multiple files.

Note: although my application started task was called COLATTLS, the ATTLS trace came out from job TCPIP, not COLATTLS.

AT-TLS trace

The trace for application is configured with the TRACE option in definitions. The documentation says (TTLSGroupAction, and TLSEnvironmentAction):

Trace

Specifies the level of AT-TLS tracing. The valid values for n are in the range 0 – 255. The sum of the numbers associated with each level of tracing selected is the value that should be specified as n. If n is an odd number, errors are written to joblog and all other configured traces are sent to syslogd. If this value is specified on the TTLSEnvironmentAction statement, it is used instead of the value from the TTLSGroupAction statement referenced by the same TTLSRule statement.

  • 0 – No tracing is enabled.
  • 1 (Error) – Errors are traced to the TCP/IP joblog
  • 2 (Error) – Errors are traced to syslogd. The messages are issued with syslogd priority code err.
  • 4 (Info) – Tracing of when a connection is mapped to an AT-TLS rule and when a secure connection is successfully initiated is enabled. The messages are issued with syslogd priority code info.
  • 8 (Event) – Tracing of major events is enabled. The messages are issued with syslogd priority code debug.
  • 16 (Flow) – Tracing of system SSL calls is enabled. The messages are issued with syslogd priority code debug.
  • 32 (Data) – Tracing of encrypted negotiation and headers is enabled. This traces the negotiation of secure sessions. The messages are issued with syslogd priority code debug.

This means that if tracing the negotiation, it will be written with priority debug. From the *.TCPIP.debug statement in my syslogd definitions, the output will be written to /var/log/TCPIPdebug… .

Info output

The information in the info output looks like (two records for one connections):

Apr 2 17:25:53 S0W1 TTLS[16842781]: 17:25:53 TCPIP
EZD1281I TTLS Map CONNID: 00000032 LOCAL: 10.1.1.2..4000
REMOTE: 10.1.0.2..60742 JOBNAME: COLATTLS USERID: START1
TYPE: InBound STATUS: Appl Control RULE: COLATTLJ ACTIONS:
TNGA TNEA TNCA

This gives information on which rule was selected. For example it gives the local and remote ip address and port; job name and userid. It shows that rule COLATTLJ was used with group TNGA, environment TNEA, and connection TNCA .

Apr 2 17:25:53 S0W1 TTLS[16842781]: 17:25:53 TCPIP
EZD1283I TTLS Event GRPID: 00000007 ENVID: 00000003 CONNID: 00000032
RC: 0 Initial Handshake 0000005011440BB0
0000005011422870 TLSV1.2 C02C

This shows that for the same session (TTLS[16842781]) the initial handshake agreed on the TLS level conversation was at TLS V1.2 and the cipher spec(C02C).

Debug output

For one connection, there were over 130 lines out output in the file.

Some example lines are

EZD1283I TTLS Event GRPID: 00000007 ENVID: 00000000 CONNID: 00000032
RC: 0 Connection Init

EZD1284I TTLS Flow GRPID: 00000007 ENVID: 00000004 CONNID: 00000032
RC: 0 Set GSK_KEYRING_FILE(201) start1/TN3270

…

EZD1282I TTLS Start GRPID: 00000007 ENVID: 00000003 CONNID:
00000032 Initial Handshake ACTIONS: TNGA TNEA TNCA
HS-ServerWithClientAuth

EZD1285I TTLS Data CONNID: 00000032 RECV CIPHER 1603010116
EZD1285I TTLS Data CONNID: 00000032 RECV CIPHER 0100011203031FDDC…
EZD1285I TTLS Data CONNID: 00000032 SEND CIPHER16030309BC0200005…

… RC: 0 Call GSK_SECURE_SOCKET_INIT – 0000005011440BB0
… RC: 0 Get GSK_CONNECT_SEC_TYPE(208) – TLSV1.2
… RC: 0 Get GSK_CONNECT_CIPHER_SPEC(207) – C02C

You get

  • Events – (trace 8 event)
  • the traffic data flowing up and down (trace 32 data)
  • the System SSL calls (with return code) (trace 16 flow)

Trace output on syslog – when syslogd not active

Having AT-TLS writing to syslog is not a good idea – it can produce a lot of output. It may be acceptable on a small, low activity, system, tracing the minimum amount of data.

IEF403I COLATTLS - STARTED - TIME=17.16.51                             
BPXF024I (TCPIP) Apr  2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP 
EZD1281I TTLS Map   CONNID: 0000002F LOCAL: 10.1.1.2..4000 REMOTE:     
10.1.0.2..43012 JOBNAME: COLATTLS USERID: START1 TYPE: InBound         
STATUS: Appl Control RULE: COLATTLJ ACTIONS: TNGA TNEA TNCA            
BPXF024I (TCPIP) Apr  2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP 
EZD1283I TTLS Event GRPID: 00000007 ENVID: 00000000 CONNID: 0000002F   
RC:    0 Connection Init                                               
BPXF024I (TCPIP) Apr  2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP 
EZD1282I TTLS Start GRPID: 00000007 ENVID: 00000001 CONNID: 00000000   
Environment Create ACTIONS: TNGA TNEA **N/A**                          
BPXF024I (TCPIP) Apr  2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP 
EZD1283I TTLS Event GRPID: 00000007 ENVID: 00000002 CONNID: 00000000   
RC:    0 Environment Master Create 00000001                            
BPXF024I (TCPIP) Apr  2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP    
EZD1284I TTLS Flow  GRPID: 00000007 ENVID: 00000002 CONNID: 0000002F   
RC:    0 Call GSK_ENVIRONMENT_OPEN - 0000005011421D10
...

The output was produced with AT-TLS trace was enable, and ATTLS was not using the syslogd daemon.

The text in bold is the initial trace entry.

  • BPXF024I (TCPIP) Apr 2 17:17:03 TTLS 16842781 : 17:17:03 TCPIP is written because syslogd is not being used.
  • EZD1281I TTLS Map CONNID: 0000002F LOCAL: 10.1.1.2..4000 REMOTE: 10.1.0.2..43012 JOBNAME: COLATTLS USERID: START1 TYPE: InBound provides information about which AT-TLS rule is being used for the connection.
  • EZD1284I TTLS Flow GRPID: 00000007 ENVID: 00000002 CONNID: 0000002F RC: 0 Call GSK_ENVIRONMENT_OPEN – 0000005011421D10 shows you information about the system ssl call being used.

Using certificates to logon to z/OS.

I was testing out certificate access to logon to z/OS using pthread_secure_applid() services . It took a little while to get my program working, but once it was working I tried to be “customer like” rather than a simple Unit Test. The documentation is not very clear, but it does work.

You can create a RACF definition for the simple case

  • For a certificate where the Subject Distinguished Name(DN) matches the definition’s Subject DN, then use userid xyz.
  • For a certificate where part of the Subject DN matches the definitions Subject DN, then use userid ijklm.

These both default to using an APPLID OMVSAPPL

I wanted to say

  • for application id ZZZ, if a certificate with this subject DN, then use userid TESTZ
  • for application id YYY if a certificate with the same subject DN, then use userid TESTY

Topics covered n this post:

How to set up your enterprise certificates

If you want to control certificate to userid mapping, at an individual level

You can use the following, where SDN is the Subject Distinguished Name from the certificate,

RACDCERT MAP ID(....  ) SDN(...)

for each individual.

To remove access you just delete the profile (and refresh RACLIST).

You can also specify the Issuers Distinguished Name (IDN) so you refer to the correct certificate.

If you want to control certificate to userid mapping, at the group level

You can use a Subject DN filter. This may mean you need to define your Subject DN with Organisation Units(OU), as

  • CN=COLIN,OU=TEST,O=MYORG
  • CN=MARY,OU=SALES,O=MYORG

You can then have a filter SDN(‘OU=TEST,O=MYORG’) to allow just those in the OU=TEST group to logon, and sales people will not get access.

If you then want to prevent individuals from getting access you need to define a specific profile, and point it to a userid which can do nothing.

You could also have your certificates issued by different CA’s For example, have a certificate with an Issuer Distinguished Name(IDN) including the department name.

  • Subject DN(CN=COLIN,OU=TEST,O=MYORG) Issuer DN(OU=TEST,OU=CA,O=MYORG)

and specify IDN(OU=TEST,OU=CA,O=MYORG).

Note: with this, if someone changes department, and moved from Sales to Test, they will need a new certificate.

Some people may require more than one certificate. For example someone who has normal access for their day to day job, and super powers for emergencies.

Note: In addition to getting access with a certificate you may still want to use a password.

Define a certificate to user mapping (simple case)

Use a fully qualified Subject DN filter

For example, a certificate has SDN CN=docec256.O=Doc.C=GB .

//COLRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//STEPLIB  DD  DISP=SHR,DSN=SYS1.MIGLIB 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 

RACDCERT DELMAP(LABEL('CP'))  ID(COLIN) 
RACDCERT MAP ID(COLIN  )  - 
   SDNFILTER('CN=docec256.O=Doc.C=GB')                  - 
   IDNFILTER('CN=SSCA256.OU=CA.O=DOC.C=GB')             - 
   WITHLABEL('CP') 
RACDCERT LISTMAP ID(COLIN) 
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH 
/*

This uses both Subject Distinguished Name, and Issuer Distinguished Name. You can use either or both filters.

With this definition when I used openssl s_client to talk to my server application running on z/OS. When I specified the client certificate with the specified Subject DN and signer, I could use:

  • AT-TLS, use BPX1IOC() and retrieve the userid. Internally this uses applid=OMVSAPPL
  • do it myself
    • retrieve the certificate with the BPX1IOC() TTLS_QUERY_ONLY call;
    • use rc = pthread_security_applid_np( __CREATE_SECURITY_ENV,…. “OMVSAPPL”) passing the certificate and the applid OMVSAPPL

The code checks my the extracted userid (COLIN) has read access to the profile OMVSAPPL in class(APPL)) and, if it has access, it returns the userid to the application.

See below if you do not want to use applid OMVSAPPL

Use a partially qualified Subject DN filter

Instead of a fully qualified Subject DN, CN=docec256.O=Doc.C=GB. You can use a partially qualified DN, for example CN=docec256.O=Doc.C=GB. Any certificate which has a DN including O=Doc.C=GB will be accepted

RACDCERT DELMAP(LABEL('CPGEN'))  ID(ADCDF) 
RACDCERT MAP ID(ADCDF  )  - 
   SDNFILTER('O=Doc.C=GB')                  - 
   WITHLABEL('CPGEN') 
RACDCERT LISTMAP ID(ADCDF) 
SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

With this definition when I used openssl s_client to talk to my server application running on z/OS. When I specified the client certificate with the specified Issuer CN, it worked the same way as the simple case above.

The code checks my the user extracted userid (ADCDF) has read access to the profile OMVSAPPL in class(APPL)) and, if it has access, it returns the userid to the application.

Define a certificate with mutliid mapping

A specific Subject DN gets a different userid depending on the application id.

As above you can specify an Issuer Distinguished Name or a Subject Distinguished Name.

Use an Issuer Distinguished Name

Use an Issuer certificate, so any certificate issued by DN this will be covered.

I defined the profile for my Issuer CA certificate. The definition below says – any certificate issued by this CA.

RACDCERT DELMAP(LABEL('CPMULTI')) MULTIID 

RACDCERT MULTIID MAP WITHLABEL('CPMULTI') TRUST - 
  IDNFILTER('CN=SSCA256.OU=CA.O=DOC.C=GB')      - 
  CRITERIA(ZZAPPLID=&APPLID)

Define which userid to use based on the CRITeria

RDEFINE DIGTCRIT ZZAPPLID=ZZZ  APPLDATA('ADCDZ') 
RDEFINE DIGTCRIT ZZAPPLID=AAA  APPLDATA('ADCDF')

In my program I had 

struct __certificate  ct; 
ct.__cert_type = __CERT_X509; 
char * pData = (char *)  ioc.TTLSi_BufferPtr; 
       
ct.__cert_length = ...
ct.__cert_ptr    = ...;
rc = pthread_security_applid_np(__CREATE_SECURITY_ENV,
         __CERTIFICATE_IDENTITY, 
         sizeof(ct), 
         &ct, 
         "", 
         0,
        "AAA");

With this, the CRITERIA(ZZAPPLID=&APPLID) becomes CRITERIA(ZZAPPLID=AAA), which maps to CRITeria

RDEFINE DIGTCRIT ZZAPPLID=AAA  APPLDATA('ADCDF'),

and so maps to userid ADCDF.

When applid ZZZ was used instead of AAA, then

RDEFINE DIGTCRIT ZZAPPLID=AAA  APPLDATA('ADCDF')

AT-TLS only seems to be able to use the APPL OMVSAPPL (the default). I could not find a way of getting it to use an APPLID, so I had to use pthead_security_appl_np to be able to use an applid different from the default.

Use a subject DN

You can use an explicit Subject DN

 RACDCERT MULTIID MAP WITHLABEL('CPMULTIU') TRUST - 
     SDNFILTER('CN=docecgen.O=Doc2.C=GB')            - 
   CRITERIA(UAPPLID=&APPLID) 

 RDEFINE DIGTCRIT UAPPLID=AAA  APPLDATA('ADCDA') 
 RDEFINE DIGTCRIT UAPPLID=BBB  APPLDATA('ADCDB')

the userid used depending which APPLID was specified in my application.

I could use a subset of the the SDN

RACDCERT DELMAP(LABEL('CPMULTIU')) MULTIID 
RDELETE DIGTCRIT APPLID=OMVSAPPL 
RDELETE DIGTCRIT APPLID=ZZZ 
                                                                   
RACDCERT MULTIID MAP WITHLABEL('CPMULTIU') TRUST - 
    SDNFILTER('O=Doc2.C=GB')            - 
  CRITERIA(UAPPLID=&APPLID) 

RDEFINE DIGTCRIT UAPPLID=AAA  APPLDATA('ADCDD') 
RDEFINE DIGTCRIT UAPPLID=BBB  APPLDATA('ADCDE')

Creating definitions

What the documentation does not say, is that you can have any keyword in the criteria, as long as the substitute value has a DIGTCRIT.

This means you can have 

RACDCERT MULTIID MAP ...    CRITERIA(ZORK=&APPLID)

and have statements like

RDEFINE DIGTCRIT ZORK=AAA  APPLDATA('IBMUSER')

Controlling access to the applid

If there is a profile for the applid in CLASS(APPL) the userid will need read access to the profile.

If the profile is not defined, then anyone can use the profile.

Problem I experienced

ESRCH: errno 143 EDC5143I No such process. errno2 0be8044c

I got messages

ESRCH:The user ID provided as input is not defined to the security product or does not have an OMVS segment defined
pthread_s… rc = -1 errno 143 EDC5143I No such process. errno2 0be8044c

Where TSO BPXMTEXT 0be8044c gives

BPXPTSEC 01/05/18
JRNoCertforUser: There is no userid defined for this certificate. Action: Ensure the userid is known to the SAF service.

Cause 1:

I used an APPLID which did not have a criteria

I also got this when I used pthread_security_applid_np(…) with an an applid which did not have a matching RDEFINE DIGTCRIT

For example the following worked

  • applid AAA and RDEFINE DIGTCRIT UAPPLID=AAA APPLDATA(‘ADCDA’)

when I used a different applid I got the “EDC5143I No such process. errno2 0be8044c” message, along with the following on the joblog.

ICH408I USER(COLIN ) GROUP(SYS1 ) NAME(COLIN PAICE )
DIGITAL CERTIFICATE IS NOT DEFINED. CERTIFICATE SERIAL NUMBER(027C)
SUBJECT(CN=docecgen.O=Doc2.C=GB) ISSUER(CN=SSCA256.OU=CA.O=DOC.C=GB).

Cause 2:

In the definitions I had the wrong case: c=GB and c=gb.

Cause 3:

NOTRUST was specified

RACDCERT MULTIID MAP WITHLABEL('CPMULTIU') noTRUST -
SDNFILTER('CN=docecgen.O=Doc2.C=GB') -
CRITERIA(UAPPLID=&APPLID)

This also had a message on the job log:

ICH408I USER(START1 ) GROUP(SYS1 ) NAME(####################)
DIGITAL CERTIFICATE DEFINED TO A RESERVED USER ID. CERTIFICATE SERIAL
NUMBER(027C) SUBJECT(CN=docecgen.O=Doc2.C=GB) ISSUER(CN=SSCA256.OU=C
A.O=DOC.C=GB).

Cause 4:

I had specified the certificate’s DN in Issuers DN. It should have been SDNFILTER(‘CN=rsarsa.O=cpwebuser.C=GB’)

EMVSSAFEXTRERR 163 EDC5163I SAF/RACF extract error. errno2 0be8081c

pthread_secuity_applid_np() rc = -1 errno 163 EDC5163I SAF/RACF extract error. errno2 0be8081c

TSO BPXMTEXT 0be8081c , just gives

BPXPTSEC 01/05/18

with no message description

Cause 1:

I got this when I used 

RACDCERT MULTIID MAP WITHLABEL('CPMULTIU') TRUST - 
    SDNFILTER('CN=docecgen.O=Doc2.C=GB')            - 
  CRITERIA(UAPPLID=&APPLID)

and the SDNFILTER value was in upper case. When I corrected it (to the above) it worked.

It looks like certificate filter not found.

Cause 2:

I had a lower case keyword such as SDNFILTER(‘CN=docecgen.O=Doc2.c=GB’)

EMVSSAFEXTRERR 163 EDC5163I SAF/RACF extract error. errno2 0be80820

TSO BPXMTEXT 0be8082 gave

BPXPTSEC 01/05/18

and no other information.

I had 

RACDCERT MULTIID MAP WITHLABEL('CPMULTIU') TRUST - 
    SDNFILTER('CN=docecgen.O=Doc2.C=GB')            - 
  CRITERIA(UAPPLID=&APPLID) 

RDEFINE DIGTCRIT UAPPLID=ZZZ  APPLDATA('ADCDB')

and userid ADCDB had no access to the RACF resource class(APPL) ZZZ.

EMVSERR 157 EDC5157I An internal error has occurred. errno2 0be800fc.

TSO BPXMTEXT 0be800fc gave

BPXPTSEC 01/05/18
JRSAFNoUID: The user ID has no UID

Action: Create an OMVS segment with a UID.

Cause:

If I display the userid LU ADCDG OMVS it gave UID= NONE.

I had created this situation using ALU ADCDG OMVS(NOUID). I reset it back using tso ALU ADCDG OMVS(UID(990098)).

How to import a certificate from openssl into z/OS

This question came up in an email exchange after someone had upgraded openssl from 1.1.1f to v3.

There are two format certificates – text, and binary.

Text certificate

The text certificate looks like

Certificate:                                        
    Data:                                           
        Version: 3 (0x2)                            
        Serial Number: 633 (0x279)                  
        Signature Algorithm: ecdsa-with-SHA384      
...
-----BEGIN CERTIFICATE-----                                      
MIICgTCCAiegAwIBAgICAnkwCgYIKoZIzj0EAwMwOjELMAkGA1UEBhMCR0IxDDAK 
BgNVBAoMA0RPQzELMAkGA1UECwwCQ0ExEDAOBgNVBAMMB1NTQ0EyNTYwHhcNMjMw 
MzE5MTIzODA1WhcNMjQwMTMwMTY0NjAwWjAuMQswCQYDVQQGEwJHQjEMMAoGA1UE 
CgwDRG9jMREwDwYDVQQDDAhkb2NlYzI1NjBZMBMGByqGSM49AgEGCCqGSM49AwEH 
...
-----END CERTIFICATE-----

This has both a text version, and a base 64 encoded version within it.

On z/OS create a dataset with this file. Then use JCL like 

//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RACDCERT CHECKCERT('COLIN.DOCEC256.NEW.PEM')

This checks the file is usable, and you can check the contents before you install it on your system.

It produced output like

Certificate 1:                                               
                                                             
  Start Date: 2023/03/19 12:38:05                            
  End Date:   2024/01/30 16:46:00                            
  Serial Number:                                             
       >0279<                                                
  Issuer's Name:                                             
       >CN=SSCA256.OU=CA.O=DOC.C=GB<                         
  Subject's Name:                                            
       >CN=docec256.O=Doc.C=GB<                              
  Subject's AltNames:                                        
    IP: 127.0.0.1                                            
    Domain: localhost                                        
  Signing Algorithm: sha384ECDSA                             
  Key Usage: HANDSHAKE, DATAENCRYPT, DOCSIGN, KEYAGREE       
  Key Type: NIST ECC                                         
  Key Size: 256    
...

It also gave me 

Chain information:                                           
  Chain contains 1 certificate(s), chain is incomplete

This message is because “A certificate chain is considered incomplete if RACF is unable to follow the chain back to a self-signed ‘root’ certificate”.

Which was true, RACF already had the CA in its database, and the CA certificate was not in the file.

When the file had already been imported I also got

Certificate 1:                                                     
Digital certificate information for CERTAUTH:
  Label: Linux-CARSA

So I know it had already been imported into RACF.

I also got

Chain contains expired certificate(s)

So I could tell I needed to get a newer certicate.

Add it to the RACF key store

There is no add-replace, so you have to delete it, then add it

RACDCERT DELETE - 
  (LABEL('DOCEC256')) ID(COLIN) 
RACDCERT ID(COLIN)    ADD('COLIN.DOCEC256.NEW.PEM') - 
  WITHLABEL('DOCEC256') 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

The first time I ran this I got 

RACDCERT DELETE   (LABEL('DOCEC256')) ID(COLIN)                                                           
IRRD107I No matching certificate was found for this user.                                                 
READY                                                                                                     
RACDCERT ID(COLIN)    ADD('COLIN.DOCEC256.NEW.PEM')   WITHLABEL('DOCEC256')                               
IRRD199I Certificate with label 'DOCEC256' is added for user COLIN.                                       
IRRD175I The new profile for DIGTCERT will not be in effect until a SETROPTS REFRESH has been issued.

The second time I ran it I got 

RACDCERT DELETE   (LABEL('DOCEC256')) ID(COLIN)                                                                   
IRRD176I RACLISTed profiles for DIGTCERT will not reflect the deletion(s) until a SETROPTS REFRESH is issued.     
READY                                                                                                             
RACDCERT ID(COLIN)    ADD('COLIN.DOCEC256.NEW.PEM')   WITHLABEL('DOCEC256')                                       
IRRD199I Certificate with label 'DOCEC256' is added for user COLIN.                                               
IRRD175I The new profile for DIGTCERT will not be in effect until a SETROPTS REFRESH has been issued.

Refresh the RACLISTed data

If the classes are RACLISTed (cached in memory), you need 

SETROPTS RACLIST(DIGTNMAP, DIGTCRIT) REFRESH

What happens if it is already in the key store under a different userid?

I tried adding it for a different userid and got

RACDCERT ID(ADCDA) ADD('COLIN.DOCEC256.NEW.PEM') WITHLABEL('DOCEC256')
IRRD109I The certificate cannot be added. Profile 0279.CN=SSCA256.OU=CA.O=DOC.C=GB is already defined.

Unfortunately it does not tell you the id it has already been defined for.

The only way I found of finding this information is

  • displaying all resources under class(DIGTCERT).
  • find the resource with the matching name.

For example

//COLRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=* 
//SYSTSPRT DD SYSOUT=* 
//SYSTSIN DD * 
RLIST DIGTCERT  * 
/* 
//

gave me output like

CLASS      NAME                                                  
-----      ----                                                  
DIGTCERT   0279.CN=SSCA256.OU=CA.O=DOC.C=GB                      
                                                                 
...                                                           
                                                                 
APPLICATION DATA                                                 
----------------                                                 
ADCDF

the current owner is ADCDF.

The serial number of the certificate is 0279, and the components are separated with periods ‘.’.

If the name has a blank in it you will get a value like

COLIN4Certification¢Authority.OU=TEST.O=COLIN

where the blank is replaced with the cent sign(¢).

Binary certificate, PKCS12, .P12

You should be able to upload the .p12 file to z/OS as a binary file and import it using the same JCL as above.

However in openssl V3 the packaging of the certificate has changed, and RACF does not yet support it.

The easiest way of getting a certificate into z/OS is to use the .pem file (grin)

How do I trace TCP/IP sockets on z/OS?

I stumbled on this by accident.

In the TCPIP.DATA configuration file you can specify 

 TRACE SOCKET

I copied the configuration file, and made the change. I used it in my JCL

//SYSTCPD DD DISP=SHR,DSN=USER.Z24C.TCPPARMS(MYDATA)

Note if you use TRACE SOCKET in the configuration file used by every one – then every one will get their sockets traced – which may not be what you want.

The output to SYSPRINT is like

request = HCreate                                                                         
                                                                                          
EZY3829I  pre   0xe3e2d9c2 00c00001 00010000 00000020 e3c3d7c9 d7404040 00000000 00000000 
EZY3830I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3831I        0x00000000 1fa77318 00000000 00000000 00000000 00000080 00000000 00000000 
EZY3832I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3833I        0xffff0002 00000000 00000000 40404040 40404040 f18681f7 f68686f8 00000000 
EZY3834I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
                                                                                          
request = HCreate                                                                         
                                                                                          
EZY3835I  post  0xe3e2d9c2 00c00001 00010000 00000020 e3c3d7c9 d7404040 00000000 00000000 
EZY3830I        0x7f5ec0f0 00010000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3831I        0x00000000 1fa77318 00000000 00000000 00000000 00000080 00000000 00000000 
EZY3832I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
EZY3833I        0xffff0002 00000031 00000000 40404040 40404040 f18681f7 f68686f8 00000000 
EZY3834I        0x00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

These messages and the content are not documented – they are for IBM Software Support.

Compiling the TCP/IP samples on z/OS

Communications server (TCPIP) on z/OS provides some samples. I had problems getting these to compile, because the JCL in the documentation was a) wrong and b) about 20 years behind times.

Samples

There are some samples in TCPIP.SEZAINST

  • TCPS: a server which listens on a port
  • TCPC: a client which connects to a server using IP address and port
  • UDPC: C socket UDP client
  • UDPS: C socket UDP server
  • MTCCLNT: C socket Multitasking client
  • MTCSRVR: C socket Multitasking server
  • MTCCSUB: C socket subtask MTCCSUB

The JCL I used is

//COLCOMPI   JOB 1,MSGCLASS=H,COND=(4,LE) 
//S1          JCLLIB ORDER=CBC.SCCNPRC 
// SET LOADLIB=COLIN.LOAD 
// SET LIBPRFX=CEE 
// SET SOURCE=COLIN.C.SOURCE(TCPSORIG) 
//COMPILE  EXEC PROC=EDCCB, 
//       LIBPRFX=&LIBPRFX, 
//       CPARM='OPTFILE(DD:SYSOPTF),LSEARCH(/usr/include/)', 
// BPARM='SIZE=(900K,124K),RENT,LIST,RMODE=ANY,AMODE=31' 
//COMPILE.SYSLIB DD 
//               DD 
//               DD DISP=SHR,DSN=TCPIP.SEZACMAC 
//*              DD DISP=SHR,DSN=TCPIP.SEZANMAC  for IOCTL 
//COMPILE.SYSOPTF DD * 
DEF(_OE_SOCKETS) 
DEF(MVS) 
LIST,SOURCE 
TEST 
RENT ILP32        LO 
INFO(PAR,USE) 
NOMARGINS EXPMAC   SHOWINC XREF 
LANGLVL(EXTENDED) sscom dll 
DEBUG 
/* 
//COMPILE.SYSIN    DD  DISP=SHR,DSN=&SOURCE 
//BIND.SYSLMOD DD DISP=SHR,DSN=&LOADLIB. 
//BIND.SYSLIB  DD DISP=SHR,DSN=TCPIP.SEZARNT1 
//             DD DISP=SHR,DSN=&LIBPRFX..SCEELKED 
//* BIND.GSK     DD DISP=SHR,DSN=SYS1.SIEALNKE 
//* BIND.CSS    DD DISP=SHR,DSN=SYS1.CSSLIB 
//BIND.SYSIN DD * 
  NAME  TCPS(R) 
//START1   EXEC PGM=TCPS,REGION=0M, 
// PARM='4000          ' 
//STEPLIB  DD DISP=SHR,DSN=&LOADLIB 
//SYSERR   DD SYSOUT=*,DCB=(LRECL=200) 
//SYSOUT   DD SYSOUT=*,DCB=(LRECL=200) 
//SYSPRINT DD SYSOUT=*,DCB=(LRECL=200)

Change the source

The samples do not compile with the above JCL. I needed to remove some includes

#include <manifest.h> 
// #include <bsdtypes.h> 
#include <socket.h> 
#include <in.h> 
// #include <netdb.h> 
#include <stdio.h>

With the original sample I got compiler messages

ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):66 Identifier dev_t has already been defined on line 98 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):77 Identifier gid_t has already been defined on line 101 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.SYS.H(TYPES):162 Identifier uid_t has already been defined on line 100 of “TCPIP.SEZACMAC(BSDTYPES)”.
ERROR CCN3334 CEE.SCEEH.H(NETDB):87 Identifier in_addr has already been defined on line 158 of “TCPIP.SEZACMAC(IN)”.


INFORMATIONAL CCN3409 TCPIP.SEZAINST(TCPS):133 The static variable “ibmcopyr” is defined but never referenced.

I tried many combinations of #define but could not get it to compile, unless I removed the #includes.

Compile problems I stumbled upon 

Identifier dev_t has already been defined on line ...                                                     
Identifier gid_t has already been defined on line ...                                                     
Identifier uid_t has already been defined on line ....

This was caused by the wrong libraries in SYSLIB. I needed

  • CEE.SCEEH.H
  • CEE.SCEEH.SYS.H
  • TCPIP.SEZACMAC
  • TCPIP.SEZANMAC

The compile problems were caused by CEE.SCEEH.SYS.H being missing.

Execution problems

I had some strange execution problem when I tried to use AT-TLS within the program.

EDC5000I No error occurred. (errno2=0x05620062)

The errno2 reason from TSO BPXMTEXT 05620062 was

BPXFSOPN 04/27/18
JRNoFileNoCreatFlag: A service tried to open a nonexistent file without O_CREAT

Action: The open service request cannot be processed. Correct the name or the open flags and retry the operation.

Which seems very strange. I have a feeling that this field is not properly initialised and that this value can be ignored.

How to use display computer hardware in a Libre Office presentation

I was writing a presentation which needed a mixture of computing hardware. There is a Libre Office extension called VRT Network equipment which you can install and have a gallery of hardware, from scanners, printers, firewalls, routers to laptops and tablets and mobile phones.

See VRT Network Equipment Gallery. There are some good instructions here on how to install and use the icons.

Sending an email from z/OS

This started off as a question to me about TLS certificates and ended up with a Python solution.

The short version of this blog post is I could send an email to Gmail using Python – but could not use the IBM CSSMTP solution because it does not support authentication. I was also sent a link to Sending Email from z/OS using Java.

As far as I can tell, SMTP authentication does not use client certificates. If you want to use userid and password, make sure the session is encrypted using TLS.

What types of email are there?

Through JES spool

On my system with id S0W1, I can use TSO XMIT S0W1.IBMUSER and send an email.

The recipient (IBMUSER) can use TSO receive to read the email.

You can use XMIT OVERTHER.MYID to send the email over NJE to other system, such as z/OS, z/VM and z/VSE. You say (on z/OS) xmit node.userid da(‘…’). or xmit node.userid . The JES2 configuration has an IP address and port to send it to a remote spool.

JES2 destinations

You can configure destinations on JES2, and have program (“a writer”) process the node name (destination) data.

You can display the defined destinations using the JES2 command

$DDESTID

Example JCL to write to a destination,

//COPY EXEC PGM=IEBGENER 
//SYSPRINT DD SYSOUT=* 
//SYSIN DD DUMMY 
//SYSUT2 DD SYSOUT=(A,MYDEST),SPIN=UNALLOC 
//SYSUT1 DD * 
line1
/*
//

This writes the data from SYSUT1 to SYOUT2 which goes to nodename MYDEST.

You configure CSSMP and specify ExtWrtName MYDEST, for CSSMTP to read from.

SMTP

There is Simple Mail Transport Protocol. A stream of data is provided such as

MAIL FROM:<ME@MYSYSTEM.COM> 
RCPT TO:<MYBOSS@MYSYSTEM.COMU> 
RCPT TO:<YOU@yoursystem.com> 
DATA 
Blah blah blah. 
QUIT

which is sent to the mail server over TCPIP (usually port 25 or 587).

SMTP is a popular protocol. It can support encrypted session encryption. The protocol can support different levels of authorisation.

z/OS provides CSSMTP which reads from the spool and sends the data over TCPIP to a server. However CSSMTP does not seems to support the passing of authentication information to the server – and as most mail servers want you to authenticate before sending emails – CSSMTP cannot be used for these. I know this from the documentation because the AUTH command is not listed in the list of supported SMTP commands, and AUTH does not work!

Using Python to send an email.

I used Python to send an email to my Gmail account. The script I used is 

#!/usr/bin/python 
import smtplib 
import ssl 
import sys 
from email.mime.text import MIMEText 
host = "173.194.76.108" 
sender = 'colinpaicetest@gmail.com' 
receivers = ['colinpaicemq@gmail.com','colinpaicetest@gmail.com'] 
msg = MIMEText('This is test mail2') 
msg['Subject'] = 'Test mail3' 
msg['From'] = 'colinpaice3@gmail.com' 
msg['To'] ='colinpaicemq@gmail.com , colinpaicetest@gmail.com' 
with smtplib.SMTP(host , 587) as server: 
      # if you need to validate the certificate sent to client.. you need a context
      context=ssl.create_default_context(
                 purpose=ssl.Purpose.CLIENT_AUTH,cafile="pem.pem") 
      server.starttls(context=context) 
      # server.starttls() # for servers that do not send a certificate 
      print(server.esmtp_features) # print info about the connection such as auth types
      server.login('colinpaice3@gmail.com', 'abcpasswordxyz') 
      server.sendmail(sender, receivers, msg.as_string()) 
      print("Successfully sent email")
  • context=ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH,cafile=”pem.pem”) is used to set up the CA cert for verifying any certificate sent from the server. GMAIL does not send a certificate, so the CA is not needed, and you can use server.starttls()
  • server.starttls(context=context) creates the TLS session. This is handled by PAGENT.
  • server.login(‘colinpaice3@gmail.com’, ‘ZYC123….’) does the authentication
  • server.sendmail(sender, receivers, msg.as_string()) sends the message.

The id in the sender must match the id in the login.

This does not use PAGENT or AT-TLS.

In the TLS handshake 57 cipher specs were sent to the server, and cipher spec TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xC02B) was used.

Setting up CSSMTP

Basic configuration

Even though I could not use CSSMTP because I could not authenticate, I learned a lot on my doomed journey.

In the CSSMTP configuration I had 

ExtWrtName    MYDEST             # the external writer name 
TargetServer 
{ 
   TargetIp aaa.bbb.ccc.ddd      # This IP address 
  # TargetName  
   ConnectPort   25              # port to connect to target server 
   Secure        Yes             # Transport Layer Security (TLS) 
} 
...

Each CSSMTP instance reads from one nodeName and sends to one server. If you want to support more nodeNames you need more CSSMTP instances.

Certificates

As part of the TLS handshake, some servers will send down a certificate (to authenticate the server).

If your TLS provider, such as AT-TLS, uses keyrings, you need to import certificate into RACF, and connect the CA certificate to the keyring. Your keyring needs the CA for this certificate. You can get the servers certificate from the network flow, or from an openssl command.

If you are not using keyrings, for example Python uses openssl, you need the certificate in a file in Unix Services.

Getting my z/OS client to talk to a remote server over TLS

I looked into this because someone asked me about using the mail client on z/OS talking to a mail server somewhere in the internet.

My z/OS running on zPDT under Linux did not have a working connection to the outside world.

Configuring Linux to pass the z/OS traffic to the external network.

A ping to the server, got no response back.

Using Wireshark on the wireless connection, I could see my request from z/OS coming via Linux and going to the outside network. The source IP address was 10.1.1.1 – a “local only” address, and so the server could not send a response back to me.

The routing through Linux worked because I had a default route; ip route gave me 

default via 192.168.xxx.xxx4 dev wlxd037450ab7ac proto dhcp metric 600 
10.1.0.0/24 dev eno1 proto kernel scope link src 10.1.0.3 metric 100 
10.1.0.0/24 via 10.1.0.3 dev eno1 proto static metric 100 
10.1.1.0/24 dev tap0 proto kernel scope link src 10.1.1.1

To get the “right” IP address passed to the external network, was surprisingly easy. It was documented in the zPDT book.

I created a script 

echo ‘Your firewall must be enabled for this command to be meaningful’
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -F FORWARD
iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o wlxd037450ab7ac -j MASQUERADE
# iptables -I INPUT -p tcp --dport 3270 -j ACCEPT

and used sudo to execute it.

I started the firewall

sudo ufw status
sudo ufw enable

I could then successfully ping the address

When I used Wireshark on the connection between Linux and z/OS, the source IP address was 10.1.1.2. When I used Wireshark on the wireless connection between Linux and the external network, the IP address was 192.168.xxx.xxx. This shows the effect of the iptables commands, it changed the 10.1.1.2 to a 192.168.xxx.xxx – and back again on the replies.

Getting the server’s certificate

I configured the AT-TLS definitions, for TLS support, specifying the choice of cipher specs I wanted to be used, and specified the keyring.

My z/OS client connected to the remote server. I could see the “client hello” flow, and the “server hello” response. In the server hello flow were the CA certificates the server supported. The client validates the certificates by checking the client’s key store.

I did not have the matching CA certificate in my keystore, and so my client failed to validate the certificate sent down, and the connection ended.

I could do it properly, and get the certificate from the official source for the server, but this was a test system, and I was happy to trust what was sent down in the handshake.

You can use wireshark or openssl to get the certificate

With openssl

openssl s_client -showcerts -tls1_2 -starttls smtp -connect smtp.gmail.com:587 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’

You’ll need to interupt it (Ctrl-C) and paste the data into a file (google.pem)

You can display the contents of the certificate using

openssl x509 -in google.pem -text -noout|less

With Wireshark

display the certificates in the Server Hello flow.

right click on the certificate and use right click, export packet bytes. I created a file user.crt. This is a binary file.

The openssl command (on Linux)

openssl x509 -inform der -in user.crt -text > usercert.pem

displays the file in .pem format.

Put the certificate into the keyring

If you are using something that needs a keyring, you need to import the certificate into RACF and connect it to a keyring. If you are using something that uses openssl, such as Python, you need a .pem file in Unix Services.

I created a file on z/OS (VB) COLIN.USERCERT.PEM, and copied the .pem file into it (between and including the BEGIN CERTIFICATE and END CERTIFICATE lines).

I imported it, and connected it to my keyring using

//IBMRACF  JOB 1,MSGCLASS=H 
//S1  EXEC PGM=IKJEFT01,REGION=0M 
//SYSPRINT DD SYSOUT=*                                           
//SYSTSPRT DD SYSOUT=*                                           
//SYSTSIN DD *                                                   
RACDCERT CHECKCERT('COLIN.USERCERT.PEM')                         
RACDCERT DELETE  (LABEL('USERTRUST.ORG'))    CERTAUTH            
RACDCERT ADD('COLIN.USERCERT.PEM')        -                      
   CERTAUTH  WITHLABEL('USERTRUST.ORG') TRUST 
RACDCERT ID(START1) CONNECT(RING(TN3270) - 
                            CERTAUTH    - 
                            LABEL('USERTRUST.ORG')) 
SETROPTS RACLIST(DIGTCERT,DIGTRING ) refresh                     
                                                                 
RACDCERT LISTRING(TN3270)  ID(START1)                            
/*

Once I had refreshed my PAGENT job, and restarted my client code, I was able to establish a TLS session.

Client certificate

In the TLS handshake, the mail servers did not send down a request for the client certificate. As no certificate was requested, or sent up to the server, no certificate authentication was done. You need to check your mail server for it authentication process – it may just be userid and password. This information is sent once the TLS session has been established – so it flows encrypted.