You’ve been asked to install some files. You are not sure of their provenance, or if they have been changed from what the author wrote, and what you have received. How can you check these files?
You have some systems which are meant to be identical how can you easily check this?
One way of doing these activities is to calculate a digital hash of the file, sign the hash value, and use it.
See background for information on public/private keys.
Digitally signing is to take value, encrypt it with your private key and send it out with your certificate. To verify the signature you use the public key from the certificate to decrypt the data. It should match your copy of the value. If it matches you know it came from the certificate owner.
On z/OS you can sign load modules. These are signed at bind time, and, if configured, are checked at execution time.
Packages like openssl, and GPG (Pretty Good Privacy) have the facilities to sign objects.
You have the Certificate Authorities file ca256.pem. This is available to every one. This could have been provided by a commercial Certificate Authority.
You have a public certificate which you will send with your package.
You have a private key matching the public certificate.
On the recipient’s machine
You have the Certificate Authority file ca256.pem.
How you do it
Create the package
You want to package data.file and distribute it. Calculate the digest of the file, and digitaly sign it
You can now periodically reissue the commands, and check the value are the same as they were previously. If the values have changed – the files have changed. You can also extend this (with a small bit of Python or shell code) to include the system name
If those files are meant to be the same on both of the systems, they clearly are not. If you run the digest command weekly you will be able to see approximately when the file was changed. It may be that maintenance was applied to one system, and the other system was overlooked.
I was asked this question by someone wanting to ensure their files have not been hacked. In the press there are articles where bad guys have replaced some code with code that steals credentials, or it allows an outsider access to your machine. One common solution to trusting a file uses cryptography.
There are several solutions that do not work
Checking the date of a file.
This does not work because there are programs that allow you to change the date and time of files.
Checking the number of bytes
You can list a file’s properties. One property is the size of the file. You could keep a list of file, and file size.
There are two problems
You can change the contents of the file without changing the size of the file. I’ve done this. Programs used to have a patch area where skilled people could write some code to fix problems in the program.
Someone changes the size of the file – but also changes your list to reflect the new size, and then changes the date on file and your list so they look as if they have not changed.
Hashing the file contents
Do a calculation on the contents of the file. A trivial function to implement and easy to exploit, is to treat each character as an unsigned integer, and add up all of the characters.
A better hashing function is to do a calculation cs = mod(c **N,M). For example when the current character is 3, n is 7 and m is 13; find the remainder of 3*3*3*3*3*3*3 when divided by 13, the answer is 3. N and M should be very large. Instead of using one character you take 8 or more. You then apply the algorithm on the file.
cs = 0 do 8 bytes of the file at a time cs = mod((cs + the 8 bytes)** N,M) end display cs
Some numbers N and M are better that others. Knowing the value cs, you cannot go back and recreate the file.
If you just store the checksum value in a file, then the bad guys can change this file, and replace the old checksum with the new checksum of the file with their change. It appears that doing a checkum on the file does not help.
Cryptography to the rescue
To make things secure, there are several bits of technology that are required
Public and private keys
How do I trust what you’ve sent me
Public and private keys
Cryptography has been around for thousands of years. This typically had a key which was use to encrypt data, and the same key could be used to decrypt the data.
The big leap in cryptography was the discovery of asymmetric keys where you need two keys. One can be used for encryption, and you need another for decryption. You keep the one key very secure (and call it the private key) and make the other key publicly available (the public key). Either key can be used to encrypt, and you need the other key to decrypt.
The keys can be used as follows
You encrypt some data with my public key. It can only be decrypted by someone with my private key.
I can encrypt some data with my private key and sent it to you. Anyone with my public key can decrypt it. In addition, because they had to use my public key, then they know it came from me (or, to be more accurate, someone with my private key).
How do I trust what you’ve sent me
I would be very suspicious if I received an email saying
This is your freindly bank heree. Please send us your bank account details with this public key. Pubic keys are very safe and we are the only peoples who can decrypt what you send me.
Digital certificates and getting a new passport
A public certificate has
Your name
You address such as Country=GB, Org=Megabank.com,
Your public key
Expiry date
What the certificate can be used for
I hope the following analogy explains the concepts of digital certificates.
Below are the steps required to get a new passport
You turn up at the Passport Office with your birth certificate, a photograph of you, a gas bill, and your public certificate.
The person in the office checks
that the photo is of you.
your name is the same as the birth certificate
the name on the gas bill matches your birth certificate
the address of the gas bill is the same as you provided for your place of residence.
The office creates the passport, with information such as where you live (as evidenced by the gas bill)
The checksum of your passport is calculated.
The checksum is encrypted with the Passport Office’s PRIVATE key.
The encrypted checksum and the Passport Office’s PUBLIC key are printed, and stapled to the back of the passport
The passport is returned to you. It has been digitally signed by the Passport Office.
How do I check your identity?
At the back of MY passport is the printout of the Passport Offices’ public key. I compare this with the one attached yo your passport – they match!
I take the encrypted checksum from your passport, and decrypt it using the Passport Office’s public key (yours or mine – they are the same). I write this on a piece of paper.
I do the same checksum calculation on your passport. If the value matches what is on the piece of paper, then you can be confident that the passport has not been changed, since it was issued by the Passport Office. Because I trust the Passport Office, I trust they have seen your birth certificate, and checked where you live, and so I trust you are who you say you are.
But..
Your passport was issued by the London Passport Office, and my passport was issued by the Scottish Passport Office, and the two public certificates do not match.
This problem is solved by use of a Certificate Authority(CA)
Consider a UK wide Certificate Authority office. The Scottish Passport Office sent their certificate (containing, name address and public key) to the UKCA. The UKCA did a checksum of it, encrypts the checksum with the UKCA PRIVATE key, attached the encrypted checksum, and the UKCA public certificate to the certificate sent in – the same process as getting a passport.
Now when the Scottish Passport office process my passport, they do the checksum as before, and affix the Scottish Passport Offices’ public certificate as before… but this certificate has a copy of the UKCA’s certificate, and the encrypted checksum stuck to it. The passport now has two bits of paper stapled to it, the Scottish Passport Office’s public certificate, and the UKCA’s public certificate.
When I validate your passport I see that the London Passport office’s certificate does not match the Scottish Passport Offices certificate, but they have both been signed by the UKCA.
I compare the UKCA’s public certificates – they match!
I decrypt the checksum from the London office using the UKCA’s certificate and write it down
I do the same checksum calculation on the London offices’s certificate and compare with what is written down. They match – I am confident that the UKCA has checked the credentials of the London office
I can now trust the London certificate, and use it to check your passport as before.
What happens if I do not have the UKCA certificate
Many “root” certificates from corporations, are shipped on Windows, Linux, z/OS, Macs etc. The UKCA goes to one of these corporations, gets their certificate signed, and includes the corporations certificate attached to the UKCA certificate. Of course it costs money to get your certificate signed by one of these companies
You could email the UKCA certificate with the public key to every one you know. This has the risk that the bad guys who are intercepting your email, change the official UKCA certificate with their certificate. Plan b) would be to ship a memory stick with the certificate on it – but the same bad guys could be monitoring your mail, and replace the memory stick with one of theirs.
How does this help me trust a file?
The process is similar to that of getting a passport.
My “package” has two files abx.txt and xyz.txt
At build time
Create the files abc.txt and xyz.txt
Calculate the checksum of each file, and encrypt the value – this produces a binary file for each abc.txt.signature
Create a directory with
Your public certificate/public key
A directory containing all of the signature files
A list of all of the files in the signature directory
A checksum of the directory listing. directory.list.signature
You ship this file as part of your product.
When you install the package
Validate the certificate in the package against the CA stored in your system.
Decrypt the list of files in the directory (directory.list.signature). Check the list of files is valid
For each line in the directory list, go through the same validations process with the file and it’s signature.
For the paranoid
Every week calculate the checksum of each file in the package and sent it to a remote site.
At the remote site compare the filename, checksum combination against last week’s values.
If they do not match, the files have been changed.
Of course if your system has been hacked, the bad guys may be intercepting this traffic and changing it.
How do I do it?
I have a certificate mycert.pem, and my private key mycert.private.pem. It was signed by ca256.
You can get into Unix Services from TSO OMVS, or you can use SSH to get directly into a Unix Services session (without access to ISPF). I was interested in using SSH. At first glance it works just like a Linux command window.
My first experience was the keys did not behave as I expected, and how can I change this.
On Linux I can type the first part of a file name, press the tab key, and it either completes the name of the file (if there was only one) or lists the names of the possible matching files. This is known as autocomplete.
I tried this in an SSH session on z/OS, and this did not work. I was using the default shell, which does not have this support. I needed to use the bash shell or the zsh shell. Note the zsh was developed on Unix and ported to z/OS, it is not a z/OS special.
How do I use a different shell?
The first question is, “is the shell available on my system?”, to which the answer is maybe.
z/OS ships with the default shell in /bin/sh.
You can get a free version of bash for z/OS. My ADCD based system does not have Bash pre-installed. I believe you can get it for free from Rocket.
My ADCD system has zsh installed (but not configured).
Your default shell
This is set in the in your RACF OMVS segment PROGRAM field . To display it, use
TSO LU COLIN OMVS
This gave me
OMVS INFORMATION ---------------- UID= 0000990021 HOME= /u/colin PROGRAM= /bin/sh ...
Once you logon to your shell various profiles may be executed if they exist for example for the default shell
/etc/profile and ~/.profile where ~ is your default home directory (see HOME= in the output of the LU command, above;
Typically the profile script will set the environment variable SHELL. I put the following in my ~/.profile so my SSH sessions get the zsh shell.
#!/bin/sh
if [[ -z "$SSH_CLIENT" ]] then # dummy xxx="$SSH_CLIENT" else # SSH_CLIENT has a value ... so an SSH terminal #cho "using the zsh shell" zsh="/bin/zsh" echo "shell $SHELL zsh $zsh" if [[ $SHELL != $zsh ]] then echo "using the zsh shell" export SHELL="$zsh" exec "$zsh" fi fi
zsh is documented in the UNIX System Services Command Reference. This is a big book, so I extracted the section of zsh using
If zsh has been installed properly you should have an environment variable $ZDOTDIR defined. If this is missing then $HOME is used instead.
There are files
$ZDOTDIR/.zshenv
$ZDOTDIR/.zprofile
$ZDOTDIR/.zshrc
$ZDOTDIR/.zlogin
$ZDOTDIR/.zlogout
Which should contain configuration information
There are comment on the IBM community about the poor documention of zsh on z/OS, an how you can get started. It gives hints on setting up colours etc.
Commands are first read from /etc/zshenv; this cannot be overridden. …
Commands are then read from $ZDOTDIR/.zshenv.
If the shell is a login shell, commands are read from /etc/zprofile and then $ZDOTDIR/.zprofile.
Then, if the shell is interactive, commands are read from /etc/zshrc and then $ZDOTDIR/.zshrc.
Finally, if the shell is a login shell, /etc/zlogin and $ZDOTDIR/.zlogin are read.
When a login shell exits, the files $ZDOTDIR/.zlogout and then /etc/zlogout are read. This happens with either an explicit exit via the exit or logout commands, or an implicit exit by reading end-of-file from the terminal
If you start with the system shell, then invoke the zsf shell, the PATH and LIBPATH statements etc will be inherited from the system shell. If you go directly to the zsh you’ll need to set up a profile based on the system profile. ( or just invoke the existing profile).
You may want to set up a profile to set the command prompt for example the default is
$LOGNAME:$PWD:
You can set it with
export prompt='%n:%/'
to give
COLIN:/u/colin
Changing what keys do … getting the delete key to work as expected
I installed zopen. zopen provides lots of Unix-like packages on z/OS. Its home page says
The zopen community is here to provide popular Open Source tools and to encourage z/OS Open Source tools development. We currently host 200+ z/OS Open Source projects and we’re looking for more!
SFTP is an FTP implementation from openssl, it copies files to and from Unix Services.
The documentation z/OS OpenSSH User’s Guide(SC27-6806-50) is very good. It is clear, and answered most of my questions.
The ADCD implementation is not 100% complete, but it does not take much to fix it.
Background
The server runs in Unix Services. When you use “START SSHD”, it runs the started task SSHD which attaches a process in Unix Services, and then the started task ends! If you use the operator command “D A,SSH*” it will show it is running in Unix Services.
The documentation has several ways of stopping it – I just use cancel!
Messages are written to the syslog daemon.
It uses port 22. ADCD.Z24C.TCPPARMS(PROF2) has the statement
PORT
...
22 TCP SSHD* ; port for sshd daemonrver
This means that jobs SSHD* are authorised to use port 22.
Digital certificates are great. They allow program to program communication so each end can get information to identify the other end, and the programs can then communicate securely, with encryption, or just checking the payload has not changed.
A certificate is basically a file with two parts (or two files) – a public certificate and a private key. You can publicize the public part to any one who wants it (which is why is is called the public part). Anyone with the private key can use it to say they are you. (If you can get access to the private key, then you can impersonate the identity)
There are times when you want to say this certificate is not longer valid. For example when I worked at IBM, I had a certificate on my laptop to access the IBM mail servers.
If my laptop was stolen, IBM would need to revoke the certificate.
When I retired from IBM, IBM revoked my certificate to prevent me from trying to access my IBM mail using my old certificate from my personal laptop.
Managing these certificate is difficult. There could be billions of certificates in use today.
Your server should validate every connection request to ensure the certificate sent from the client is still valid.
A client should validate the certificate sent by the server to ensure that it is connecting to a valid server.
In the early days of certificates, there was a big list of revoked certificates – the Certificate Revocation List(CRL). If a certificate is on the list then it has been revoked. You tend to have an LDAP server within your firewall which contains these lists of revoked certificates.
This was a step in the right direction, but it is difficult to keep these lists up to date, when you consider how many certificates are in use today, and how many organizations generate certificates. How often do I need to refresh my list? If the CRL server was to refresh it every day, it may be up to one day out of date, and report “this certificate is ok” – when it had been revoked.
These days there is a technique called Online Certificate Status Protocol (OCSP). Basically this says go and ask the site which issued this certificate if it is still valid. This is a good idea – and they say the simple ideas are the best.
How do you know who to ask? A certificate can have url information within in it Authority Information Access: OCSP – URI:http://ColinsCert.Checker.com/, or you can specify URL information in the queue manager configuration for those certificates without the Authority Information Access(AIA) information.
Often the URL in the certificate is outside of your organization, and outside of your firewall. To access the OCSP site you may need to have an SSL Proxy server which has access through the firewall.
You can configure MQ to use a (one) OCSP server for those certificates not using AIA information. If your organization is a multinational company, you may be working with other companies who use different Certificate Authorities. If you have certificates from more than one CA, you will not be able use MQ to check all of them to see if they are still valid. You may want to set up an offline job which runs periodically and checks the validity of the certificates.
Starting the MQ channel can be slow
When an TLS/SSL MQ channel is started, you can use OCSP or CRL to check that a certificate is valid. This means sending a request to a remote server and waiting for a response. The LDAP server for CRL requests is likely to be within your domain, as your organization manages it. The OCSP server could be outside of your control, and in the external network. If this server is slow, or the access to the server is slow, the channel will be slow to start. For many customers the network time within a site is under 1 millisecond, between two sites could be 50 ms. Going to an external site the time could be seconds – and dependent on the traffic to the external site.
This time may be acceptable when starting the channel, first thing in the morning, but restarting a channel during a busy period can cause a spike in traffic because no messages flow while the channel is starting. For example
No messages will flow while the channel is starting, and this delay will add to the round trip time of the messages.
How do I check to see if I have a problem
This is tough. MQ does not provide any information. I used MQ internal trace when debugging problems, but you cannot run with trace enabled during normal running.
There are two parts to the validation request. The time to get to and from the server, and the time to process your request once it has got to the server.
You can use TCP Ping to get to the server (or to the proxy server). If you are using a proxy server you cannot “ping” through the proxy server.
Openssl provides a many functions for creating and managing certificates.
You can use the command
time openssl s_client -connect server:port or time openssl s_client -connect -proxy host:port server:port
The “time” is a linux command which tells your the duration in milliseconds of the command following it.
The openssl s_client is a powerful ssl client program. The -connect… tries connecting to server:port. You can specify -proxy host:port to use the proxy.
The server at the remote end may not recognize the request – but you will get the response time of the request.
Running this on my laptop I got
time openssl s_client -connect 127.0.0.1:8888 CONNECTED(00000005) 140566846001600:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332: — no peer certificate available — No client certificate CA names sent — SSL handshake has read 5 bytes and written 233 bytes
You can use openssl ocsp …. to send a certificate validation request to the OCSP server and check the validity of the system – but you would have to first extract the certificates from the iKeyman keystore.
There is a good article in the IBM Knowledge Centre here .
The article says To check the revocation status of a digital certificate using OCSP, IBM MQ determines which OCSP responder to contact in one of two ways:
Using a URL specified in an authentication information object or specified by a client application.
Using the AuthorityInfoAccess (AIA) certificate extension in the certificate to be checked.
Configure a QM for OSCP checking for certificates without AIA information.
You can configure a queue manager to do OCSP checking, for those certificates without AIA information within the certificate.
There is a queue manager attribute SSLCRLNL ( SSLCertificate RevocationList Name List) which points to a name list. This name list has a list of AUTHINFO object names.
The name list can have up to one AUTHINFO object for OCSP checking and up to 10 AUTHINFO objects for CRL checking.
You define an AUTHINFO object to define the URL of an OCSP server.
Use alter qmgr SSLCRLNL(name) and refresh security type(SSL)
You need to change the qm.ini file SSL stanza of the queue manager configuration file see here.
OCSPAuthentication=REQUIRED|WARN|OPTIONAL
OCSPCheckExtensions= YES|NO
SSLHTTPProxyName= string
If you have a fire wall around your network, you can use SSLHTTPProxyName to get through your fire wall.
Configure client OSCP checking for certificates without AIA information.
You need the CCDT created by a queue manager rather than a JSON CCDT.
When you configure a queue manager with the AUTHINFO objects and the queue manager SSLCRLNL attributes, the information is copied to the CCDT.
For example if you think you have a problem with OCSP server response time.
In the mqclient.ini you can set
ClientRevocationChecks = DISABLED No attempt is made to load certificate revocation configuration from the CCDT and no certificate revocation checking is done
OCSPCheckExtensions = NO This says ignore the URL in the AIA information within a certificate.
There are no events or messages which tell you the response time of requests.
You may get message AMQ9716
AMQ9716
Remote SSL certificate revocation status check failed for channel …
Explanation
IBM MQ failed to determine the revocation status of the remote SSL certificate for one of the following reasons:
(a) The channel was unable to contact any of the CRL servers or OCSP responders for the certificate.
(b) None of the OCSP responders contacted knows the revocation status of the certificate.
(c) An OCSP response was received, but the digital signature of the response could not be verified.
You can change the queue manager configuration to not produce these messages, by setting ClientRevocationChecks = OPTIONAL
From this message you cannot tell if the request got to the server.
The easiest way may be to ask the network people to take a packet trace to the URL(s) and review the time of the requests and the responses.
Using the AuthorityInfoAccess (AIA) certificate extension in the certificate.
You can create certificates containing the URL needed to validate the certificate. Most of the IBM MQ documentation assumes you have already have a certificate with this information in it.
You can use openssl to create a certificate with AIA information, and import it into the iKeyman keystore. See here.
You cannot use IBM GSKIT program iKeyman to generate this data because it does not support it. You can use iKeyman to display the information once it is inside the keystore.
Timing the validation request
Openssl has a command to validate a certificate for example