How do you download and use a dataset from z/OS.

Transferring a dataset from z/OS to Windows or Linux and using it can be a challenge.

A record in a data set on z/OS has a 4 byte Record Descriptor Word on the front of the record. The first two bytes give the length of the record (and the other two bytes are typically 0)

FTP has two modes for transferring data ASCII and BIN.

ASCII

With ASCII mode, FTP reads the record,

  • Removes the RDW
  • Converts it from EBCDIC to ASCII
  • Adds a “New Line” character to the end of data
  • Sends the data
  • Writes the data to a file stream.

On Unix and Windows a text file is a long stream of data. When the file is read, a New Line character ends the logical record, and so you display the following data on a “New Line”.

Binary mode

Binary mode is used when the dataset has hexadecimal content, and not just printable characters. The New Line hex character could be part of a some hexadecimal data, so this character cannot be used to delineate records.

FTP has an option for RDW

quote site RDW

The default is RDW FALSE.

If RDW is FALSE then FTP removes the RDW from the data before sending it. At the remote end, the data is a stream of data, and you have no way of identifying where one logical record ends, and the next logical record starts.

If RDW is TRUE, then the 4 byte RDW is sent as part of the data. The application reading the file can read the information and calculate where the logical record starts and ends.

For example on z/OS the dataset has (in hex) where the bold data is displayed when you edit or browse the dataset. The italic data is not displayed.

00040000C1C2C3C4
00020000D1CD2
00050000E1E2E3E4E5

If the data was transmitted with RDW FALSE the data in the file would be

C1C2C3C4D1D2E1E2E3E4E5

If the data was transmitted with RDW TRUE the data in the file would be

00040000C1C2C3C400020000D1CD200050000E1E2E3E4E5

Conceptually you can process this file stream using C code:

short RDW;  // 2 byte integer
short dummy; // 2 byte integer

RDW = fread(2); // get the length
dummy = fread(2); // ignore the 0s
mydata = fread(RDW -4); // -4 for the RDW already read

...
RDW = fread(2); // get the length
dummy = fread(2); // ignore the 0s
mydata = fread(RDW -4); // -4 for the RDW already read

(Thanks to pjfarley3 who pointed out the RDW length includes the 4 byte RDW – so the application data length is RDW -4.)

In practice this will not work because z/OS has numbers which are Big Endian, and X86 and ARM machines are Little Endian. (With Big Endian – the left byte is most significant, with Little Endian, the right bit is most significant – the bytes are transposed.)

On z/OS 0x0004 is decimal 4. On X86 and ARM 0x0400 is 4.

In practice you need code on X86 and ARM, like the following, to get the value of a half word from a z/OS data set.

char RDW[2];  // 2 characters
RDW = fread(2); // get the length
length = 256 * RDW[0] + RDW[1]

and similarly for longer integers.

Python

If you are using the Python struct facility, you can pass a string of data types and get the processed values.

  • The string “>HH” says two half words, and the > says the numbers are Big Endian.
  • The string “<HH” says two half words and the < says they are Little Endian
  • The string “HH” says two half words – read in the default representation.

Conversion

You’ll need to do your own conversion from EBCDIC to ASCII to make things printable!

How do you trust a file?

I was asked this question by someone wanting to ensure their files have not been hacked. In the press there are articles where bad guys have replaced some code with code that steals credentials, or it allows an outsider access to your machine. One common solution to trusting a file uses cryptography.

There are several solutions that do not work

Checking the date of a file.

This does not work because there are programs that allow you to change the date and time of files.

Checking the number of bytes

You can list a file’s properties. One property is the size of the file. You could keep a list of file, and file size.

There are two problems

  1. You can change the contents of the file without changing the size of the file. I’ve done this. Programs used to have a patch area where skilled people could write some code to fix problems in the program.
  2. Someone changes the size of the file – but also changes your list to reflect the new size, and then changes the date on file and your list so they look as if they have not changed.

Hashing the file contents

Do a calculation on the contents of the file. A trivial function to implement and easy to exploit, is to treat each character as an unsigned integer, and add up all of the characters.

A better hashing function is to do a calculation cs = mod(c **N,M). For example when the current character is 3, n is 7 and m is 13; find the remainder of 3*3*3*3*3*3*3 when divided by 13, the answer is 3. N and M should be very large. Instead of using one character you take 8 or more. You then apply the algorithm on the file.

cs = 0
do 8 bytes of the file at a time
cs = mod((cs + the 8 bytes)** N,M)
end
display cs

Some numbers N and M are better that others. Knowing the value cs, you cannot go back and recreate the file.

If you just store the checksum value in a file, then the bad guys can change this file, and replace the old checksum with the new checksum of the file with their change. It appears that doing a checkum on the file does not help.

Cryptography to the rescue

To make things secure, there are several bits of technology that are required

  • Public and private keys
  • How do I trust what you’ve sent me

Public and private keys

Cryptography has been around for thousands of years. This typically had a key which was use to encrypt data, and the same key could be used to decrypt the data.

The big leap in cryptography was the discovery of asymmetric keys where you need two keys. One can be used for encryption, and you need another for decryption. You keep the one key very secure (and call it the private key) and make the other key publicly available (the public key). Either key can be used to encrypt, and you need the other key to decrypt.

The keys can be used as follows

  • You encrypt some data with my public key. It can only be decrypted by someone with my private key.
  • I can encrypt some data with my private key and sent it to you. Anyone with my public key can decrypt it. In addition, because they had to use my public key, then they know it came from me (or, to be more accurate, someone with my private key).

How do I trust what you’ve sent me

I would be very suspicious if I received an email saying

This is your freindly bank heree. Please send us your bank account details with this public key. Pubic keys are very safe and we are the only peoples who can decrypt what you send me.

Digital certificates and getting a new passport

A public certificate has

  • Your name
  • You address such as Country=GB, Org=Megabank.com,
  • Your public key
  • Expiry date
  • What the certificate can be used for

I hope the following analogy explains the concepts of digital certificates.

Below are the steps required to get a new passport

  • You turn up at the Passport Office with your birth certificate, a photograph of you, a gas bill, and your public certificate.
  • The person in the office checks
    • that the photo is of you.
    • your name is the same as the birth certificate
    • the name on the gas bill matches your birth certificate
    • the address of the gas bill is the same as you provided for your place of residence.
  • The office creates the passport, with information such as where you live (as evidenced by the gas bill)
  • The checksum of your passport is calculated.
  • The checksum is encrypted with the Passport Office’s PRIVATE key.
  • The encrypted checksum and the Passport Office’s PUBLIC key are printed, and stapled to the back of the passport
  • The passport is returned to you. It has been digitally signed by the Passport Office.

How do I check your identity?

At the back of MY passport is the printout of the Passport Offices’ public key. I compare this with the one attached yo your passport – they match!

I take the encrypted checksum from your passport, and decrypt it using the Passport Office’s public key (yours or mine – they are the same). I write this on a piece of paper.

I do the same checksum calculation on your passport. If the value matches what is on the piece of paper, then you can be confident that the passport has not been changed, since it was issued by the Passport Office. Because I trust the Passport Office, I trust they have seen your birth certificate, and checked where you live, and so I trust you are who you say you are.

But..

Your passport was issued by the London Passport Office, and my passport was issued by the Scottish Passport Office, and the two public certificates do not match.

This problem is solved by use of a Certificate Authority(CA)

Consider a UK wide Certificate Authority office. The Scottish Passport Office sent their certificate (containing, name address and public key) to the UKCA. The UKCA did a checksum of it, encrypts the checksum with the UKCA PRIVATE key, attached the encrypted checksum, and the UKCA public certificate to the certificate sent in – the same process as getting a passport.

Now when the Scottish Passport office process my passport, they do the checksum as before, and affix the Scottish Passport Offices’ public certificate as before… but this certificate has a copy of the UKCA’s certificate, and the encrypted checksum stuck to it. The passport now has two bits of paper stapled to it, the Scottish Passport Office’s public certificate, and the UKCA’s public certificate.

When I validate your passport I see that the London Passport office’s certificate does not match the Scottish Passport Offices certificate, but they have both been signed by the UKCA.

  • I compare the UKCA’s public certificates – they match!
  • I decrypt the checksum from the London office using the UKCA’s certificate and write it down
  • I do the same checksum calculation on the London offices’s certificate and compare with what is written down. They match – I am confident that the UKCA has checked the credentials of the London office
  • I can now trust the London certificate, and use it to check your passport as before.

What happens if I do not have the UKCA certificate

Many “root” certificates from corporations, are shipped on Windows, Linux, z/OS, Macs etc. The UKCA goes to one of these corporations, gets their certificate signed, and includes the corporations certificate attached to the UKCA certificate. Of course it costs money to get your certificate signed by one of these companies

You could email the UKCA certificate with the public key to every one you know. This has the risk that the bad guys who are intercepting your email, change the official UKCA certificate with their certificate. Plan b) would be to ship a memory stick with the certificate on it – but the same bad guys could be monitoring your mail, and replace the memory stick with one of theirs.

How does this help me trust a file?

The process is similar to that of getting a passport.

My “package” has two files abx.txt and xyz.txt

At build time

  • Create the files abc.txt and xyz.txt
  • Calculate the checksum of each file, and encrypt the value – this produces a binary file for each abc.txt.signature
  • Create a directory with
    • Your public certificate/public key
    • A directory containing all of the signature files
    • A list of all of the files in the signature directory
    • A checksum of the directory listing. directory.list.signature

You ship this file as part of your product.

When you install the package

  • Validate the certificate in the package against the CA stored in your system.
  • Decrypt the list of files in the directory (directory.list.signature). Check the list of files is valid
  • For each line in the directory list, go through the same validations process with the file and it’s signature.

For the paranoid

Every week calculate the checksum of each file in the package and sent it to a remote site.

At the remote site compare the filename, checksum combination against last week’s values.

If they do not match, the files have been changed.

Of course if your system has been hacked, the bad guys may be intercepting this traffic and changing it.

How do I do it?

I have a certificate mycert.pem, and my private key mycert.private.pem. It was signed by ca256.

Build

Run the command against the first file

openssl dgst -sign mycert.key.pem abc.txt   > abc.txt.signature

Move the abc.txt.signature to the package’s directory,

Create the trust package

/
.. /mycert.pem
.. /directory.list.txt
.. /directory.list.txt.signature
.. /signatures/
.. .. /abc.txt.signature
.. .. /xyz.txt.signature

Validate the package

Validate the certificate in the package.

openssl verify -CAfile ca256.pem mycert.pem 

extract the public key from the certificate.

openssl x509 -pubkey -noout -in mycert.pem > mycert.pubkey

validate the checksum of the abc file using the public key.

openssl dgst -verify ./mycert.pubkey  -signature abc.txt.signature  abc.txt

Does it work with data sets ?

On z/OS I created a signature file with

openssl dgst -sign tempcert.key.pem  "//'COLIN.JCL(ALIAS)'"  > jcl.alias.signature

and validated it with

openssl dgst -verify tempcert.pubkey -signature jcl.alias.signature  "//'COLIN.JCL(ALIAS)'"   

Can I automate new SSH connections to z/OS?

I’m running on Linux, and using remote z/OS systems. Being from a performance background I hate having to waste seconds, manually starting SSH sessions to my backend systems.

I found I can automate this!

From my gnome-terminal I can issue the command

gnome-terminal --tab --working-directory=~ --title=COLINS --profile=blue  -- ssh colin@10.1.1.2 

This

  • created a new terminal session as a tab in the existing terminal window
  • did a cd ~
  • called the tab COLINS
  • selected the profile called blue (go to the hamburger of your current terminal and select profile to see what profiles you have)
  • executes the command after the ‐‐, so executes ssl colin@10.1.1.1

You can issue the command

gnome-terminal --help-all

to get a list of all options.

Getting SSH to work to z/OS

I have two versions of z/OS, old and new(!). I had problems getting ssh to work because of key problems.

The problem

I tried to update my laptop key to the server

ssh-copy-id colin@10.1.1.2

This gave

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
ERROR: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
ERROR: Someone could be eavesdropping on you right now (man-in-the-middle attack)!
ERROR: It is also possible that a host key has just been changed.
ERROR: The fingerprint for the ED25519 key sent by the remote host is
ERROR: SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98.
ERROR: Please contact your system administrator.
ERROR: Add correct host key in /home/colin/.ssh/known_hosts to get rid of this message.
ERROR: Offending ED25519 key in /home/colin/.ssh/known_hosts:1
ERROR: remove with:
ERROR: ssh-keygen -f '/home/colin/.ssh/known_hosts' -R '10.1.1.2'
ERROR: Host key for 10.1.1.2 has changed and you have requested strict checking.
ERROR: Host key verification failed.

Searching the internet I got suggestions saying “delete the old line from the file”. I didn’t want to do this because it meant I would not be able to go back to the old system and work as before.

Solutions

I edited /home/colin/.ssh/known_hosts and commented out line 1, with a # at the front (the :1 above is the first line). I repeated the command and it report the same message for line :2. I commented that out as well.

I got further

colin@ColinNew:~$ ssh-copy-id colin@10.1.1.2
The authenticity of host '10.1.1.2 (10.1.1.2)' can't be established.
ED25519 key fingerprint is SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys
colin@10.1.1.2: Permission denied (publickey,hostbased).

I had to start the SYSLOGD on z/OS to capture the output from SSHD.

In the /var/logSSHD (your’s may be different) it said

FOTS2307 User COLIN from 10.1.0.2 not allowed because not listed in AllowUsers 

In my SSHD config file /etc/ssh/sshd_config I had

# Allow specific user IDs 
AllowUsers IBMUSER

I added COLIN to the list and restarted SSHD. (I do not know how to refresh SSHD)

This time the error log had

trying public key file /u/tmp/zowet/colin/.ssh/authorized_keys 
Could not open authorized keys '/u/tmp/zowet/colin/.ssh/authorized_keys': ...

I fixed this, tried to logon, and this time it worked.

On Linux, I edited /home/colin/.ssh/known_hosts and un-commented the lines I had commented out before.
I tried the ssh command again, and it still worked!

Linux mapping the keyboard, and on z/OS SSH

I wanted to configure the keyboard in zsh and bash on z/OS so the delete key (top right of the keyboard) deleted the current character. (Backspace deletes the previous character) I could not find any good documentation on how to do it.

I found some bash for Linux documentation and have based this blog post on that. I’ve given the command I used, then sections on the background of how it works.

Note the binding of key to action depends on the shell. The shells zsf and bash support binding. shell does/may not.

Having written the original blog post, I found some additional useful information which will make the document clearer.

How to set the delete key to delete

With my Bash shell, I initially had to do it in stages:

  • Type bind “ that’s bind space double quote
  • Control+V, delete key to insert the value of the key. This made it bind “^[[3~
  • Type “:delete-char with no space after the :
  • It now looks like bind “^[[3~”:delete-char
  • Press enter
  • The delete key should delete the character under the cursor

The command bind -p |grep delete gave

"\C-h": backward-delete-char
"\C-?": backward-delete-char
"\C-d": delete-char
"\M-[3~": delete-char
"\\": delete-char
# delete-char-o

Which shows that Ctrl+d and \\ also deletes the current character.

In a shell (in ASCII) this is (in ISPF edit with hex on)

bind '" ∇3~":delete-char' 
6666222153723666676266672222
29E4072BB3E2A45C545D38127000

Where the incantation is x1b5b337e.

What does the mapping mean?

It took me many hours of looking for a good description of what the key mapping is.

The Wikipedia page ANSI escape codes, was very helpful and clear.

  • You can press the “a” key – and “a” is displayed in the command window.
  • You can press shift “a” – and “A” is displayed in the command window
  • You can press Control (Ctrl) “a” – and this may do something
  • You can press Meta “the Windows” key “a” and this may do something
  • You can press Alt + “a” and it may do something.
  • You can press combinations of those keys, such as Ctrl+shift + “a” and this may do something.

The operating system may intercept the key and not pass it to the window. For example on my Linux, Ctrl+Alt+Delete is logout.

In the command window the shell code will intercept a key press combination and take an action, for example move the cursor left, clear the screen, delete word, or just the the character.

You can change the mapping using the bind or bindkeys command for bash and zsf shells.

You can find the code for a key combination by using ctrl+v. For example with the left cursor key

Ctrl+v, < gives ^[[D

How to decode ^[[D ?

There are several formats.

  • <esc>[ number(;modifier)~
  • <esc>[ letters

The string starts with an escape sequence. ^[ This can be written as \e ,\033,  0x1b, or ^[. Where \0 is octal 33, which is hex 1B, (and decimal 27). When specifying a key sequence any of the values can be used. I mentioned above the incantation x1b5b337e.

Because there is no trailing ~ we need to lookup the “[D” in the Wikipedia page section xterm sequences. This has

<esc>[D     - Left 

So for ^[[D we have <esc>, Left key pressed

For ^[[3;5~ we have

  • ^[ is escape
  • [3 isDelete key
  • ; is a modifier
  • 5 is Control + shift
  • ~ end of escape

so the key presses was Control + Delete key (top right on my keyboard). Shift seems to be ignored!

Another popular key is ^M which is Carriage Return – (and process any data on the line) and move the cursor to column 0.

\C is the Ctrl key, \M is the meta key. For keyboards without the Meta key this is displayed as Escape \e.

Keys like \M-OD comes from

Keypad mode
"\M-OD":backward-char
"\M-OC":forward-char
"\M-OA":previous-history
"\M-OB":next-history

This information is not well documented.

What is key mapping

On Linux Ubuntu the control-right arrow combination moves the cursor right one word. The control-left arrow moves the cursor left one word. Control-L clears the screen, but leaves the command line intact.

You can list the possible actions

bind - l

This gave

...
arrow-key-prefix
backward-byte
backward-char
backward-delete-char
backward-kill-line
backward-kill-word
...

Get a list of functions and what keys use them

With upper case -P

bind -P |grep backward

This gave

backward-byte is not bound to any keys
backward-char can be found on "\C-b", "\eOD", "\e[D".
backward-delete-char can be found on "\C-h", "\C-?".

Where does \eOD come from ? See here.

Get a list of keys and their functions in bind format

with lower case -p

bind -p |grep backward

gave

"\eOD": backward-char
"\e[D": backward-char
"\C-h": backward-delete-char
"\C-?": backward-delete-char
...

where

  • Control-h moves the cursor left one character and deletes the character
  • \e is the escape key. The alt key is usually mapped to the escape key by terminal emulators.

This output is slightly untrue. If there is no blank after the : you can use bind….

If you have a blank after the : you need to enclose it in single quotes.

bind ‘”\C-h”: backward-delete-char’

to set the key.

What is the code for a key press combination?

The Linux command Ctrl-V (verbatim insert) followed by a key, gives the mapping for that key.

Ctrl+V LeftArrow
^[[D

Ctrl+V Ctrl+Leftarrrow
^[[1;5D

Where ^[ means the escape key and [D is OD.

^[[3~ is escape Delete key.

Verbatim insert gives the code of the key which was pressed. This works on z/OS if you have the Bash or zsf shell installed.

What keys are mapped?

In the zsh shell you can issue

bindkey -L

(With bash you can use the bind command).

This gives output like

bindkey "^E" end-of-line
...
bindkey "^L" clear-screen

Where ^ is the ctrl key.

If you type (on Linux) man bash , and locate Readline Command Names if lists the function names and what they mean.

The bash command bind -l lists all of the functions

$ bind -q backward-char
backward-char can be invoked via "\C-b", "\M-OD", "\M-[D".

It gets very confusing

A sequence can be created in different formats. For example many commands begin with the Escape (or Meta key). This can be written as \e ,\033,  0x1b, or ^[. Where \0 is octal 33 which is hex 1B, ( or decimal 27). It is confusing when you display information with different commands.

Where does \eOD (or \M-OD)come from?

It was a challenge to find this information.

In the Linux terminfo documentation it says

The codes sent by the left arrow, right arrow, up arrow, down arrow, and home keys can be given as kcub1, kcuf1, kcuu1, kcud1, and khome respectively. If there are function keys such as f0, f1, …, f10, the codes they send can be given as kf0, kf1, …, kf10. If these keys have labels other than the default f0 through f10, the labels can be given as lf0, lf1, …, lf10.

The linux command infocmp

infocmp|grep kcu 

gave

kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA,

so this all says the left arrow key maps to backward character. – Phew

Problems

If there was a space after the : I got

readline: delete-char: no key sequence terminator

from the bind command.

Putting the command into my .profile file didn’t work because of the wrong character set.

I could put the command into a shell script, and invoke it – but I could not get it to work from the .profile.

Why does one ping work, and the same ping doesn’t?

I was trying to check connectivity from z/OS running on my laptop. For some remote sites I could issue ping and get a response back. For some other sites I issue the ping and did not get a response back.

When I issued the pings from Linux – they both worked.

I noticed that for the pings from z/OS the field Timestamp from icmp data (relative): was 27 seconds behind. This was caused by z/OS adding leap seconds. It was not the problem.

By comparing all the fields in a successful and an unsuccessful ping, I could see that z/OS send 256 bytes of data, and Linux sent only 40 bytes of data.

On Linux, when I used

ping …. -s 256

it failed. When I used

ping …. -s 20

it worked.

Similarly on z/OS.

ping .... (length 20

The defaults lengths are different between z/OS and Linux.

The moral of this tale is

If ping does not return any data – try a very short ping.

How do I configure my new laptop to run my work?

This is an aid to help me when I had a new laptop, and the things I had to do. Some people may find some of the things I use, useful in their work. Because I made some big mistakes, it was worth writing things down.

If people have other good Ubuntu tools which they consider essential, please let me know


Note: It is a list of things to do – not a set of instructions.

Windows

  • mangage bitlocker – disable so you can resize the partition
  • set fast boot off so you can resize the partition

Lenovo bios change

  • F1-F12 as primary function -> On. Without this ISPF P9 gives print screen.
  • Enable trackpoint
  • Disable trackpad
  • Swap Ctrl FN

Prepare the laptop

  • Create Linux bootable image for the correct architecture
  • Boot the Linux USB
  • Change partitions, shrink Windows allocate Linux partition, allocate user partition
  • Install Linux
  • Check Windows still starts

Boot the installed Linux

  • Connect to Wifi
  • Sudo apt update
  • Sudo apt install ssh
  • Sudo apt install sshfs
  • Sudo snap install vivaldi
  • Use disks to create a user parition on the SSD.
    • Format it
    • Edit mount options
    • Unselect User session defaults
    • Display Name: colins
    • Mount point /mnt/Colin
    • Identify as LABEL=Colin
    • Reboot and check disk is mounted
  • Add backup userid
    • sudo adduser ColinPaice
    • sudo passwd ColinPaice
    • sudo adduser ColinPaice sudo
    • sudo mkdir /home/ColinPaice
    • sudo chwon ColinPaice:users /home/ColinPaice
    • su – ColinPaice Check it works
  • Change userid to its parition on /mnt/Colin
    • logon as the backup userid ColinPaice
    • usermod --home /mnt/colin colin
    • su – colin to check it works
  • sudo apt install x3270 do this before installing openssh-server because of font problems
  • sudo apt install openssl-server
  • sudo apt install traceroute
  • set up sshd
    • sudo ufw allow ssh
    • sudo ufw enable
    • sudo systemctl enable ssh
    • sudo systemctl start ssh
    • sudo systemctl status ssh
  • use system to change Ethernet network to
    • Manual 10.1.0.4 255.255.255.0
  • From old laptop ssh colin@10.1.0.4
  • sudo snap install discord
  • sudo apt install wmctrl so I can use hot keys to switch ispf windows
  • Set up networking
    • sudo ip -4 addr add 10.1.0.2/24 dev enp1s0f0
  • save/restore keyboard mappings
    • dconf dump /org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/ > custom.txt
    • cat custom.txt | dconf load /org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/
    • all: dconf dump / > dconf-backup.conf
    • all: cat dconf-backup.conf | dconf load /
  • Move window buttons to the top left sudo apt install gnome-tweaks; gnome-tweaks; windows-: Placement left
  • Set dash icon size. Settings-> Ubuntu desktop -> Dock -> Icon size
  • Add my hot directories to the Gnome file manager side bar.
    • Display the directory and its contents, click on ⋮ select “add to bookmarks”.
  • sudo apt install dconf-editor
  • sudo apt install gnome-shell-extension-manager
    • super – extension manager browse clipboard-history – install
      • Shift super sudo apt install gnome-screenshot
  • For backups sudo apt install deja-dup duplicity
  • Setting icon size for file manager.
    • dconf /org/gnome/nautilus/list-view
    • use default value off
    • custom value ‘small’

Setting up applications to use z/OS on a remote Linux machines

  • For hot key to ISPF sessions sudo apt install wmctrl . system -> keyboard -> keyboard short cuts -> custom
    • Name: colin@ wmctrl -a colin@ shortcut ctrl + 3
    • Name: mst cons wmctrl -a mstcon shortcut ctrl + 1
    • Name: tso@ wmctrl -a tso@ shortcut ctrl + 2
  • sudo apt install wireshark
  • sudo apt install curl

Why can’t I change the colour of my Gnome terminals on my new Linux image?

I’ve got a new laptop, and I’ve spent quite a lot of time migrating stuff from my old one.

It all seemed to be working, except I could not change the colour of my Gnome terminals.

The documentation (and many comments on the internet) say hamburge(≡) -> Profile -> pick your favourite colour from the options. Unfortunately I did not have “Profile” as an option.

Gnome configuration information is stored in a directory tree format under /org/gnome/terminal/legacy/profiles:

This can be see by using:

  • dconf-editor is a GUI tool to allow direct editing of the dconf configuration database.
  • dconf is a simple tool for manipulating a dconf database.
  • gsettings offers a simple command line interface to GSettings. It lets you get, set or monitor an individual key for changes.

From these I could see that for the userid with problems (another userid was OK) I was missing some configuration information.

On my old system I exported the part of the configuration tree using the command

dconf dump /org/gnome/terminal/legacy/profiles:/ > dconf.dump

I sent this file across to my new system, backed up .config/dconf/user and used

dconf load /org/gnome/terminal/legacy/profiles:/ < dconf.dump

I immediately had access to the hamburger icon; it had “Profile >”; and I could pick a colour.

What is in the dumped configuration file?

The dconf.dump file user above contained

[/]
default='f8084ff0-88c6-43ad-b674-d901f5f818a5'
list=['b1dcc9dd-5262-4d8d-a863-c897e6d979b9', '2e126889-4012-485a-a363-057135d6b038', 'f8084ff0-88c6-43ad-b674-d901f5f818a5', '990b09a4-8a20-4ba5-aab6-ad88fdc531dd', 'd963eefc-9bf1-4ae1-8653-e8d36bd6127a', '110cf44a-82fd-454c-bc12-b3918b987cde']

[:110cf44a-82fd-454c-bc12-b3918b987cde]
background-color='rgb(56,111,67)'
use-theme-colors=false
visible-name='ddd'

[:2e126889-4012-485a-a363-057135d6b038]
background-color='rgb(238,238,236)'
foreground-color='rgb(46,52,54)'
use-theme-colors=false
visible-name='blue'
...
[:f8084ff0-88c6-43ad-b674-d901f5f818a5]
default-size-columns=100
use-theme-colors=true
visible-name='default'

There is a definition (at the bottom)

  • id :[:f8084ff0-88c6-43ad-b674-d901f5f818a5]
  • default-size-columns=100
  • use-theme-colors=true
  • visible-name=’default’

I refer to this as default (the visible name) using the system theme colours.

There is a definition called ‘blue’.

  • use-theme-colors=false says do not use the standard theme colour; the colours are overridden
  • the background colours are specified
  • the foreground colours are specified
  • it has an id of 2e126889-4012-485a-a363-057135d6b038

At the top of the file is

  • default=’f8084ff0-88c6-43ad-b674-d901f5f818a5′ this refers to the section which I’ve called default.
  • the list of possible values includes those for default and blue.

Whoops – where has my Firefox configuration gone? oh snap!

On Ubuntu Firefox now comes as a snap package.

Snaps are containerised software packages that are simple to create and install. They auto-update and are safe to run.

All that is true, but nowhere did it say that the Firefox configuration is now in a different place.

Before snaps, profile and configuration files were stored in the ~/.mozilla directory. Now they are stored in ~/snap/firefox/common/.mozilla/firefox/

I had configured my backups to include useful directories including directories ~/.*, and had excluded all ~/snap directories because I could easily download the programs when needed (or so I thought).

I had a problem with Firefox so I deleted the snap files and reinstalled them – to find that my configuration information was not available, and was not backed up.

I’ve now moved to Vivaldi browser.

Lesson learned

I checked where the profiles for vivaldi are stored. They are under ~/snap as well.

Looking into what files are backed up, I had specified which directories I wanted. I think I’ll now say backup all files under my userid, except for….. I have a 2TB solid state external disk drive, so I should have plenty of space as I’ve only used 59 GB of backed up data.

Secure store aren’t

Applications such as Zowe can store secure information on the end user’s machine. This is not very secure! It is built into the operating systems. It is a bit like securing a door with a bit of string. Joshua Waters pointed out

The fact of the matter is that regardless of whether or not you are storing your credentials on a machine, if there is a virus or malicious actor on it, your credentials are up for grabs while the user is logged in. The only time they wouldn’t be up for grabs is if you were using an application that either require some master key to access the credentials store for it, or every authed request to the server requires user to re-enter credentials.

On Linux

The information is in the gnome-keyring ~/.local/share/keyrings/login.keyring .

You can use the Linux command seahorse to display the contents of the gnome-keyring. The user’s password is used to decrypt the store.
The following python code display the keyring contents

import secretstorage
conn = secretstorage.dbus_init()
collection = secretstorage.get_default_collection(conn)
for item in collection.get_all_items():
    print('='*30)
    print('label:', item.get_label())
    print('attributes:')
    for k,v in item.get_attributes().items():
        print('\t%-12s: %s' % (k,v))
    print('secret:',item.get_secret())

This gave

label: Zowe/secure_config_props
attributes:
account : secure_config_props
service : Zowe
xdg:schema : org.freedesktop.Secret.Generic
secret: b'eyIva...9fQ=='

The secret is based64 encoded. You can decode it (on Linux) with

base64 -d <<<"eyIva...9fQ=="  

This gave

{"/home/colinpaice/ssl/ssl2/zowe.config.json":
{"profiles.project_base.properties.user":"colin",
"profiles.project_base.properties.password":"password"
}
}

Where /home/colinpaice/ssl/ssl2/zowe.config.json is the name of the configuration file it applies to.

You can delete an entry using

import secretstorage
conn = secretstorage.dbus_init()
collection = secretstorage.get_default_collection(conn)
for item in collection.get_all_items():
print('='*30)
print('label:', item.get_label())
if item.get_label() == "Zowe/secure_config_props":
item.delete()
print("delete")
continue

This deletes all of the entries for that component – so all the Zowe data.

Who can see the contents of the store?

Your gnome-keyring is encrypted with your password, so you can access it. Someone one else would need your password to be able to decrypt it and see the contents.

What happens on other platforms?

On Windows and Mac’s it is essentially the same. There is a secure disk, and you need to be running as the owner to access it.

If your machine is infected with a virus, which runs under your userid, it can access the key stores and so get userid and password information store in the “secure store”.