Why is Ubuntu is running out of space? It is /var/log/journal/…

Low Disk space on “Filesystem root”

I’ve been getting this message more frequently – and I’ve found out why.

There is a “systemd journal file” with content like

Jul 12 15:50:09 colinpaice rtkit-daemon[1385]: Successfully made thread 2682 of process 2540 owned by ‘1000’ RT at priority 10.
Jul 12 16:44:41 colinpaice rtkit-daemon[1385]: Supervising 5 threads of 3 processes of 1 users.
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 12 16:45:01 colinpaice CRON[7076]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 12 16:45:01 colinpaice CRON[7075]: pam_unix(cron:session): session closed for user root
Jul 12 15:58:32 colinpaice kernel: irq_thread+0xda/0x170

This goes back to when I first installed Ubuntu about 4 years ago, but I think a month’s worth of data would be enough.

You can display the disk space used by using

sudo journalctl –disk-usage

and display the contents of the file using

sudo journalctl -n 50 |less

Note: Without sudo you get the userid’s log size… with sudo you get total log size.

The log file is in /var/log/journal/ and was 1.4 GB in size. The size of this file is controlled by the /etc/systemd/journald.conf configuration file. I edited this file (using sudo gedit /etc/systemd.journald.conf).

  • I uncommented SystemMaxFileSize and gave it a value of 500M.
  • I uncommented SystemMaxFiles and gave it a value of 10

You can either reboot, or use

service systemd-journald restart

to restart the systemd journal.

Although I set the value to 500M, after the journal was restarted – it had size 100MB!

I think 100MB is plenty big enough, and I get a log of disk space back.

Why is this Linux slower to download than that one

I have a laptop which is my primary work station, and an under desk server for running my z/OS system on top of the same Linux.

Running “apt update” on the laptop was always faster on the laptop compared to the server. Was this because all traffic for the server was going through my laptop? How do I tell?

The boxes are connected with an Ethernet cable, I had to purchase a wireless dongle for my server, my laptop has a built in wireless adapter.

The linux ifconfig or the ip command gives information about the configuration. For example ip a

eno1: flags=4163 mtu 1500
    inet netmask broadcast
    inet6 fe80::.... prefixlen 64 scopeid 0x20
    ether 00:... txqueuelen 1000 (Ethernet)
    RX packets 5136 bytes 1445665 (1.4 MB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 4933 bytes 1692274 (1.6 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 17 memory 0xb1200000-b1220000  
wlxd037450ab7ac: flags=4163 mtu 1500
    inet 192.... netmask broadcast 192....
    inet6 2a00:... prefixlen 64 scopeid 0x0
    inet6 fe80::... prefixlen 64 scopeid 0x20
    inet6 2a00:... prefixlen 64 scopeid 0x0
    ether d0:... txqueuelen 1000 (Ethernet)
    RX packets 42427 bytes 60919847 (60.9 MB)
    RX errors 0 dropped 1 overruns 0 frame 0
    TX packets 25996 bytes 2397812 (2.3 MB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
  • The EtherNet connection eno1 has received 5136 packets and 1.4 MB of data
  • The WireLess connection wlx037450ab7ac has received 42427 packets and 60.9 MB of data.

As I had just done an apt upgrade, the wireless had all of the traffic to download the files, so the traffic was not coming through my laptop.

Once the system was updated, the only traffic flowing was down the Ethernet cable as I used the server from my laptop.


A ping from each system gave a similar response time.


traceroute shows you the hops to the destination.

For example

traceroute abc.xyz.com

To specify the interface you need to run as a superuser.

sudo traceroute abc.xyz.com -i wlxd037450ab7ac


1 bthub.home (192....) 4.654 ms 38.438 ms 38.425 ms
2 * * *
3 * * *
4 ( 75.897 ms 75.890 ms 75.861 ms

If you are not running as a superuser you will get:

setsockopt SO_BINDTODEVICE: Operation not permitted

What else is there to help me?

On z/OS the netstat command gives a lot of information about the session, for example the send window size, the receive windows size etc. This information tends not to be available on other platforms.

On linux there is the ss (Socket Statistics) command.

Example output from ss -t -i included

  • ESTAB 0 0
  • https
  • cubic the congestion algorithm name, the default congestion algorithm is “cubic”
  • wscale:9,7 if window scale option is used, this field shows the send scale factor and receive scale factor
  • rto:232
  • rtt:29.236/5.693
  • ato:40 mss:1452
  • pmtu:1500
  • rcvmss:880
  • advmss:1460
  • cwnd:10 congestion window size
  • bytes_sent:68335
  • bytes_acked:68336
  • bytes_received:16334
  • segs_out:276
  • segs_in:202
  • data_segs_out:151
  • data_segs_in:123
  • send 4.0Mbps number of bits sent/time of send.
  • lastsnd:376 how long time since the last packet sent, the unit is millisecond
  • lastrcv:348 how long time since the last packet received, the unit is millisecond
  • lastack:348 how long time since the last packet acknowledged, the unit is millisecond
  • pacing_rate 7.9Mbps
  • delivery_rate 2.6Mbps
  • delivered:152
  • app_limited busy:3700ms
  • rcv_space:14600
  • rcv_ssthresh:64076
  • minrtt:23.773

I did not find most of this information very useful. It is all to easy for a developer(I have done it myself) to provide statistics from information which is readily available, rather than ask what information would be useful to debug problems – then collect and publish that information.

Using CISCO openconnect to tunnel to another system from Linux

I needed to use openconnect from CISCO to be able to logon from my Ubuntu system to someone else’s z/OS system.

This was pretty easy, but understanding some of the under the cover’s bits took a bit of time.

Basic install

  • Use sudo apt install openconnect
  • Download the VPNC script from http://www.infradead.org/ 
  • Create the configuration script
    • I saved the script as vpnc-script.sh
    • Using ls /etc/vpnc showed the directory did not exist. Create it and move the file
    • sudo mkdir /etc/vpnc/
    • sudo mv vpnc-script.sh /etc/vpnc/
    • sudo chmod +x /etc/vpnc/vpnc-script.sh
  • You need information from the owners of the vpn server.
    • vpn userid
    • vpn password
    • name of their system
    • IP address of their internal system
    • tso userid
    • tso password
  • I created a script (openc.sh), where XXXXX is the short userid, and password is your long userid:
    • printf ‘%s’ “password” | sudo openconnect –user=XXXXXX –script=/etc/vpnc/vpnc-script.sh vpn.customer.com
  • When you run openc.sh it prompts for your su password on the machine. The print… means you can store the password in the shell script. If you do not specify it, openconnect will prompt you for it.
  • Once the connection is made you can use ping, or x3270 -model 5 to access the system, where is the IP address the owner of the vpn server gave you.


The owner of the vpn server gave me the address of the z/OS machine, my userid and password.

I then used

x3270 -model 5 to logon to the system.

Hot key

I like to hot key to my z/OS sessions. I used Ubuntu “Settings”-> Keyboard shortcuts, and added a shortcut

  • name: mvsCust
  • Command: wmctrl -a
  • Hot key: Ctrl + H

The wmctl -a says make the window active which has in the window page title.

When I press Ctrl +H it makes the customers x3270 session the active window.

Change the x3270 colours

I wanted to change the screen colours, to distinguish it from other 3270 sessions. See Making x3270 green screens blue or red, or yellow with green bits.


I had to use SFTP colin@ to ftp to the remote z/OS system (where colin is my TSO userid).

What happens with openconnect, under the covers.

The handshake has several stages

  • Establish a TLS session using the certificate from the server. Once this has completed, any traffic is encrypted. In my case I used the vpn userid and password. The vpn server can be configured to accept certificates instead of userid and password.
  • The server sends down configuration information from the vpn server’s configuration. For example
    • The IP addresses it supports , such as and netmask
    • Any changes to the DNS configuration, so it knows to route via the VPN session.
    • The “banner” such as “Welcome to mycom.com. Users of this system do so at their own risk”.
    • A default domain.
    • Which tunnelling device to use – such as tun0.
    • How many configuration statements.
    • Each set of configuration statements.
    • You can see this information by using the -v option on the openconnect command.
  • Using the information sent from the the vpn server, the openconnect client creates environment variables.
  • The script defined (or defaulted, for example /etc/vpnc/vpnc-script.sh) on the openconnect command is invoked, and it uses these environment variables to manage the ip and dns configuration, changing files like /etc/resolv.conf (the local DNS file).

Oh p*x, I’ve lost my changes

I have been using pax to backup the files in my Unix Services directory and needed to restore a file so I could compare it with the last version ( and work out why my updates didnt work). Unfortunately I managed to overwrite my latest version instead of creating a copy.
I backed up my directory using

pax -W “seqparms=’space=(cyl,(10,10))'” -wzvf “//’COLIN.PAX.PYMQI2′” -x os390 /u/tmp/pymqi2/

This created a data set COLIN.PAX.PYMQI2 with the give space parameters, and os390 format.

To list the contents of this file use

pax -f “//’COLIN.PAX.PYMQI2′”

To display a subset of the files use

pax -f “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/code

which gave


And provide more information using the -v option

drwxrwxrwx 1 COLIN    1000      0 Jan 22 17:04 /u/tmp/pymqi2/code/
drwxr-xr-x 1 COLIN    1000      0 Feb 11 13:10 /u/tmp/pymqi2/code/pymqi/
-rw-r--r-- 1 OMVSKERN 1000 133011 Feb 22 13:15 /u/tmp/pymqi2/code/pymqi/init.py
-rw-r----- 1 COLIN    1000 119592 Feb  3 12:59 /u/tmp/pymqi2/code/pymqi/old__init__.old
-rwx------ 1 OMVSKERN 1000 119565 Jan 22 16:43 /u/tmp/pymqi2/code/pymqi/aa

The whoops

To restore an individual file and overwrite the original I used the -r option.

pax -rf “//’COLIN.PAX.PYMQI2′” /u/tmp/pymqi2/pymqi/__init__.py

I was expecting the file to be restored relative to the directory I was in; No – because I had backed up the files using an absolute path it restored the file to the same place, and so it overwrote my changes to the file. I had changed to a temporary directory, but I had not realised how the command worked.

There are several ways of doing it properly.

Restore with rename

pax -rf “//’COLIN.PAX.PYMQI2′” -i /u/tmp/pymqi2/pymqi/__init__.py

The -i option means rename.

I ran the command and it prompted me to rename it

Rename “/u/tmp/pymqi2/pymqi/__init__.py” as…


Set “do not overwrite”

I could also have used the -k option which prevents the overwriting of existing files.

Rename on restore

I could also have used the rename

pax -rf “//’COLIN.PAX.PYMQI2′” -s#/u/tmp/pymqi2/pymqi#/tmp/# /u/tmp/pymqi2/pymqi/__init__.py

Where the -s#/u/tmp/pymqi2/pymqi#/tmp/# / says use the regular expression to change /u/tmp/pymqi2/pymqi to /tmp and so restore it to a different place. Note: The more obvious -s/abc/xyz/, where / is used as the delimiter, would not work, as there is a ‘/’ in the file path.

All of the above

I could have use all of the options -i -k -s…. .

A better way to backup.

I had specified an absolute directory /u/tmp/pymi2/. If I was in this directory when I did the backup I could have used

pax … -x os390 .

Where the . at the end means from this directory, and so backup a relative directory.

If I list the files I get

pax -f “//’COLIN.PAX.PYMQI2A'” ./aa

And now if I restore the file…

pax -rf “//’COLIN.PAX.PYMQI2A'” ./aa

It restored the file into my working directory /tmp/aa .

So out of all the good ways of backing up and restoring – I chose the worst one. It only took me about 2 hours to remake all the changes I had lost.

How do I look at TLS 1.3 handshakes?

With TLS before 1.3, encryption of the data on the session occurred after the handshake had completed, so the handshake was visible in Wireshark. With TLS 1.3 the traffic is encrypted after the “Client Hello”, so you cannot immediately see the remainder of the handshake.

Tools like OpenSSL, can write out the magic data needed for decryption. For example

openssl s_client -keylogfile /tmp/kl -connect -cert /home/colinpaice/ssl/ssl2/ecec.pem -key /home/colinpaice/ssl/ssl2/ecec.key.pem -CAfile /home/colinpaice/ssl/ssl2/colinpaice.pem

This writes information to the specified file, in my case /tmp/kl.

The file has data like

SSL/TLS secrets log file, generated by OpenSSL

On Linux (Ubuntu) you can tell Wireshark to use this through

edit -> preferences -> protocols -> SSL -> (pre)-master-secret log filename

Different versions of Wireshark have TLS or SSL, use whichever one is available to you.

Specify the name of your file (/tmp/kl in my case) and Wireshark will be able to decrypt the data.

Can I define a disk Read Only to z/OS?

As part of migrating z/OS to a new service level, I wanted to mount old volumes Read-Only, so they were not updated when the new level was used. (For example z/OS updates the dataset last access time in the VTOC). I was running on zPDT, or z/OS on top of Linux, so all of the hardware is emulated. On a real machine you may be able to configure the storage subsystem.

I had four options

  • Make the disk on Linux read only – this worked, and was easy.
  • Copy the disks of interest so I had write access to a copy. This worked, and was easy.
  • Use the zPDT command awsmount 0ac5 -m /mnt/zimages/zOS/A4USR1 –readonly . This worked and was easy.
  • Update the Hardware Configuration Definition (HCD) to make a disk read only. I could define it, but not activate it because this read-only support is for PPRC mirrored disks. I could not vary the address online.

This blog post describes how I changed the HCD to add a read only disk.

This was a journey going into areas I had not been in before (creating IODFs).

The Hardware Configuration Definition(HCD) defines the configuration of the hardware. In day’s gone by the systems programmer would have to do a “sysgen” and used macros to define devices, then assemble it and use it. Nowadays you can maintain the configuration using ISPF panels.

What does the HCD do, and what is an OSCONFIG?

The documentation is not very clear about HCD. There are tiny clues, where it mentions making disks read-only, in OSCONFIG, but does not explain how to display and use the OSCONFIG. Now I know, it is easy.

  • You define each device, or group of similar devices in the HCD.
  • For each OS Configuration (OSCONFIG) you define each operating system image, and which devices belong in which OSCONFIG. See, … simple!

For example you define your configuration, including production and test devices, in the HCD. You then configure

  • A test system with only the test volumes
  • A production system with only the production volumes
  • The sysprog’s system with both test and production devices. From this machine, the systems programmer can create production or test configurations.

Getting started with HCD

The HCD is panel driven from ISPF.

You have to work with a copy of the IODF, and the system will generate a copy for you (suffixed with .WORK). I created a copy, made changes, then created a new IODF.

What is currently being used?

From the main HCD panel

  • 2. Activate or process configuration data
    • 5. View active configuration

Create a copy

From main menu use

  • 6. Maintain I/O definition files
    • 2. Copy I/O definition file

and follow the prompts.

On the home page it has the name of the current IODF being worked on, update it if necessary.

Display the OSCONFIG

Use the ISPF configuration panels for HCD:

  • 1. Define, modify, or view configuration data
    • 1. Operating system configurations

It then lists the available OSCONFIGs. Use / to select one, then select

  • 7. Work with attached devices

This lists the devices. You can scroll or use “L AF0” to locate the devices.

Put / in front to display the options. At the right it gives the command, so

  • 8. Delete . . . . . . . . . . . . . . (d)

I can either use /, and 8, or use ‘d’ (instead of the /) to delete an entry.

PF3 to return to “Define, Modify, or View Configuration Data”.

Add new devices


  • 5. I/O devices

This lists the devices. Use F11 to add

  • Device number 0af0
  • Number of devices 16
  • Device type 3390

Press enter.

It displays a list of OS Configs, select one.

  • option 1 select

You are prompted to configure the devices

  • OFFLINE No Device considered online or offline at IPL
  • DYNAMIC Yes Device supports dynamic configuration
  • LOCANY No UCB can reside in 31 bit storage
  • WLMPAV Yes Device supports work load manager
  • READ-ONLY Sec Restrict access to read requests (SEC or NO)
  • SHARED No Device shared with other systems
  • SHAREDUP No Shared when system physically partitioned

Press enter. To make this read-only I specified Shared=no and read-only=sec. (Sec is for secondary device. The read write copy of the mirrored is is the primary device).

Use PF3 to return.

Activate the configuration

From the HCD home page,

  • 2. Activate or process configuration data
    • 1. Build production I/O definition file

Create production eg “‘SYS1.IODF88”


  • 6. Activate or verify configuration dynamically

This displays

  • Currently active IODF . : SYS1.IODF99
  • IODF to be activated . : SYS1.IODF88
  • Test only . . . . . . . . Yes (Yes or No)

Use Test only = YES to validate it, then repeat with Test only = NO. This will make it live.

For me, the SYS1.IODFxx dataset, was created on the wrong volume. It has to be on the same volume as the SYS1.IPLPARM and other IPL information for a successful IPL.

Move the SYS1.IODF to the IPL parm volume.

Change your IPL loadxx member in SYS1.IPLPARM to point to the new IODF.

Although I had specified A4SYS1 as the volume for the SYS1.IODF88, SMS allocation routines allocated it on a different volume. I had to move it to the correct volume. See here.

Once I had IPLed with the new IODF

The command

D U,,,,0AF0,1 gave

0AF0 3390 F-NRD-RO                /RSDNT     0   

Which says there is no device mounted, but it has been defined as RO.

I varied it online and I got

IEE763I NAME= IECDINIT CODE= 000000000110088F

Which means it was unable to mount my disk as it was not part of a PPRC mirrored DASD environment. I had defined a disk as Read Only, but was not able to use it.

Running a headless Linux meant I was running disk less, and had no backups.

I had a Linux server and had a USB attached disk which I used to do backups. When I logged on after boot using the locally attached screen and keyboard the USB disk was visible. I configured an auto backup procedure, and checked it worked whenever I powered on the server.

I got into the a habit of using telnet to logon and accessing the system remotely. By chance, I checked to see if the backup disk was full, and found the disk was not visible. When I logged on with a local screen and keyboard, the disk was there, and had not been updated for over 100 days.

Digging around I found that USB disks can be mounted at startup or when a user logs on.

The mount information is in a file /etc/fstab.

I used the Ubuntu program “disks” to display and manage the disks. I selected a USB disk, clicked on the settings button, and selected “Edit Mount Options”. By default it had “User Session Defaults” on – which means mount the USB when a user logs on locally. I set

  • User Session Defaults off
  • Mount at system startup
  • Show in user interface
  • Mount Point /mnt/backup1

Next time I rebooted in headless mode, the disk was there as /mnt/backup.

I checked my backups – and they were done. I remember one of the points from when I use to do a MQ health check with customers.

Always check your backups are being done – and are backup what you expect.

Colin Paice

I should have paid more attention!

Initial setup for using a keystore on a HSM USB stick.

You can use a keystore on disk, but this inherently insecure, as people with administrator access to the machine, can copy the keystore. Using an external device (such as a USB Hardware Security Module) as a keystore, is more secure as you need physical access to the machine to physically access the keystore. If you have 3 failed attempts to access the keystore using a PIN code, the device locks up.

I found this document a good high level introduction to smart keys.

This post describes the initial set up for using the Hardware Security Module from Nitrokey for securely storing my digital certificates. It comes as a USB device. I chose it because it cost under 80 euros. There are other suppliers, such as yubico , and other suppliers but either they did not supply a price, or it was “call us and to discuss it”.

I found the Instructions that came with it via here, and a user blog very useful.

The Nitrokey HSM is open sourced, and uses open source facilities.

Software needed to use the key.

My machine is Linux Ubuntu 18.04.
You need software installed to configure it.

sudo apt install opensc pcscd pcsc-tools

To be able to use openssl you need an “engine” interface.

sudo apt install libengine-pkcs11-openssl

Once install you need to start it

sudo systemctl start pcscd
sudo systemctl status pcscd


● pcscd.service – PC/SC Smart Card Daemon
Loaded: loaded (/lib/systemd/system/pcscd.service; indirect; vendor preset: enabled)
Active: active (running) since Tue 2021-03-02 08:49:27 GMT; 2s ago

Display it (note it is two hypens)

opensc-tool ‐‐list‐readers

Gave me

#Detected readers (pcsc)
Nr. Card Features Name
0   Yes           Nitrokey Nitrokey HSM (DENK01051600000 ) 00 00

This shows the card is in

  • slot number 0. You may need this number when configuring keys, for example in openssl. This number is like a USB port number.
  • it is a physical card
  • it has no features listed
  • it comes from NitroKey and is a Hardware Security Module with the given serial number.

Set up

The device has

  • a device pin (SO-PIN) which is needed for administration, such as reinitialising the device or setting a user pin. This is 16 hex characters
  • a user pin to allow users access to modify keys. The user pin is a 6-15 digit string.

You need to consider how you use your device. You can have it self contained, and the private information is private to the device. This may be acceptable for a test device, but not in production, where you want to securely backup the keystore, and securely shared the key store between different machines. This can be done using Device Key Encryption Key (DKEK). The DKEK key is a 256-Bit AES key.

You can configure this so that you need more than one person to be able to enable a new device with this DKEK. You can configure n out of m people are needed. This is described here under Using key backup and restore.

You can use sc-hsm-tool – smart card utility for SmartCard-HSM, to

  • Initialize token, removing all existing keys, certificates and files.
  • Create a DKEK share encrypted under a password and save it to the file given as parameter.
  • Read and decrypt DKEK share and import into SmartCard-HSM
  • Define device pin for initialization
  • Force removal of existing key, description and certificate.
  • Define the token label to be used in –initialize.
  • Backup a private key to an encrypted external file. (Using the DKEK).
  • Restore a private key from an external encrypted file into the device, using the DKEK

You need to initialise the device see here.

I used

sc-hsm-tool –initialize –so-pin 3537363231383830 –pin 648219 –dkek-shares 1 –label mytoken
sc-hsm-tool –create-dkek-share dkek-share-1.pbe
sc-hsm-tool –import-dkek-share dkek-share-1.pbe

The command pkcs11-tool -L gave

Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM (DENK01051600000         ) 00 00
  token label        : UserPIN (mytoken)
  token manufacturer : www.CardContact.de
  token model        : PKCS#15 emulated
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 24.13
  firmware version   : 3.4
  serial num         : DENK0105160
  pin min/max        : 6/15

The token label : UserPIN (mytoken) has the name I entered in the –label option above. When using MQ and GSKIT with this device,I needed to identify device with “UserPIN (mytoken)” not just “mytoken”.

You can create a private key using

pkcs11-tool –keypairgen –key-type rsa:2048 –id 10 –label “my_key”

Using slot 0 with a present token (0x0)
Key pair generated:
Private Key Object; RSA
label: Private Key
ID: 10
Usage: decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
label: Private Key
ID: 10
Usage: encrypt, verify, wrap

You can omit the -id, and it will generate a (long) id for you. You can list the objects (in-use slots) in the device

pkcs11-tool -O

You can delete the one we just created

pkcs11-tool -l –pin 648219 –delete-object –type privkey –id 10

You can delete -type with privkey, pubkey and cert