I’m running on Linux, and using remote z/OS systems. Being from a performance background I hate having to waste seconds, manually starting SSH sessions to my backend systems.
I have two versions of z/OS, old and new(!). I had problems getting ssh to work because of key problems.
The problem
I tried to update my laptop key to the server
ssh-copy-id colin@10.1.1.2
This gave
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ERROR: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ ERROR: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ERROR: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! ERROR: Someone could be eavesdropping on you right now (man-in-the-middle attack)! ERROR: It is also possible that a host key has just been changed. ERROR: The fingerprint for the ED25519 key sent by the remote host is ERROR: SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98. ERROR: Please contact your system administrator. ERROR: Add correct host key in /home/colin/.ssh/known_hosts to get rid of this message. ERROR: Offending ED25519 key in /home/colin/.ssh/known_hosts:1 ERROR: remove with: ERROR: ssh-keygen -f '/home/colin/.ssh/known_hosts' -R '10.1.1.2' ERROR: Host key for 10.1.1.2 has changed and you have requested strict checking. ERROR: Host key verification failed.
Searching the internet I got suggestions saying “delete the old line from the file”. I didn’t want to do this because it meant I would not be able to go back to the old system and work as before.
Solutions
I edited /home/colin/.ssh/known_hosts and commented out line 1, with a # at the front (the :1 above is the first line). I repeated the command and it report the same message for line :2. I commented that out as well.
I got further
colin@ColinNew:~$ ssh-copy-id colin@10.1.1.2 The authenticity of host '10.1.1.2 (10.1.1.2)' can't be established. ED25519 key fingerprint is SHA256:2mUOVfdSedJVQIzZiGsRkOe9Vkc1bkyuDNp5H+VrZ98. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys colin@10.1.1.2: Permission denied (publickey,hostbased).
I had to start the SYSLOGD on z/OS to capture the output from SSHD.
In the /var/logSSHD (your’s may be different) it said
FOTS2307 User COLIN from 10.1.0.2 not allowed because not listed in AllowUsers
In my SSHD config file /etc/ssh/sshd_config I had
# Allow specific user IDs AllowUsers IBMUSER
I added COLIN to the list and restarted SSHD. (I do not know how to refresh SSHD)
This time the error log had
trying public key file /u/tmp/zowet/colin/.ssh/authorized_keys Could not open authorized keys '/u/tmp/zowet/colin/.ssh/authorized_keys': ...
I fixed this, tried to logon, and this time it worked.
On Linux, I edited /home/colin/.ssh/known_hosts and un-commented the lines I had commented out before. I tried the ssh command again, and it still worked!
I was downloading some stuff on one machine, and noticed that my Ethernet connection had a very high throughput – but it was doing nothing useful. This blog post gives some of the things I did to identify and resolve the problem.
Mount the file system
I used the command
sshfs colin@10.1.0.3:/home/zPDT/ ~/mountpoint
to mount the file system from 10.1.03 on my local machine.
Identify the problem
I used the Linux command nload to show the network activity.
For my wireless link (downloading a big file) the output was
I cannot currently reproduce the sustained Ethernet usage problem.
Wireshark showed my a lot of activity for SSH from port 55401 to port 22.
If you do not have access to Wireshark, the following command show all the socket activity which may help.
ss -t -a -i -O |grep delivery|awk '{print $4,$5, " ", $30,$31 }'
To find the owner of port 55401 I used the show socket command
The documentation for sshfs says use the fusermount3 command.
$fusermount3 -u ~/mountpoint fusermount3: failed to unmount /home/colinpaice/mountpoint: Device or resource busy
I needed to use the lazy unmount option -z
fusermount3 -z -u ~/mountpoint
and this successfully unmounted the remote file system
Chaff
I found out that information can be obtained from the profile of key strokes, and so chaff has been added to the SSH flow.
I fixed it by using setting ObscureKeystrokeTiming no in /etc/ssh/ssh_config. The documentation says
Specifies whether ssh(1) should try to obscure inter-keystroke timings from passive observers of network traffic. If enabled, then for interactive sessions, ssh(1) will send keystrokes at fixed intervals of a few tens of milliseconds and will send fake keystroke packets for some time after typing ceases. The argument to this keyword must be yes, no or an interval specifier of the form interval:milliseconds (e.g. interval:80 for 80 milliseconds). The default is to obscure keystrokes using a 20ms packet interval. Note that smaller intervals will result in higher fake keystroke packet rates.
ColinPaice 