How do you download and use a dataset from z/OS.

Transferring a dataset from z/OS to Windows or Linux and using it can be a challenge.

A record in a data set on z/OS has a 4 byte Record Descriptor Word on the front of the record. The first two bytes give the length of the record (and the other two bytes are typically 0)

FTP has two modes for transferring data ASCII and BIN.

ASCII

With ASCII mode, FTP reads the record,

Removes the RDW
Converts it from EBCDIC to ASCII
Adds a “New Line” character to the end of data
Sends the data
Writes the data to a file stream.

On Unix and Windows a text file is a long stream of data. When the file is read, a New Line character ends the logical record, and so you display the following data on a “New Line”.

Binary mode

Binary mode is used when the dataset has hexadecimal content, and not just printable characters. The New Line hex character could be part of a some hexadecimal data, so this character cannot be used to delineate records.

FTP has an option for RDW

quote site RDW

The default is RDW FALSE.

If RDW is FALSE then FTP removes the RDW from the data before sending it. At the remote end, the data is a stream of data, and you have no way of identifying where one logical record ends, and the next logical record starts.

If RDW is TRUE, then the 4 byte RDW is sent as part of the data. The application reading the file can read the information and calculate where the logical record starts and ends.

For example on z/OS the dataset has (in hex) where the bold data is displayed when you edit or browse the dataset. The italic data is not displayed.

00040000C1C2C3C4
00020000D1CD2
00050000E1E2E3E4E5

If the data was transmitted with RDW FALSE the data in the file would be

C1C2C3C4D1D2E1E2E3E4E5

If the data was transmitted with RDW TRUE the data in the file would be

00040000C1C2C3C400020000D1CD200050000E1E2E3E4E5

Conceptually you can process this file stream using C code:

short RDW;  // 2 byte integer
short dummy; // 2 byte integer

RDW = fread(2); // get the length
dummy = fread(2); // ignore the 0s
mydata = fread(RDW -4); //  -4 for the RDW already read 

...
RDW = fread(2); // get the length
dummy = fread(2); // ignore the 0s
mydata = fread(RDW -4); //  -4 for the RDW already read

(Thanks to pjfarley3 who pointed out the RDW length includes the 4 byte RDW – so the application data length is RDW -4.)

In practice this will not work because z/OS has numbers which are Big Endian, and X86 and ARM machines are Little Endian. (With Big Endian – the left byte is most significant, with Little Endian, the right bit is most significant – the bytes are transposed.)

On z/OS 0x0004 is decimal 4. On X86 and ARM 0x0400 is 4.

In practice you need code on X86 and ARM, like the following, to get the value of a half word from a z/OS data set.

char RDW[2];  // 2 characters
RDW = fread(2); // get the length
length = 256 * RDW[0] + RDW[1]

and similarly for longer integers.

Python

If you are using the Python struct facility, you can pass a string of data types and get the processed values.

The string “>HH” says two half words, and the > says the numbers are Big Endian.
The string “<HH” says two half words and the < says they are Little Endian
The string “HH” says two half words – read in the default representation.

Conversion

You’ll need to do your own conversion from EBCDIC to ASCII to make things printable!

How to get a file from z/OS to a different z/OS without using FTP

I have a userid on a z/OS production system, which does not support FTP. To run my tests, I needed to get some files on to this system. Getting the files there was a challange.

The 3270 emulator has support for transferring files. It uses the IND$FILE TSO command to send data packaged as 3270 datastream As far as I can tell, this only works with data sets, not Unix files.

Creating a portable file from a data set.

You can package a data set into a FB Lrecl 80 dataset using the TSO XMIT (TRANSMIT) command.

Create a portable dataset from a Unix file.

On my home system I created a PAX dataset from a file in a Unix directory.

Use cd to get into the directory you want to package. If you specify a file name like /tmp/mypackage, the unpax will store the output in /tmp/mypackage which may not be where you want to store the data.

If you use relative directories such as ‘.’ it will unpax into a relative directory. I used the cd command to get into my working directory

pax -W "seqparms='space=(cyl,(10,10))'" -wzvf  "//'COLIN.ZOWE.PAX'" -x os390  myfile

You need both the single and double quotes around the data set name.

This created a data set with record format FB, and Lrecl 80.

A 360 MB file became a 426 CYL data set.

If you run out of space ( B37-04 abend). Delete the dataset before you reissue the pax command, otherwise the space parameters on the pax command are ignored; and increase the amount of space in the pax command.

I FTPed this down to my Linux machine in binary mode.

Send the file to the remote z/OS over 3270 emulator

Because FTP was not available I had to use the TSO facility IND$FILE. One of the options from the “file” menu was “File Transfer”.

You fill in details of the local file name, the remote data set name, and data set attributes.

In theory you need to be in TSO option 6 – where you can enter TSO commands, but when I tried this I kept getting “input field too small”. I had to exit ISPF and get into native TSO before the command worked.

The transfer rate is very slow. It sends one block at a time, and waits for the acknowledgement. With TCP/IP you can send multiple blocks before waiting for the ack, and use big blocks. For a 300MB file, I achieved 47KB per second with a 16000 block size – so not very high.

With IND$FILE, pick the biggest block size you can. I think it supports a maximum size of 32767. I got 86 KB/second with a 32767 block size with DFT mode.

For a dataset packaged with TSO XMIT

Use the TSO command RECEIVE INDSN(…) to restore the data set.

Un PAX the file to recreate it

On the production system, I use went into Unix, and used the cd command to get to the destination directory.

pax -ppx -rf  "//'COLIN.ZOWE.PAX'"

CS IP Filtering: Trying to use FTP

With my FTP, the initial connection worked on port 21, then it switched to use a different port for the file transfer. These were not set up on my system.

On my FTP client, I received

229 Entering Extended Passive Mode (|||1028|)

which means it was trying to use port 1028. This was not configured, and so the transfer failed.

You have to configure FTP to use a range of ports, and then configure the Policy Agent with these ports.

In the FTPD JCL I have

//FTPD   PROC MODULE='FTPD',PARMS='' 
//FTPD   EXEC PGM=&MODULE,REGION=4096K,TIME=NOLIMIT, 
//       PARM='POSIX(ON) ALL31(ON)/&PARMS' 
//CEEDUMP  DD SYSOUT=* 
//SYSFTPD DD DISP=SHR,DSN=TCPIP.FTP.DATA 
//*       SYSTCPD explicitly identifieS which file iS to be 
//*       uSed to obtain the parameterS defined by TCPIP.DATA. 
//*       The SYSTCPD DD Statement Should be placed in the JCL of 
//*       the Server.  The file can be any Sequential data Set, 
//*SYSTCPD DD DISP=SHR,DSN=TCPIP.SEZAINST(TCPDATA) 
//SYSTCPD DD DISP=SHR,DSN=ADCD.&SYSVER..TCPPARMS(TCPDATA)

Within the SYSFTPD dataset you need to configure the PASSIVEDATAPORTS statement.

See setting up FTP Daemon on z/OS.

Setting up FTP Server on z/OS

I had problems using the FTP server on z/OS once I had configured IP Filtering (any IP addresses and ports which are not in the configuration are dropped).

Once an FTP client has established contact using port 21, the transfer switches to a different port. You can control which port range is used.

Setting up the FTP server.

If you start the FTP Daemon (S FTPD) it starts up another procedure (FTPD1) and the initial job ends.

You need to configure the SYSLOGD daemon to capture any output from the FTP task.

In /etc/syslog.conf I set up

*.FTPD*.*.*         /var/log/FTPD.%Y.%m.%d

so any output from jobs with a name FTPD* will go into the specified file.

In my file I had

EZYFT46E Error in dd:SYSFTPD=TCPIP.SEZAINST(FTPSDATA) file: line 1283 near column 1.
EZY2642E Unknown keyword: PASSIVEDATAPORTS(8000,8100)

Configuring ports

To limit which ports FTP uses you need to specify PASSIVEDATAPORTS

PASSIVEDATAPORTS (8000,8100)

with a blank between the keyword and the (.

You also need to tell TCPIP that the port range is reserved for TCPIP’s use for example

PORTRANGE
... 50000 100 TCP AUTHPORT

Where AUTHPORT Indicates that all ports in the port range are not available for use by any user except FTP, and only when FTP is configured to use PASSIVEDATAPORTS. AUTHPORT is valid only with the TCP protocol.

When you try to transfer a file you get a message

ftp> get ‘…’ … local: …: ‘…’
229 Entering Extended Passive Mode (|||8061|)

where 8061 is the port which was used.

IPSEC definitons for IP Filtering

For my very restrictive access from my laptop to z/OS (and no other devices) I used

IpFilterRule FTPnI21 
{ 
  IpSourceAddrGroupRef     zGroup 
  IpSourceAddr   10.1.0.2 
  IpDestAddr     10.1.1.2 
  IpGenericFilterActionRef permitlog 
 IpService 
 { 
    Protocol Tcp 
    DestinationPortRange 21 
    Direction inbound 
    Routing local 
 } 
 IpService 
 { 
    Protocol Tcp 
    DestinationPortRange 8000-8100 
    Direction inbound 
    Routing local 
 } 
}

and

IpFilterRule FTPO21 
{ 
  IpSourceAddr   10.1.1.2 
  IpDestAddr     10.1.0.2 
  IpGenericFilterActionRef permitlog 
 IpService 
 { 
    Protocol Tcp 
    SourcePortRange  21 
    Direction outbound 
    Routing local 
 } 
 IpService 
 { 
    Protocol Tcp 
    SourcePortRange  8000-8100 
    Direction outbound 
    Routing local 
 } 
}