Help ! My ZFS has filled up

The file system I was using in Unix Services filled up – but it didn’t tell me,I just had a truncated file. I piped the output of a shell script to a file. As the file system filled up – it could not write the “file system full” message.

To solve this file system full problem, I had to explore other areas of z/OS which I was not so familiar with – ADRDSSU to move data sets, zfsadmin commands, and how to stop SMS from being too helpful.

Having made the ZFS.USERS larger, I then found IEC070I 104-204 is data set is > 4GB. So some of this blog post is wrong!

Later I found the ZFS.USERS data set had 123 extents – and could not be expanded.

The Unix Services command

df -P /u/pymqi

tells you the file system – and how full it is. This gave me

Filesystem 512-blocks   Used Available Capacity Mounted on
ZFS.USERS      204480 203954       526     100% /u

So we can see the data set is ZFS.USERS and it is 100% full.

What is using all of the space?

du -a ./ | sort -n -r | head -n 30

The command

zfsadm fsinfo ZFS.USERS

give more (too much) information, and

zfsadm aggrinfo ZFS.USERS

doesnt quite give enough info. df -P … is best

I used the command

zfsadm grow ZFS.USERS -size 144000

to make it bigger, but I got the Unix Services message

IOEZ00326E Error 133 extending ZFS.USERS

and on the system log

IOEZ00445E Error extending ZFS.USERS. 591
DFSMS return code = 104, PDF code = 204.
IOEZ00308E Aggregate ZFS.USERS failed dynamic grow, (by user COLIN).
IOEZ00323I Attempting to extend ZFS.USERS to 36000 4096 byte control intervals.
IEF196I IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS,
IEF196I IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IEC070I 104-204,OMVS,OMVS,SYS00022,0A9E,C4USS2,ZFS.USERS, 594
IEC070I ZFS.USERS.DATA,CATALOG.Z24C.MASTER
IOEZ00445E Error extending ZFS.USERS. DFSMS return code = 104, PDF code = 204.

DFSMS return code = 104, PDF code = 204 means no space on the volume.

I used ISPF 3;4 to display the volume the ZFS.USERS data set was on; C4USS2.

I used ISPF 3;4 to display what data sets were on the C4USS2 volume. If you use PF11 to can see the space allocated to each data set.

I could try moving this dataset to another volume, but that would mean unmounting it, moving it, remounting it. I thought it easier to move other dataset off the volume.

On the volume, I found a ZFS which I was not using and unmounted it

unmount filesystem(‘ZFS.Z24C.ZCX’) normal

and trying to move it, looked easy using DFDSS COPY DATASET .

//IBMUSER1 JOB 1,MSGCLASS=H
//STEP1 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
ODY(C4USS1) DELETE CATALOG
/*

When I ran this job – it moved the dataset, but moved it to a USER00 volume – filling up most of the space on this volume. I had just moved the problem. SMS intercepted my request and “managed” the disk storage for me.

I added BYPASSACS and NULLSTORCLA

COPY DATASET(INCLUDE(ZFS.Z24C.ZCX))-
BYPASSACS(ZFS.Z24C.ZCX) –
NULLSTORCLAS –
ODY(C4USS2) DELETE CATALOG

and this worked.

• BYPASSACS – do not use ACS routines to decide where to put the data set
• NULLSTORCLAS ( or STORCLAS(xxxxx)) do not use a Storage class.
• ODY OUTDYNAM specifies that the output DASD volume is to be dynamically allocated.

I then had enough space to be able to grow the zfs.