Understanding Storage Controller caching and the caching statistics.

This started off with an investigation to see if my data set were using the new zHyperlink synchronous IO.   I wrote a program to process the SMF 42, dataset statistics, but did not understand the information.   The documentation talked about cache hits – which cache; the SMSPDSE cache or the Storage controller cache?

A short history of disk evolution

Computer DASD has been around for around 60 years.

• With early disks the operating system had to send commands to move the heads over the disk, and to wait until the right sector of the disk passed over the head.
• The addition of read cache to the disks. For hot data, the requested record may be in the cache – and so avoids having to read from the physical disks.
• Adding write, non volatile, cache and a battery meant there were two steps. 1) Send the data to the write cache, 2) move the data from the write cache to the spinning disks.
• The use of PC type disks – and solid state disks with no moving parts.

I found this document from 1996 very interesting. It describes the disk caching and other disk concepts.

The statistics in the data set portion of the SMF 42.6 (dataset), and the SMF 42.5 (storage class) come from the disk controller such as a 3990, so references to cache are for the cache in the controller.

Cache for reading from disk.

There are different data scenarios which affect the cache.

1. A program is reading sequentially;  for example a dataset with many records.  It would be good to say “read a whole cylinder’s worth of data from the disk into the controller’s cache, so the next records are in the cache when the next read is issued”.  The data which has been read can be removed from the cache.
2. A program is reading a record in a database to update it, or is getting  records from a file (or PDSE) where the data is in 4KB records scattered across the data set.  This is known as direct access. Just get one track’s worth of data – the minimum amount possible to cache.
3. Normal reading of a small file where requesting a record will cause the next few tracks to be loaded into the cache.

When a Channel Control Program (consisting of Channel Control Words) is issued to read some data, a hint can be passed down. The hints are, matching the scenarios above, with the fields in the SMF 42 records

1. Sequential access, Number of sequential I/O operations
2. Record (direct)  access, Number of record level cache I/O operations.
3. Normal cache, (Number of cache candidates – Number of write cache candidates)

For requests that may be cache unfriendly, the hint can be – “if it is not in the cache – go do the disk, and get it – and do not use the cache”. This is called Inhibit Cache Load, or ICL.

Cache for writing to disk

If you go back 30 years, a write to disk actually had to write to the disk. With the availability of Non Volatile Storage (NVS) and batteries, a write could go to the NVS, and be written to disk asynchronously possibly seconds or minutes later. This was known as DASD Fast Write (DFW).

If the NVS cache was full – the IO went through to the disk without using the cache – and was consequently much slower- the speed of disks, rather than memory speeds.

By default DFW is enabled, but the application could give a hint called “Bypass DFW” so the data went to the disk directly, and not to the cache. One example could be to avoid an application flooding the NVS cache and which would impact other applications using the disks.

What is a cache hit and cache miss – and how do I tell if this is being used?

There is a lot of documentation which mentions cache hit, and cache miss. Of course every one knows what a cache hit is, but I could not find how you tell if you have a cache hit or not. The DASD controller can report the number of cache hits, and misses, but not at a data set level. I haven’t found a document which officially tells me how you determine a cache hit or miss, but I believe the following is true.

If you break down the journey of an IO request from the Start Sub CHannel (SSCH) through the controller, to the disk, and the response returning, there are two parts called Connect and Disconnect.

Looking at a read request, during the connect stage, there is a request to the storage controller for a piece of data. If this is in the cache, it can be returned immediately, along with ‘end of request’. This time in the controller, the “Connection Time” duration, is passed back in a performance block to the originator. The disconnect time (see below) is 0.

If the data is not in the cache, the status goes to “disconnect”, and the request is passed to the disk itself. When the disk has retrieved the data, it passes it to the controller, which connects to the original request and passes the data back to the application. The connection duration, and the disconnect duration are passed back in a performance block to the host.

How do you tell if the data was in the cache or not? Easy – if the disconnect time is zero – the data was in the cache and only the cache was used. If the disconnect time duration is greater than 0, the data was not in the cache – so this is a cache miss.

For a write, if the write to the cache was successful – there the disconnect duration is 0. If the write had to go to the disk, the disconnect duration will be greater than 0. You can use the same argument. If the disconnect time duration is zero – the cache was used. If there is disconnect time, the cache could not be used, so this is a cache miss.

Why is there cache hits and write cache hits, and no read hits reported?

I think this is historical. Initially there was no cache. When volatile cache was available, there were statistics on cache usage. When non volatile cache for disk writes was implemented, they kept the “cache statistics” unchanged to save confusion, and added “write statistics”. This meant you had two fields – all cache requests, and write only cache requests. When you are looking at cache usage, you have to do the calculations yourself to obtain the read cache statistics.