Setting up and using SMDS

MQ on z/OS provides shared queue, where a coupling facility structure is shared amongst queue managers in a sysplex to provide a queue sharing group.

You can set up Shared Message Data Set(SMDS) to handle large messages over 63 KB in size, or to provide additional capacity, so as the structure fills up messages are written to a data set – but available from all queue managers in the QSG. Ultimately the structure just has pointers to messages on the data set.

For messages under 63 KB in size, instead of using SMDS you can use the z/OS structure overflow called Storage Class Memory. I think of this as paging for a structure. If you have many messages on a queue it will “page out” the messages in the middle out of the queue, and as the queue is processed sequentially it will “page in” in the messages before they are needed.

There is lots of good information on SMDS, but I found it hard to locate the information I needed.

• Planning your shared message data set (SMDS) environment
• Set up the SMDS environment

This blog post aims to fill some of the holes.

Before you start

The term CFLEVEL is used in two contexts, MQ and z/OS

• z/OS Coupling Facility Resource Management (CFRM) uses a CFLEVEL. For example ,the CFRM definitions include if the structure is duplexed. If CFRM CFLEVEL > 21 then it supports asynchronous duplexing.
• MQ has a CFLEVEL. MQ CFLEVEL(5) allows offload, and allows the queue manager to tolerate a loss of connectivity to the CF structure.

SMDS requires the CF structures to have MQ CFLEVEL(5). This is not a major change, but needs to be planned. Increasing the MQ CFLEVEL in the z/OS CF structure definition, will probably increase the size of the structure.

In “Backing up and recovering data in SMDS” below, it discusses having a queue manager within the QSG just for backing up and recovering messages in structures and especially when there are a large number of messages in SMDS. You need to consider this, and decide if you want to implement it. This can be done at any time. You may need to review how often you backup your MQ CF structures.

Backing up (and recovering) data in SMDS

Unlike MQ pagesets you do not recover an SMDS by restoring it from a backup. The CF structure and the SMDS are one logical entity, with the structure pointing to the location on disk. If you restore a backup of an SMDS, the CF will not have the matching information and you will have corrupt data.

Instead of backing up the SMDS you use the MQ BACKUP CFSTRUCT command. This reads the CF and any messages on the SMDSs, and writes the content to the log data sets. Ideally you do this when the CF structure is close to empty. If you have a large SMDS full of messages, this will write a lot of data to the active logs, possibly impacting normal throughput.

When the RECOVER CFSTRUCT command is issued (to recreate the data) , the log is read to recover the data. If you have normal message data logged, this could mean that a lot of log data needs to be read to recover the structure, because the normal message traffic and the backup data will be intermixed on the log.

Some customers have a queue manager dedicated to backing up the CF Structures. This has the following advantages:

• The backup activity writing to the active log does not impact normal business activity
• When recover cfstruct is issued, only the data for CF structures is in the logs, and so less data overall needs to be read from the logs, which means that recovery is faster.

Create the SMDS

Each queue manager needs its own SMDS, the queue manager can read and write to its own SMDS.

A queue manager can read the data in another queue manager’s SMDS, (but updates are only made from the owning queue manager).

The SMDS can be of different sizes, for example if two queue managers are “customer facing”, and two queue managers are “back end facing” the “customer facing” SMDS may get a lot of activity, and the “back end facing” may get no activity because these queue managers only do gets.

You allocate an SMDS with a primary size, and can specify a secondary size. If a secondary size is zero or not specified, and expansion is allowed, then the queue manager will try expanding with a secondary space of 10% of the existing size. This means the 100^th extent will be larger than the 10^th extent.

An SMDS can have up to 255 extents. If the data set is SMS managed and the SMS configuration has Extent Constraint Removal specified in the DFSMS Data Class, then the data set can be on up to 59 volumes, with up to 123 extents per volume (or 7257 extents). See here for more information.

You cannot make the SMDS smaller.

If enabled, automatic expansion is attempted when the SMDS is 90% full. If the message rate is not too high then the automatic expansion may be enough so the SMDS does not get 100% full.

If there is a spike in workload, then the expansion may not be able to keep up, and so some requests will get no space available, until any in-progress expansion has completed. In this case to avoid applications getting the “no space” condition you should allocate the SMDS with a primary extent big enough to hold the expected peak of messages, or allocate a queue on the CF structure, and put many messages to cause expansion (then drain or delete the queue).

Note: The time taken to expand the SMDS depends on the size of the increment. Larger extents mean there are more pages to format, and so take longer.

Allocating the data set

When the CF Structure is not full, then most of the messages may be in the CF Structure. As the CF becomes fuller, more messages are written to the SMDS. If all messages are on an SMDS, and there is a high message rate, the SMDS will have a lot of I/O activity, read and write.

You need to allocate the SMDS on fast disks to minimise response time of processing messages.

If you are not using SMS managed data sets, you need specify additional volumes to allow for expansion.

What logical block size should be used?

You need to know the profile of the size of your messages so you know what value to use. Each page within SMDS is 4KB. The block size defines how many of these 4KB pages are allocated to each message. I/O is done at the block level.

The default DSBLOCK is 256K.

Once an SMDS has been opened the DSBLOCK value cannot be changed, so you need to plan this before you start to use the SMDS.

If you find the value is not optimum, then

• if the value is too large, the impact is more disk space will be used and more 64 bit buffers are used.
• if the value is too small, then more I/O operations (of smaller blocks) will be done, and the time to process messages may take longer (possibly just a few milliseconds).

If you have big messages then a DSBLOCK value of 256K may be OK. If you have many small messages (under 64K) then 64K or smaller may be better.

Defining SMDS to MQ

If you define a new CF Structure and specify CFLEVEL(5) then the offload parameter defaults to SMDS.

If you alter an existing CF Structure from a lower CFLEVEL to CFLEVEL(5) the OFFLOAD defaults to DB2. This was done for migration reasons.

Once you have created your SMDS for each queue manager. You use the

• alter cfstruct.(..) cflevel(5)
• alter cfstruct(..) DSGROUP(‘data.set.name.*’) DSBLOCK(…)
• alter cfstruct(…) offload(SMDS)

The number of buffers (of size DSBLOCK) is specified in DSBUFS. This can be set on a queue manager using the ALTER SMDS DSBUFS command.

This value can be changed dynamically, it defaults to 100.

You need to work with the z/OS systems programmer before using this to make sure there is enough resources. For example I defined the block size as 1MB, and defined 9999 buffers. This is over 9 GB of virtual storage and z/OS ran out of auxiliary storage and real storage. I had to shutdown the QSG and reset the size in CSQINP2.

When reading from a different queue managers’ SMDS, a buffer is allocated from the pool, data is read in, and the buffer is released back to the pool.

When a queue manager is writing to its own SMDS, a buffer is obtained, written to the SMDS and the buffer released(still with its content). If an MQGET is issued, the data may be in a buffer, and so avoid disk I/O.

Generally SMDS works ok with a small number of buffers – as long as there are enough buffers for the concurrent requests. Having more buffers may make little difference to performance. You need to look at the statistics to see if there were buffer shortages.

Other SMDS activities

If you need to work with the SMDS, you can use the RESET SMDS command to disable access to the SMDS. Message processing which does not use the SMDS can continue.

The RESET SMDS ACCESS(DISABLED) causes all of the queue managers to close it normally and deallocate it. When the data set is ready to be used, it can be altered to ACCESS(ENABLED) allowing the queue managers to access it again.

Media recovery

If there is a problem with shared queue, the CF structure may need to be rebuilt, and the messages restored from the log data sets.

One queue manager does the Media recovery. This needs access to the log of all the queue managers in the QSG, and needs update access to the SMDS data sets.

Automatic expansion

You can enable automatic expansion of the SMDS using DSEXPAND(YES) .

You can disable automatic expansion of the SMDS using DSEXPAND(NO). You may want to do this is you have an exceptional peak, and do not want the SMDS to expand(because you cannot make an SMDS smaller). If the CF structure or the SMDS is full a put request will get no space available reason code.

Automatic expansion may fail

• If the number of extents exceeds the limit
• If there is no space available to expand. You may be able to fix this by making more volumes available to be used. You may need to talk to your data manager about the best way to resolve this problem.

What to monitor?

SMF

You should collect SMF data from a good day, so if you have a bad day, you have some comparison data.

SMDS writes statistics data to SMF. A program to print the SMF data, is available in SupportPac MP1B. This describes the fields, and how to interpret the data. There are other tools which format the data, but they do not tend to interpret the data and give useful information.

Information available includes

• I/O requests to an SMDS, write to local, read from local, read from other qmgr’s SMDS
  • count of requests,
  • number of pages,
  • average I/O duration,
  • average wait before the I/O could be started.
• Numbers of buffers
  • Total,
  • In use,
  • Current count of waiting for a free buffer,
  • Highest count in the interval of waiting for a free buffer,
  • Current count of waiting for a busy buffer,
  • Highest count in the interval of waiting for a busy buffer,
  • Number of times there were no buffers available.

Display command

SupportPac MP16: Capacity Planning and Tuning for IBM MQ for z/O describes SMDS. It gives information on DSBLOCK and impact of message size, and other good information.

If there are insufficient buffers to handle the maximum concurrent number of I/O requests, then requests will have to wait for buffers, causing a significant performance impact. This can be seen in the CSQE285I message (issued as a response to the “DISPLAY USAGE TYPE(SMDS)” command) when the “lowest free” is zero or negative and the “wait rate” is non-zero. If the “lowest free” is negative, increasing the DSBUFS parameter by that number of buffers should avoid waits in similar situations.

MP16

The number of extents

You can use MQ commands (or MQ SMF data) to monitor the size of SMDS. You can also automate the expansion messages as expansion should be an exception rather than an every day occurrence.

You might prepare for SMDS expansion deciding in advance if you want to use SMDS expansion, or disable expansion, and prepare the system operations to handle this.

MQ commands relating to SMDS

Some of the commands for SMDS are dissimilar to normal MQ commands. For example

DIS smds(*) CFSTRUCT(CSQSYSAPPL)

gets a reply from all queue managers in the QSG. You can issue the following command from queue manager QM01, about queue manager QM02

QM01 DIS SMDS(QM02) CFSTRUCT(CSQSYSAPPL)

Or you could just issue the command to QM02

QM02 DIS SMDS(QM02) CFSTRUCT(CSQSYSAPPL)

I forget this every time!

• ALTER CFSTRUCT. Define OFFLOAD parameters, data set name, thresholds for offloading messages to SMDS. Change numbers of buffers for QSG specific value, the number of buffers for individual queue managers, can be overwritten by ALTER SMDS.
• ALTER SMDS. Change the number of buffers for a queue manager; Change DSEXPAND().
• DISPLAY CFSTATUS. Display the status of one or more CF application structures.
  • DISPLAY CFSTATUS TYPE(SMDS). Display shared message data set information. You get a reply from each queue manager.
• DISPLAY CFSTRUCT. Display the attributes of one or more CF application structures.
• DISPLAY SMDS. Display the parameters of existing SMDSs associated with a specified application structure.
• DISPLAY SMDSCONN. Display status and availability information about the connection between the queue manager and the shared message data sets for the specified CFSTRUCT
• RECOVER CFSTRUCT. Recreate the persistent messages in the CF (and SMDS) from the logs.
• RESET SMDS. Modify availability or status information relating to one or more shared message data sets associated with a specific application structure. For example make SMDS not available to queue managers, while it is made larger.