Migrating your queue manager in an enterprise.

Migrating an isolated queue manager is not too difficult and some of it is covered in the MQ Knowledge Center.

Since my first blog post on this topic, Ive had some comments asking for more detailed steps… so I’ve added a section at the bottom.

Below are some other things you need to think about when working in an enterprise when you have test and production, multiple queue managers per application (for HA or load balancing) and multiple applications. I am focusing on midrange MQ, and not z/OS though many of the topics are relevant to all platforms.

Consider the scenario where you have 4 queue managers
QMA and QMB supporting MOBILE and STATUS applications,
QMX and QMY supporting PAYROLL application.

You want a low risk approach, so you decide to upgrade the STATUS application first. This application uses QMA and QMB which are also used by business critical application MOBILE. This would be a high risk change.
It would be safer to to first migrate application PAYROLL on QMX and QMY.

Looking at QMX and QMY.
You could migrate both queue managers the same weekend – this would be least work, but has a risk that you do not have a good fall back plan if it does not work as expected.
You could migrate QMX this weekend, and QMY next weekend if there were no problems found.
If QMX has problems you can continue using QMY while you resolve problems. If QMX has problem, then if QMY has problems or is shut down you have an availability issue, so you may want to define a new environment with QMZ (and the web server etc – so not a trivial task).

As well as production QMX and QMY you have test systems: You need to plan to migrate and test these pre-production systems before considering migrating production. While the test and production levels of MQ are different, you may want to freeze making application changes, and factor this in the plan.

If you have a machine with one MQ level of code, and multiple queue managers on it, you cannot just migrate one queue manager, as you delete the MQ executables and install the new version. You can use multiple installed levels of MQ – but you may have to migrate to this before exploiting it. See Multiple Installations.

Clustering. Remember to migrate your full repositories first – you might want to consider creating dedicated queue managers for your repositories if this is a problem.

License: You will need licenses for the versions of MQ you use. The MQ command dspmqver gives you information about your existing installation. Some licenses entitle you to support from IBM, others are for development or trial use.

There are three stages to migrating applications.

• Run the applications with no changes on the upgraded system. These should run successfully, but MQ may do more checks, for example some data is meant to be on a 4 byte boundary. MQ now polices this.
• Recompile the applications to use the newer MQ libraries. Some application MQ control blocks may be larger, and this may uncover application coding problems. For example uninitialised storage.
• Exploitation of new function. Do this once you have successfully migrated the existing queue managers.

Testing: You need to test the normal working application, plus error paths, such as messages being put to a Dead Letter Queue, and making sure this process works.

Update your infrastructure harness: You need to review what new messages your automation processes, and what actions to take.
You need to decide what additional statistics etc to use and what reports you want to product for capacity planning, health review and day to day running.

You have to worry about applications coming in to your queue manager. For example what levels of MQ are they using. They may need to be rebuilt with the newer libraries. The client code may need to up upgraded. You can use the DIS CHS(…) RVERSION to display the level of MQ client code. Of course your challenge will be to get people outside of you organization to update their code – especially when they say they have lost the source to the program.

MQ is rarely used in isolation. You may need to upgrade web servers to a newer level which support the new level of MQ.

You may need to upgrade the hardware and operating system.

Going down to the next level of detail.

Exits

You need to check any exits you have can support new functions and different levels of control blocks. For example there are shared connections, and the MQMD can change size from release to release.

If you cannot have one exit that supports all level of MQ. You’ll have to manage how you deploy the exit matching the queue manager level.

TLS and SSL setup

• You need to review the TLS and SSL support. Newer levels of MQ removes support for weaker levels of TLS.
• You need to review the end user certificates to make sure they are using supported levels of encryption.
• You need to review the cipherspecs used by SSL channels, and upgrade them before you migrate the queue manager. (You could migrate to a newer version and see which channels fail to start, then fix them, but this is not so good).
• As part of this cipherspec review you may wish to upgrade to strong cipher specs which use less CPU, or can be offload on z/OS.
• You may have a problem sharing keystores, and make sure you include the keystore files in your backups. See APAR IT16295.

Building your applications

• In some environments application developers compile programs on their own machines; in other environments, there is a process to generate applications on a central build machine. You will need to change the build environment to have the newer version header files, and change the build process to be able to use them.
• You will need to set up a build environment so you can use the MQ V9 header files for just the application being migrated.
• You many need to change your deploy tool so that the program compiled at MQ V9 is only deployed to TESTQMA, ( at MQ V9) and not to TESTQMB(still at MQ v7).
• You need to change your deploy tool for test, pre-production and production.

Using the Client Channel Definition Table (CCDT)

• Older clients must continue to use existing CCDT
• Newer clients are able to understand older CCDTs.
• For an application to use a newer version CCDT, you must update the MQ client.
• So you need to be careful about moving the CCDT file around

System management applications

You may have home grown applications that are used to manage MQ. These need to be changed to support new object types( such as chlauth records and topics) and new fields on objects. You cannot rely on a field of interest being the 5th in the list as it was in MQ V5.

MQ Console (MQWEB)

If you are using the MQ Console server to provide a web browser or REST API to a queue manager, you may need to do extra work for this.

You have an instance of MQ Console to support MQ V9.0 and a different instance to support MQ 9.1

If you have multiple queue managers on a box, and plan to to use MQ Multiple Installation to migrate one queue manager at a time, then you will to consider the following

• The box has QMA and QMB on it at MQ V9.0
• These box use MQCONSOLE-XX with port 9090
• Install MQ 9.1 on the same box.
• Migrate QMA to 9.1
• Create an MQCONSOLE-YY at MQ 9.1 with port 9191
• Change your web browser URL and REST api apps to use port 9191
• Wait for a week
• Migrate QMB to 9.1
• Migrate MQCONSOLE-XX to 9.1
• Web browser URL and REST API url can continue using port 9090
• Shutdown MQCONSOLE-YY
• Undo any changes to change your web browser URL and REST api apps to use port 9191 and go back to port 9090

“The rest of the stuff”

I remember seeing a poster of child sitting on a potty with a caption saying “no job is complete until the paper work is complete”.

Someone said that doing the actual migration of a queue manager took 1 hour. Doing the paper work ; planning, change management, talking to user etc took two weeks per queue manager.

And yes, you do need to update your documentation!

Education

You need to talk to the teams around your organization. This is mainly applications – but other teams as well ( eg monitoring, networking)

• Tell them what changes you will be making, the time scales etc..
• There will be an application freeze during the migration.
• The application teams will need to test their applications, and may need to make changes to them.
• The application teams will get these new events/alerts which they need to handle.
• You may learn about how they use MQ, and how this will affect your migration plans. (We used this unsupported program for which we have no source and no one knows how it works – which is critical to our business).
• You may get a free trip to an exotic location to talk to the application teams (or you may get told to go to some hell hole)
• You need to talk to people outside if your organization. The hard bit may be finding out who they are

Security

• You need to protect any new libraries.
• MQ may have new facilities such as topics which you need to develop and implement a security policy for. In V9 MQ midrange now publishes statistics to a topic.
• Your tools for processing MQ security events, may need to be enhanced to handle new resource types or new events.

New messages and events

You need to review all new events or messages, and add automation to process them. You need to decide who gets notified, and what actions to take.

You need to review changed messages or alerts in case you are relying on “constant” information in particular place in the message, which has been changed.

Backups

People often dump the configuration of their queue managers every day, so they can use runmqsc to recreate the queues etc. You need to backup all objects including topics and chlauth records, and check you can recreated them in a queue manager.

Backup your mq libraries for queue manager and clients – or be able to redeploy them from your systems management software.

Performing the migration

This is documented in the Knowledge Centre. One path for migration involves deleting the old level of MQ and installing the new level of MQ. If you need to go back to the old level, you need to have a copy of the old level of MQ base + CSD level as you were running on!

Carefully check the documentation for the hops.

The Migration paths documentation says

• You can migrate from V8.0 or later direct to the latest version.
• To migrate from V7.0.1, you must first migrate to V8.0.
• To migrate from V7.1 or V7.5, you must first migrate to V8.0 or V9.0.
• You might have an extra step to go to MQ V9.1

I found some really old doc saying

“If you are still on MQ version 5.3, you should plan a 2 step migration: first migrate to MQ v7.0.1 then migrate to 7.1 or 7.5”. This could be a challenge as you can not get the MQ 7.0.1 or the MQ 7.1 product. One of the reasons for this two stage approach is that the layout of files changed, so you have to restart at MQ 7.0.1 to make these file system changes.

Finally…

If I have missed anything or got something wrong, please let me know and I’ll update the list

Checklist for implementation

Different stages

• Pre-reqs
• Education for team doing migration
• Investigate – until you have done the investigation you cannot plan the work. For example how many exits are used, and how many need to be changed.
• Plan. The first time you do something you may be slow. Successive times should be faster as you should know what you are doing.
• Implement/Migrate
• Exploit new features.

Before you start

• People doing the work need access to systems
• Need to draw up a schedule (but you may need to do the investigation work before you know how much work there is to do)
• Appoint a team leader.
• Determine what skills you need, eg TLS, application design, build
• Which people do admin – which people handle code eg review exit programs
• Reporting and status
• Communication with other teams – we will be migrating in.. and you will be asked to do some work..
• Extract configuration to common disk, so people do not need to access each queue manager.
• External customers – provide one list of changes for them if possible. This is better than giving them multiple lists of changes, and will help them understand the size of their work.

Education

• Ensure every one has basic knowledge
  • MQ commands
  • Unix commands
  • TLS and security( and stop using SSL)
  • Manage remote MQ from one site using remote runmqsc command or logon to each machine
  • Efficient way of processing data
    • Use GREP on a file to find something, pipe it … sort it, do not find things by hand
• How the project will be tracked

Areas for migration

• TLS parameters and using stronger encryption
• TLS certificate strength
• Exits
• Applications
• Queue manager
• Clients using the queue manager. A client may be able to connect to many queue managers.

Investigate SSL/TLS

• Which TLS parameters are being used
• Which ones are not supported in newer versions of MQ?
  • SSLCIPH
  • Need to worry about both ends of the connection
• Identify “right” TLS parameters to use
  • eg Strong encryption which can be offloaded on z/OS.
• Will these cost more CPU? Is this a problem?
• If TLS not being used – document this

Implement TLS

• Need a plan to change any cipher specs which are out of support.
• May need to make multiple changes across multiple queue managers at “same time” – coordinate different ends
• Can be done before MQ migration.
• Can be done AFTER MQ migration if you set a flag.
  • May make implementation easier
  • Still need a plan to change any which are out of support.
  • Still may need to make multiple changes across multiple queue managers at “same time”

Investigate certificates

• Investigate if certificates are using weak encryption
  • Which certificates need to be changed? May need RACF/Security team to help report userids that need to change
• Plan to roll out updated certificates
  • Include checking external Business partners
• Investigate any other changes in MQ configuration
• Check changes to your TLS keystore in APAR UT16295.

How to check a certificate

• /opt/mqm/java/jre64/jre/bin/ikeycmd -cert -details -db key.kdb -label …
• A password is required to access the source key database.
• Please enter a password:
• Label: CLIENT
• Key Size: 2048
• Serial Number: ….
• Issued by: CN=colinpaiceCA, O=Stromness Software Solutions, ST=Orkney, C=GB ? Check this is still valid
• Subject: CN=colinpaice, C=GB
• Valid: From: Thursday, 17 January 2019 18:22:45 o’clock GMT To: Sunday, 31 May 2020 19:22:45 o’clock BST ? Check ‘to’ date
• Signature Algorithm: SHA256withRSA (1.2.840.113549.1.1.11) ? I think this needs to be SHA256withRSA
• Trust Status: enabled

Implement certificate change

• This can be done at any time before migrating a queue manager

Investigate exits

• Find which exits are being used
  • DIS CHL(*)… grep for EXIT
  • dis qmgr grep for exit
• Queue manager and clients
• /var/mqm/qmgrs/QMNAME/qm.ini, channel definition (grep for EXIT)
• Check exits at the correct level on all queue managers and clients. (change date,size)
• May need emails to business partner.
• Do exits need to be converted from 31 bit to 64 bit?
• Locate exit source
• Review source
• Control blocks may be bigger
• May have to support new functions, eg shared connections
• Is function still needed?
• Document exit usage

Implement exit changes

• Recompile all exits and deploy to all platforms before you do any migration work – check no problems
• Change and test exits
• Need to change build tools to allow builds with new levels of header files etc, and roll out to selected queue managers
• Should work on old and new releases
• May need a V9.1 MQ to test exits on before migration
• Can be deployed before MQ Migration ? Or do you have requirements for specific levels of exits.
• Create documentation for exits

Investigate applications

• External business partners as well as internal
• Need to get named contact for each application
• Check level of MQ client code
• Check TLS options
• Identify where connection info is stored (AMQCHLTAB)
• What co-req products need to be updated
  • Web servers
• Is there test suite which includes error paths etc.
• Identify build and deploy tools
• Need capability to compile application using newer MQ header files, and deploy to one MQ

Implement application changes

• Need to have change freeze during migration
• Build project plan
• Duration for testing
• Which systems to be used for testing
• Create process to update MQ client code
• Make sure there is process to roll out changes in future
• Need to allow buffer

Application recompile

• Recompile programs using existing libraries and jar files -to make sure every think works before you migrate
• Deploy and test
• Change deploy process to use new versions of libraries
• Recompile using newer versions of libraries
• Deploy and test
  • Any problems found need to be validated at previous levels, or have conditional statements around it
• Once all queue managers upgraded
  • Comment out code for compiling with previous libraries
  • To prevent accidents
  • In case of problems in production (before migration) needing a fix.

Investigate queue managers

• Does the hardware need to be upgraded?
• Are there any coreqs – eg multi instance or HA environments?
• Any co-reqs eg upgrade web server, database?
• Does the Operating System need to be upgraded
  • For example MQ now 64 bit. Early versions were 31 bit
  • Newer versions of Java
• Identify which applications run on this queue manager
  • Need plan for each application
• Identify pre-reqs
  • TLS
  • Exits

Plan how you are going to update the Client Channel Definiton Table

• If you migrate a queue manager, then its CCDT will be migrated to the newer level.
• Clients cannot use a CCDT from a higher level queue manager.
• If you migrate your clients to the latest level you will have no problems with the CCDT
• If you migrate the CCDT owner queue manager first, you need to be careful about copying the CCDT to other machines, to prevent a mismatch.

Plan queue managers

• Plan software and hardware upgrades
• Identify order of queue manager migration
  • Test, pre-prod, production
  • Full repositories then partial
    • Consider setting up new QM just for full repository?
  • Do one server, test applications, do other servers
  • Need to worry about multi instance and HA queue. These need to be coordinated and done at the same time.
• Check license for MQ
• May need to migrate queue manager multiple times
  • from MQ V5.3 to V7.x
  • from 7.x to V9.0
  • from 9.0 to 9.1
• Clients first/later

Automation

• Need to set up automation for new messages and new events

Backups etc

• Make sure you have back up your queue managers (and other tools such as build configuration files before you make any changes).

Do the migration

• Follow the MQ knowledge centre.