The fast and slow of clients doing MQCONN and the impact of DNS

I was doing some investigation into the rate at which clients could connect and disconnect from MQ.  I found that about every 50-70th request took over 5 seconds!

I used export MQSERVER=SYSTEM.DEF.SVRCONN/TCP/’LOCALHOST(1414)’  and a client program which did MQCONN, MQDISC and looped.  It timed each call.

I was running on Ubuntu 18.04.

I did a bit more digging and found the following in AMQERR01.LOG
06/10/18 16:39:06 – Process(7703.1) User(colinpaice) Program(connc)
                    Host(colinpaice) Installation(Installation1)
                    VRMF(9.1.0.0)
                    Time(2018-10-06T15:39:06.124Z)
                    ArithInsert1(5)
                    CommentInsert1(LOCALHOST)
                    CommentInsert2(getaddrinfo)
                  
AMQ9788W: Slow DNS lookup for address ‘LOCALHOST’.EXPLANATION:
An attempt to resolve address ‘LOCALHOST’ using the ‘getaddrinfo’ function call
took 5 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.

If the address was an IP address then the slow operation was a reverse DNS
lookup. Some DNS configurations are not capable of reverse DNS lookups and some
IP addresses have no valid reverse DNS entries. If the problem persists,
consider disabling reverse DNS lookups until the issue with the DNS can be
resolved.

There are some discussions about this on various fora, and there are discussions about the use of IP V6.

This failed consistently for a couple of days, but when I was sitting in an airport waiting for my plane, I turned MQ trace on – and the problem did not occur.  I turned trace off, and I have not seen the problem since, so it must have been related to sitting in a hotel room.

If I specify export MQSERVER=SYSTEM.DEF.SVRCONN/TCP/’LOCALHOST(1414)’ I get

MQCONN Mean:, 11.08,Max:, 15.51,Min:, 8.92,ms rate, 90.24 a second

If I specify export MQSERVER=SYSTEM.DEF.SVRCONN/TCP/’127.0.0.1(1414)’ I get

MQCONN Mean:, 1.31,Max:, 2.60,Min:, 1.09,ms rate,766.23 a second.

Lessons learned

  1. Use dotted IP addresses to avoid the DNS lookup time
  2. Look at the MQ error logs!
  3. There may be things going on outside of your control which should have no impact on you – but do have an impact ( eg my hotel room)
  4. Morag suggested you can disable revdns, but see What is the impact of REVDNS(DISABLE) and use of host names?

Does this matter?

If your clients are well behaved and connect once a week this is not a problem.  If your clients are connecting for every message, this will be a killer due to the extra CPU, and the elongated time.

CHLAUTH may be using DNS or reverse ( to convert dotted address into a name).   This will add to the extended duration.

Many “frameworks” designed to make it easier to use MQ have the settings to do an MQCONN for every message – so check your configurations – and tune it so it connects infrequently!

You may get better performance by having 10 threads connected all of the time, rather than have the framework expand and shrink the number of connections.

Notes on DNS

On Ubuntu I used the dig command to measure the response time to the DNS. For example

dig -4 -u LOCALHOST any

this gave me a lot of output – one line of which was

;; Query time: 354 usec

I then put the -4 -u LOCALHOST any into a file and repeated it for 100 lines.

I then used

dig -f dns.txt |grep Query |sort -n -k4 |less

This ran the commands in dns.txt, extracted the line with the response time, sorted it by the 4th field (ascending time), and displayed it

The majority of the requests were under 100 microseconds but one took 3829 microseconds

Another command

I found

systemd-resolve --status

an interesting command, and gave more info about the DNS

 

 

 

One thought on “The fast and slow of clients doing MQCONN and the impact of DNS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s