Non functional requirements: metrics and checking your product is performing within spec

This blog post is part of a series on non functional requirements, and how they take most of the effort.

The scenario

You want a third party to implement an application package to allow people to buy and sell widgets from their phone. Once the package has been developed, they will hand it over to you to sell, support, maintain and upgrade and you will be responsible for it,

At the back-end is a web server.

Requirements you have been given.

• We expect this application package to be used by all the major banks in the world.
• For the UK we expect the number of people who have an account to be about 10 million people
• We expect about 1 million trades a day.

See start here for additional topics.

Why measure?

Your management team want this product to be a success. How do you know if you are achieving required performance, and how do you avoid a twitter storm of people complaining that your product is slow? Some businesses, like those buying and selling on the stock exchange, get fined if they do not meet externally specified performance targets.

Three areas you may be interested in:

1. Will there be a problem next week, next month? Can we see trends in the existing data, such as CPU usage is going to max out, or the end user response time at peak time will be out of spec.
2. Is there a performance problem now? If so, can you identify the problem area?
3. Last month when you had a performance problem – have you kept enough data to identify the problem. The problem may not have shown up in your real time monitoring.

What do you want to measure?

The people specifying the product have said the average response end user response time needs to be under 50 milliseconds. This needs some clarification

• You cannot control or influence the phone network, so you do not want to be blamed for a slow network. A better metric might be “time in your system” must be less than 40 milliseconds
  • This means you need to capture the entry and exit time, and report the transaction duration on exit.
  • You need a dash board to display this information.
• The “average response time”: if you get excellent response time at night, when no one is using the system, and poor response time during the lunch break – the average of these response times may be under 40 milliseconds. This way of reporting the data is useless. You might want to report the information on a granularity of 1 minute.
• You might want to display a chart of the count of any transactions taking over 40 milliseconds, and display the maximum count value in the interval. The should always be zero.
• You might want to report the average response time per hour, and report this over the last 3 months (or the last 13 months). This should allow you to see if there are any trends in response time, and gives you time to take action before it is a problem.
• Your application could record the time spent doing database work, and if this time is over 20 milliseconds, report and plot this.
• If you have any exceptions, such as long response time, then you could send an event to a thread which captures these, and writes a data base record. You you not want to have a database write after your transaction has completed, just before it returns to the end user.

How to capture data

You can have the transaction collect the per transaction response time within your application server. You could also have a mobile phone that does a transaction every minute – so you get an early notification if there is a network problem

How to look at the data

You need to have a profile of expected/historical data. For example during the data the response time is 40 ms. Overnight it is 10 ms. If you start getting a response time of 20 ms overnight, this should be investigated; it is still below 40 ms, but it does not match the profile, and is a good indicator of a problem.

Different days have different profiles, perhaps Monday is always a peak day, and Christmas day is a quiet day.