It was a great day, it was 0830, the sun was shining, the view was amazing, as I was walking back the long way from the shops. There was a good hill en route, it starts nice and steep, then gradually leveled off to flat. I sat on the convenient seat at the top of the hill to admire the view and got my breath back. The shape of the hill reminded me of a graph in a performance report I had seen a few days before; It started steeply at the bottom left of the graph, and levelled off at the top right of the graph and the description was “so we can see it scales”. As I sat there enjoying the sun, I realised that it may scale – but it does not scale very well.
What does it mean to scale?
I remember 40+ years ago sitting in a meeting where the performance team were presenting status. They showed many graphs, one of which showed the transaction rate as they increased the work load. The graph started going up at an angle, nice and straight, and then levelled out until they achieved 100 transactions a second, and they could do no more than this. They explained that this was because of a lock being held for 1/10 of a second while some work was done.
They worked with the developer who realised that they could use a counter instead of a lock, and use the hardware instruction to atomically increase and decrease the counter. As a result the graph was now a straight line. Round of applause etc, well done etc.
The performance guy then went on to say, so in order to find the next bottlenecks we will need a box that is even bigger and even faster!
I also remember this performance guy said his job was to make curved lines straight!
You can hide the truth with charts
People often use graphs to show information, but often tell the wrong story. For example if you plot the total cost vs transaction rate you get a straight line
Great – this looks linear!
However If you plot the cost per transaction vs rate you get
In actual fact there is an increase in the cost per transaction as the rate increases. So good scaling – but not perfect. It is linear – when it should have been flat!
If you want to scare people – change the axis
It is the same data as the previous chart – but now the axis is 20 to 22.5 instead of 0 to 25, so any difference appears much larger.
So depending on what story you want to tell, and if you want to hide the truth, think about what data you present, and the axis you use.
Every one wants a bigger box or two
On the subject of wanting a bigger box…
One of the senior tester’s presented to management about testing strategy. He said you can do testing in one of two ways:
- If we were careful, and picked the right set of results we could show all test working. We could quietly ignore the tests that failed.
- No matter what we did, we could not seriously break it. Even when we deliberately misconfigured it during testing it carried on running – slowly – but it carried on running.
The manager said “hmmm I’m always being asked for graphs to present status so the first would be better, but from a customer perspective the second would be better”.
The tester said “That’s the right answer, we now need two bigger boxes. One to run the workload, the other to drive it. We need to achieve higher throughput than our customers”.
One customer told me that their test box was larger than production, and they needed to prove that they could run at expected peak production throughput + 50% with no major degradations – they needed many very big boxes.