How fast is your software? The simpler the software setup, the easier to answer this question. The more complex the software, the more the answer will “it depends”. But just peek at F1-racing – the answer will depend on the driver and the track.
This article focuses on the foundations of solid benchmarking, so it helps you to decide which discussions to have with your team. It is not the full book.
There will be multiple blog posts coming in this series, which will be linked at the end of the post when published.
The questions to ask
Even when it depends on various variables, answers do can be given. These answers are best be described as ‘insights’ and this blog is about that.
First the commercial message, so we can focus on the main subject. As benchmark-design is not always obvious, we help customers to set up a system that plugs into a continuous integration system and gives continuous insights. More about that in an upcoming blog.
We see benchmarking as providing insights in contrast with the stopwatch-number. Going back to F1 – being second in the race, means the team wants to probably know these:
- What elements build up the race? From weather conditions to corners, and from other cars on the track to driver-responses
- How can each of these elements be quantified?
- How can each of these elements be measured for both own cars and other cars?
- And as you guessed from the high-level result, the stopwatch: how much speedup is required in total and per round?
