How fast is your software? The simpler the software setup, the easier to answer this question. The more complex the software, the more the answer will “it depends”. But just peek at F1-racing – the answer will depend on the driver and the track.

This article focuses on the foundations of solid benchmarking, so it helps you to decide which discussions to have with your team. It is not the full book.

There will be multiple blog posts coming in this series, which will be linked at the end of the post when published.

The questions to ask

Even when it depends on various variables, answers do can be given. These answers are best be described as ‘insights’ and this blog is about that.

First the commercial message, so we can focus on the main subject. As benchmark-design is not always obvious, we help customers to set up a system that plugs into a continuous integration system and gives continuous insights. More about that in an upcoming blog.

We see benchmarking as providing insights in contrast with the stopwatch-number. Going back to F1 – being second in the race, means the team wants to probably know these:

• What elements build up the race? From weather conditions to corners, and from other cars on the track to driver-responses
• How can each of these elements be quantified?
• How can each of these elements be measured for both own cars and other cars?
• And as you guessed from the high-level result, the stopwatch: how much speedup is required in total and per round?

This translates quite well to software. While we speak about “doing benchmarks”, the better name would be “getting insights” – benchmarking is just the tool to get these. Most used are:

• Functional, measured in time
• Data flow, measured in Bytes/s
• High level, measured in time and Bytes/s

Foundation 1: Profiling

Profiling is what many developers do, with different levels of expertise:

• Not knowing where to start
• Seeking very advanced tools
• Finding a few (types of) problematic areas
• Actually finding all the problematic areas, but not fast
• Quickly finding the problematic areas

So it drills down to understanding what types of problems there are, how much one understands the mechanics behind tools like GDB, and experience with such tools.

Foundation 2: Automation

Developers by training and mindset already think of automation by default. Still automating to support automation is not always a logical step. Think of liking to make algorithms that solve labyrinths and writing code that measures efficiency of the algorithm – for various reasons the automation-effort of the second part is more often done manually.

Foundation 3: Balancing of effort

This is a part of the art of problem solving, of which we briefly discussed one aspect recently. How to make something that is neither under-engineering nor over-engineering? The first steps of problem solving are all around clearly defining the problem, where the later steps make sure a possible solution has the original problem in mind.

Problems are easily found, by measuring the summed time that profiling and benchmarking take as part of the whole project. Compare that to the priorities of the original problems that need to be solved.

In conclusion

Deciding that benchmarking is important for a project, is half the work. By having the right discussions using the points in this blog article, you can easily get to a good benchmark-foundations.

Benchmarking is initially obvious, but then gets a specialization quickly. We have thought out how to make it less steep for longer, which we will share later.

Want to know more? Watch this blog for follow-ups. Want to know more right now? Then get in contact.