Be careful when timing one-liners

Be careful when differences between your timing tests are one-liners, and these one-liners are the entirety of your timing code. This applies to small chunks of code too. I’m actually extending a remark I said while discussing multiplications, additions and bit shifts:

This allows you to increase the percentage of identical parts between the tests, and thus highlight the difference in only the calculation method

The small code chunks you’re testing are sensitive to timing, meaning their timing results are likely to fluctuate. Instead of testing the code chunks in isolation, surround them in code used in a typical situation. Why should you do this?

Suppose you’re testing 2 methods. In isolation, method A took 2 seconds. Method B took 5 seconds. Wow! Hands down, method A is faster than method B.

Then when you use method A in production, you find no real speed benefit. Why?

Because in a typical use situation, the surrounding code together with method A takes 60 seconds. A 3 second savings isn’t much. In reality, it doesn’t matter which method you use. Practically speaking, they took the same amount of time.

This was what I meant by increasing the percentage of identical parts.

A physics experiment: timing falling balls

Suppose you want to time how fast a ball falls from a specific height to reach the ground. You start with experiment A (shown in picture), with a low height.

Timing falling balls

Then you try starting from a higher height, say 3 times higher, in experiment B. You take your measurements and do the relevant calculations. Taking into account the different heights, you divide the timings in experiment B by 3 to normalise them with those of experiment A. And you find they differ quite a bit. Why?

Because you forgot to take into account that when you start and stop a timer, your reaction time comes into play.

Your reaction time took a larger percentage in the total time taken in experiment A because the ball hits the ground fairly fast. In experiment B, the time taken to hit the ground is longer, so your reaction time takes a smaller percentage.

So what’s the larger “percentage of identical parts” in experiment B? Air, or more height.*

* assuming wind and air eddies don’t foul up the experiment