Combinatorial explosion of branches

[pozorvlak]

I've been running benchmarks again. The basic workflow is

1. Create some number of directories containing the benchmark suites I want to run.
2. Tweak the Makefiles so benchmarks are compiled and run with the compilers, simulators, libraries, flags, etc, that I care about.
3. Optionally tweak the source code to (for instance) change the number of

Comments

[fanf]

Is version control really the right tool? Shouldn't there be just one version which is parametrized according to the experiments you want to run? Then you only need to fix each bug once. It should also make the setup more easily reproducible so you could hand it to a grad student to run the numbers on a compute cluster or something. I often find that turning a repetitive problem into a programming problem makes it more rewarding.

[pozorvlak]

Hmmmm, you could be right. In EEMBC 1.1, the iteration counts are specified in each benchmark using C #defines, and I altered these by editing the code (well, with perl -i, but the principle is the same). But I guess I could have changed
#define ITERATIONS 50000 to
#define ITERATIONS (50000 * MULTIPLIER) and then added -DMULTIPLIER=100 to CFLAGS.

you could hand it to a grad student to run the numbers on a compute cluster or something.

In this particular case, we cared about wallclock time, and I'd be worried about introducing variability if we ran it on a cluster - perhaps unreasonably, and I suppose we could do multiple runs and average. Actually, we should probably be doing that anyway. But yeah, good idea.