(Thanks to Michael Trick for the tip!)
Okay, so what is “better” anyway? I get the sense that for many operations research insiders, “better” is another word for “faster”, but that is wrong, wrong, wrong. “Better” means different things to different people. For example:
- More accurate.
- Less prone to failure.
- Easier to use by a broader set of people.
- Faster to develop a solution.
- Easier to integrate with other systems.
- Better supported.
- Easier to customize and modify.
- Easier to share.
- Uses less memory.
- Unencumbered by intellectual property concerns.
as well as…
“Better” is a multiobjective problem: most of us actually want many of the things on the list. How we weight the various factors depends in part on what the software is being used for:
- For academic research,
- For rapid prototyping,
- To create a model for a consulting engagement,
- For a production system.
Some of these factors can be measured (and are, thanks to the tireless efforts of Hans Mittelmann and others) while others are more subjective. Even if we are focusing exclusively on “faster”, the picture remains complicated. In a production system what matters is how quickly users get the results they want. So we care about not only the time that the solver takes, but also:
- How long it takes to retrieve the data and assemble the model to be solved,
- The predictability of response time over different user requests,
- How the solver performs in the face of many simultaneous requests.
Solver runtime differences of 5-10% don’t matter that much, generally speaking. I like to categorize how long an operation takes in real-world terms (I stole this idea from somebody else, but I don’t remember who):
- instantaneous (subsecond)
- the time it takes to check espn.com and/or twitter (5-10 seconds)
- get coffee (a few minutes)
- have lunch (30 minutes or so)
- a weekend
It’s usually not worth the effort making an engineering decision based solely on performance if you can’t move to a different bucket. Otherwise you probably have better things to do.