Update: Gurobi vs. CPLEX
First benchmark results for the new (and still unreleased) solver of Gurobi (the new company of Gu, Rothberg, and Bixby) have been published yesterday by Hans Mittelmann and compared to CPLEX 11.2. The results look pretty decent and it looks like the gurobi code is very much comparable to the one of CPLEX in terms of performance.
From the Intechne Blog >>Less is more<<:
- Gurobi solves MIPs faster on single-processor machines: It beats or equals CPLEX’s performance 55% of the time. (A third solver – MOSEK – is also compared. It wins or ties for first 5% of the time.)
- Gurobi parallelizes robustly: Having used it in a number of projects over the years, I can attest that CPLEX’s parallel branch and bound is a very solid piece of code. In this one-on-one comparison on a 4-processor machine, Gurobi wins or ties CPLEX 58% of the time.
- Aggregate solution times are less on Gurobi: Considering only problems on which at least one integer solution was found, on a single-CPU system Gurobi runs through the test-set in 25% less time than CPLEX, as measured using the geometric mean of times. On 4-CPU systems, the improvement is still considerable – 15%.
- Gurobi is good at finding integer feasible solutions: CPLEX fails to identify a single integer feasible solution on 2 instances in either mode. Gurobi fails on one instance in single-threaded mode. It finds at least one integer solution on all test-set problems when using four processors.
- CPLEX takes better advantage of parallelization: CPLEX’s speed-up in going from one to four processors is 40%, whereas Gurobi only manages 30%.
Also check out the post on Michael Trick’s Blog.
UPDATE: A trial version of the Gurobi solver is available on http://www.solver.com/gurobi. For details, see the comment of Daniel H. Fylstra in the comment section of this post.
UPDATE 05/20/2009: The standalone version of Gurobi is available now. (see also here)
Sebastian Pokutta's Blog 
Although Gurobi Optimization plans to offer the Gurobi Solver directly (with its own APIs and tool set) starting in April, it’s available in a full commercial release this month (February) as embedded in several modeling systems (AIMMS, Frontline, GAMS, MPL). We have it available for download right now (15 day free trial with no other restrictions) at http://www.solver.com/gurobi. You can run the Mittleman benchmarks yourself, or solve your own models in MPS, LP and OSIL format using a little program GurobiEval that we include. You can create models from scratch, and solve them with Gurobi, either in Excel with our modeling system, Risk Solver Platform, or in C, C++, C#, VB6, VB.NET, Java or MATLAB using our Solver Platform SDK.
About Gurobi’s performance: We’ve been running it for months, and some of our tough customer models form part of Gurobi’s internal test set. The benchmark results on the Mittleman test problems are clearly very good for Gurobi (take the geometric mean of the ratio of solve times across all the problems for a fair comparison). But the Mittleman results for 4-core processors tend to understate Gurobi’s typical performance on the customer models we’ve seen. I understand from Gurobi that this is due to many factors, including the fact that Gurobi can solve a number of these problems at the root node via cuts without ever branching — and of course its parallelism comes fully into play only when it does start branching. Our experience with our own customer models suggests that Gurobi’s speed advantage vs. CPLEX and other codes is often greater on multi-core processors than it is on single-core processors.
Daniel H. Fylstra
February 8, 2009 at 10:45 pm
Daniel,
you state that the performed tests might understate the performance for multi-core processors. Do you have some numbers how the two solvers compare in the tests you performed?
I also included the link to the trial version in the post, so that it is not somewhere lost in the comment section. thx!
Sebastian
February 9, 2009 at 10:00 am
[…] The recent discussions about the new Gurobi solver and its performance compared to CPLEX ([1], [2]) as well as the discussion about using open source solvers ([3]) made me wonder how many people […]
Poll: Which MIP solver do you use? « Sebastian Pokutta’s Blog
February 9, 2009 at 3:59 pm
If you take the Mittleman benchmark times, compute CPLEX1/GUROBI1 and CPLEX4/GUROBI4 and the geometric means of these ratios, you’ll see that Gurobi’s outperformance is better in the 1-core case than in the 4-core case. While correct for the Mittleman test set, we believe this is unusual — we’ve observed on other test sets that the opposite is true: Gurobi’s outperformance is better in the 4-core case.
As one example, we have a selection of 50 of the MIPLIB problems (some old, some new) that seem to be very hard to parallelize: On this test set the geomean of XPRESS4/XPRESS1 is just 1.11, and the ratio is <= 1.0 (i.e. same or worse performance with 4 cores) on 25 of the 50 problems. For GUROBI4/GUROBI1 the geomean is 1.255 and the ratio is <= 1.0 on just 8 of 50 problems. Gurobi outperforms the other Solver in both the 1-core and 4-core case, as it does on the Mittleman problems, but the outperformance is roughly twice as good on 4-cores than on 1-core on this test set.
Daniel H. Fylstra
February 9, 2009 at 7:40 pm
Three more brief comments:
— The 50 MIPLIB problems in my previous post were selected as hard to parallelize. I expect many users will experience a better speedup than 1.255 with Gurobi on 4-cores. 1.5 is difficult to achieve, but Gurobi often achieves it.
— For people who aren’t into parallel computing, please note that 1.5x is really very good — if you’re wondering why you can’t get 4.0x, read http://en.wikipedia.org/wiki/Amdahl%27s_law.
— MIP performance is highly problem-dependent, and it seems this is even more true for multi-core computing than for single-core. Most people care about how a Solver performs on THEIR OWN model(s). That’s why we’ve made the free Gurobi eval available for download at http://www.solver.com/gurobi.
Daniel H. Fylstra
February 9, 2009 at 8:32 pm
Now, it is possible to download a trial version of CPLEX 12.1:
http://www14.software.ibm.com/webapp/download/preconfig.jsp?id=2009-11-03+08%3A14%3A19.245593R&S_TACT=104CBW71&S_CMP=
Daniel Severin
November 8, 2009 at 2:49 am