Benchmark Results - Problem U3 - Solver Presets
I. Introduction
We present results of the different built-in solver presets on problem U3, size=100. We run each preset for increasing durations and evaluate final constraint & diversity score of the solution. Each run is performed with a different seed, so also the influence of the seed is evaluated.
As the different seeds cause some randomness in the results, we estimate q10-q90 uncertainty bounds, by performing quantile regression through the data points of each preset using a cubic spline with monotonicity constraints.
The resulting uncertainty bounds give an idea of the result that can be expected by e.g. taking the best result out of 10 runs (with different seeds) as this is expected to lie around ~q90. Uncertainty bounds are only estimated and shown for the relevant metric (constraint score if the problem is constrained and infeasible; diversity score otherwise)
II. Results
A. Figures
B. Tables
Diversity Score vs Total Iterations
(estimated q10...q90 ranges)
| Total Iterations | RANDOM |
GUIDED |
SMART |
THOROUGH |
|---|---|---|---|---|
| 1_000 | - | - | 3(0.60...1.18) | 3(1.51...2.10) |
| 2_000 | - | 29.7...30.3 | 32.(043...391) | 32.(744...952) |
| 5_000 | 3(0.94...1.18) | 31.(520...864) | 33.(033...116) | 33.1(389...916) |
| 10_000 | 31.(797...952) | 32.(487...639) | 33.(171...207) | 33.(179...215) |
| 20_000 | 32.(363...459) | 32.(896...963) | 33.2(042...283) | 33.(187...221) |
| 50_000 | 32.8(236...735) | 33.(079...115) | 33.2(166...362) | 33.(193...227) |
| 100_000 | 33.0(075...410) | 33.1(305...625) | 33.2(250...416) | 33.2(009...326) |
| 200_000 | 33.1(033...301) | 33.1(596...922) | 33.2(312...456) | - |
| 500_000 | 33.1(592...857) | 33.(184...218) | 33.2(342...480) | - |
| 1_000_000 | 33.(174...204) | 33.(198...231) | 33.2(401...516) | - |
| 2_000_000 | 33.(187...219) | 33.2(123...413) | - | - |
| 5_000_000 | 33.2(087...374) | 33.2(304...517) | - | - |
| 10_000_000 | 33.2(171...454) | 33.2(419...571) | - | - |
| 20_000_000 | 33.2(206...502) | - | - | - |
Diversity Score vs Total Time
(estimated q10...q90 ranges)
| Total Time | RANDOM |
GUIDED |
SMART |
THOROUGH |
|---|---|---|---|---|
| 10.0s | 32.(405...547) | 32.(204...513) | 3(1.69...2.13) | 3(0.85...1.41) |
| 15.0s | 32.7(025...920) | 32.(680...844) | 32.(471...721) | 3(1.56...2.03) |
| 30.0s | 3(2.98...3.02) | 33.0(013...690) | 33.(020...108) | 32.(542...792) |
| 45.0s | 33.0(526...886) | 33.(070...118) | 33.1(150...708) | 3(2.87...3.02) |
| 1m0s | 33.(089...121) | 33.(099...141) | 33.1(504...936) | 33.(007...106) |
| 2m0s | 33.1(419...707) | 33.1(434...775) | 33.(193...220) | 33.1(360...871) |
| 5m0s | 33.(171...202) | 33.(175...208) | 33.2(119...320) | 33.(172...210) |
| 10m0s | 33.(185...218) | 33.(190...223) | 33.2(193...373) | 33.(177...213) |
| 15m0s | 33.(196...227) | 33.(198...230) | 33.2(241...409) | 33.(182...216) |
| 30m0s | 33.2(113...396) | 33.2(129...411) | 33.2(303...459) | 33.(190...221) |
| 45m0s | 33.2(161...442) | 33.2(215...464) | 33.2(319...476) | 33.(193...223) |
| 1h0m | 33.2(182...466) | 33.2(273...496) | 33.2(326...486) | 33.(196...225) |
| 1h15m | 33.2(194...482) | 33.2(315...518) | 33.2(331...494) | 33.(198...227) |
| 1h30m | 33.2(202...494) | 33.2(348...534) | 33.2(337...500) | 33.2(007...284) |
| 1h45m | 33.2(207...503) | 33.2(374...547) | 33.2(343...506) | 33.2(030...298) |
| 2h0m | 33.2(212...510) | 33.2(395...557) | 33.2(349...511) | 33.2(051...311) |
| 2h15m | 33.2(215...517) | 33.2(414...565) | 33.2(354...516) | 33.2(070...322) |
| 2h30m | 33.2(219...522) | 33.2(430...573) | 33.2(360...521) | 33.2(088...332) |
| 2h45m | 33.2(221...526) | 33.2(444...579) | 33.2(365...525) | 33.2(104...341) |
| 3h0m | 33.2(224...530) | 33.2(456...584) | 33.2(370...528) | 33.2(119...349) |
| 3h15m | 33.2(226...534) | 33.2(467...589) | 33.2(375...531) | 33.2(133...356) |
| 3h30m | 33.2(228...537) | 33.2(478...593) | 33.2(380...534) | 33.2(146...363) |
| 3h45m | 33.2(229...540) | 33.2(487...597) | 33.2(385...537) | 33.2(157...369) |
| 4h0m | 33.2(231...543) | 33.2(495...600) | 33.2(389...540) | 33.2(168...374) |