criterion performance measurements
overview
want to understand this report?
| readPng | |
| threshold | |
| mean |
readPng/Friday
38 40 42 44 46
mean |
5 8 10 13 15 2.5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 38.2 ms | 41.6 ms | 45.1 ms |
| R² goodness-of-fit | 0.966 | 0.981 | 0.994 |
| Mean execution time | 40.3 ms | 41.6 ms | 43.0 ms |
| Standard deviation | 2.30 ms | 2.73 ms | 3.39 ms |
Outlying measurements have moderate (19.0%) effect on estimated standard deviation.
readPng/Yarr
14 15 16 13.5 14.5 15.5
mean |
10 15 20 25 5 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.5 ms | 15.2 ms | 15.9 ms |
| R² goodness-of-fit | 0.976 | 0.986 | 0.993 |
| Mean execution time | 14.3 ms | 14.6 ms | 15.0 ms |
| Standard deviation | 862 μs | 985 μs | 1.08 ms |
Outlying measurements have moderate (30.6%) effect on estimated standard deviation.
readPng/Repa
14 15 16 17 18 13.5 14.5 15.5 16.5 17.5
mean |
10 15 20 25 5 iters 200 300 400 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 13.4 ms | 13.6 ms | 13.8 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 14.0 ms | 14.3 ms | 15.1 ms |
| Standard deviation | 648 μs | 1.17 ms | 1.76 ms |
Outlying measurements have moderate (40.7%) effect on estimated standard deviation.
readPng/OpenCV
13.5 13.5 13.5 13.5 13.6 13.6 13.6
mean |
10 15 20 25 5 iters 100 150 200 250 300 350 400 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 13.5 ms | 13.5 ms | 13.6 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 13.5 ms | 13.6 ms | 13.6 ms |
| Standard deviation | 25.7 μs | 39.1 μs | 51.3 μs |
Outlying measurements have slight (3.7%) effect on estimated standard deviation.
threshold/Friday
84 85 86 87 83.5 84.5 85.5 86.5
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 82.8 ms | 83.7 ms | 84.3 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 84.1 ms | 84.7 ms | 85.7 ms |
| Standard deviation | 343 μs | 1.18 ms | 1.58 ms |
Outlying measurements have slight (9.0%) effect on estimated standard deviation.
threshold/UnmHip
720 740 760 780 800
mean |
1 2 2 3 3 4 4 0.5 iters 2 3 0 s 500 ms 1 s 1.5 2.5 3.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 606 ms | 780 ms | 1.10 s |
| R² goodness-of-fit | 0.969 | 0.979 | 1.000 |
| Mean execution time | 719 ms | 751 ms | 809 ms |
| Standard deviation | 0 s | 50.7 ms | 55.4 ms |
Outlying measurements have moderate (19.6%) effect on estimated standard deviation.
threshold/Yarr
15 20 25 17.5 22.5 27.5
mean |
10 15 20 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 20.3 ms | 22.6 ms | 24.5 ms |
| R² goodness-of-fit | 0.937 | 0.966 | 0.987 |
| Mean execution time | 20.9 ms | 22.3 ms | 23.2 ms |
| Standard deviation | 1.93 ms | 2.89 ms | 4.44 ms |
Outlying measurements have severe (58.3%) effect on estimated standard deviation.
threshold/Repa
75 80 85 90 77.5 82.5 87.5
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 84.9 ms | 87.6 ms | 92.5 ms |
| R² goodness-of-fit | 0.976 | 0.993 | 0.999 |
| Mean execution time | 80.5 ms | 83.3 ms | 86.4 ms |
| Standard deviation | 3.23 ms | 4.71 ms | 6.44 ms |
Outlying measurements have moderate (18.0%) effect on estimated standard deviation.
threshold/OpenCV
250 275 300 325 350 375 400
mean |
200 300 400 500 600 700 800 100 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 259 μs | 262 μs | 264 μs |
| R² goodness-of-fit | 0.993 | 0.996 | 0.998 |
| Mean execution time | 281 μs | 290 μs | 302 μs |
| Standard deviation | 33.5 μs | 39.7 μs | 49.0 μs |
Outlying measurements have severe (87.9%) effect on estimated standard deviation.
mean/Friday
2.44 2.46 2.48 2.5 2.52 2.54
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 12 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.34 s | 2.54 s | 2.82 s |
| R² goodness-of-fit | 0.998 | 0.998 | 1.000 |
| Mean execution time | 2.44 s | 2.48 s | 2.55 s |
| Standard deviation | 0 s | 57.4 ms | 61.2 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
mean/UnmHip
588 590 592 594 596
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 563 ms | 591 ms | 612 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 588 ms | 593 ms | 595 ms |
| Standard deviation | 0 s | 4.43 ms | 4.67 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
mean/Yarr
75 80 85 90 95 100 105 110 115
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 80.8 ms | 101 ms | 122 ms |
| R² goodness-of-fit | 0.891 | 0.944 | 0.996 |
| Mean execution time | 80.5 ms | 85.4 ms | 95.5 ms |
| Standard deviation | 4.30 ms | 11.4 ms | 17.8 ms |
Outlying measurements have moderate (47.8%) effect on estimated standard deviation.
mean/Repa
100 150 200 250 300
mean |
2 3 4 5 6 7 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | -30.1 ms | 91.1 ms | 126 ms |
| R² goodness-of-fit | 0.010 | 0.517 | 0.994 |
| Mean execution time | 127 ms | 160 ms | 253 ms |
| Standard deviation | 5.78 ms | 73.5 ms | 101 ms |
Outlying measurements have severe (84.9%) effect on estimated standard deviation.
mean/OpenCV
3 2.7 2.80 2.90 3.1 3.2
mean |
20 30 40 50 60 70 80 90 10 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.75 ms | 2.83 ms | 2.91 ms |
| R² goodness-of-fit | 0.996 | 0.997 | 0.999 |
| Mean execution time | 2.89 ms | 2.92 ms | 2.95 ms |
| Standard deviation | 75.1 μs | 93.5 μs | 127 μs |
Outlying measurements have moderate (16.7%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.