criterion performance measurements
overview
want to understand this report?
readPng/Friday
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.592166238708975e-2 | 4.628361955056407e-2 | 4.700951031194007e-2 |
| Standard deviation | 4.7119822172436357e-4 | 9.962740446722706e-4 | 1.5798497023495276e-3 |
Outlying measurements have slight (6.632653061224478e-2%) effect on estimated standard deviation.
readPng/Yarr
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6025523530974473e-2 | 1.626891263787119e-2 | 1.6578346227111254e-2 |
| Standard deviation | 4.3538076160130077e-4 | 7.898146672040198e-4 | 1.073998816025749e-3 |
Outlying measurements have moderate (0.19634405687482653%) effect on estimated standard deviation.
readPng/Repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6274320185283776e-2 | 1.676667313185711e-2 | 1.7346311513184526e-2 |
| Standard deviation | 1.069331746763325e-3 | 1.4091634531377061e-3 | 2.0997752706419393e-3 |
Outlying measurements have moderate (0.4006652897906183%) effect on estimated standard deviation.
readPng/OpenCV
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4671628521328218e-2 | 1.5112458818875093e-2 | 1.574152043576227e-2 |
| Standard deviation | 1.098158420810931e-3 | 1.3377873080713304e-3 | 1.6513493138621716e-3 |
Outlying measurements have moderate (0.42455878434604705%) effect on estimated standard deviation.
threshold/Friday
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1720337149880751 | 0.19482774020756632 | 0.22602944879826362 |
| Standard deviation | 1.9917854563711705e-2 | 3.598427272158489e-2 | 4.734295008222239e-2 |
Outlying measurements have moderate (0.48091531871752463%) effect on estimated standard deviation.
threshold/UnmHip
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5466276201194556 | 0.5790097892533407 | 0.5998098962405846 |
| Standard deviation | 0.0 | 3.126725839275371e-2 | 3.6026842104775005e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
threshold/Yarr
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6523015013561744e-2 | 1.757968285228895e-2 | 1.889456038468252e-2 |
| Standard deviation | 1.9750257275887796e-3 | 2.890689883258136e-3 | 4.145467451633809e-3 |
Outlying measurements have severe (0.7254041366506138%) effect on estimated standard deviation.
threshold/Repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.340206020070236e-2 | 5.216098528623745e-2 | 7.561980315930487e-2 |
| Standard deviation | 2.493683300234885e-3 | 2.7951033142379806e-2 | 4.855541296015526e-2 |
Outlying measurements have severe (0.9195103731862186%) effect on estimated standard deviation.
threshold/OpenCV
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.63084448612534e-4 | 3.796219428714806e-4 | 3.9713929965271076e-4 |
| Standard deviation | 4.695908657966157e-5 | 5.6918753533267894e-5 | 6.943302226991386e-5 |
Outlying measurements have severe (0.8948971346511211%) effect on estimated standard deviation.
mean/Friday
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.907766683264005e-2 | 7.144388485036172e-2 | 7.477324797053388e-2 |
| Standard deviation | 3.637952760466181e-3 | 5.10695662776518e-3 | 7.505101698437368e-3 |
Outlying measurements have moderate (0.17262834612733957%) effect on estimated standard deviation.
mean/Yarr
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.545918547589593e-2 | 7.751686860149257e-2 | 7.89699718232568e-2 |
| Standard deviation | 2.138230755088676e-3 | 2.7505968551194537e-3 | 3.8982862137044616e-3 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
mean/Repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.16356433859500794 | 0.16973096176762967 | 0.17545753292070523 |
| Standard deviation | 4.777619319062829e-3 | 8.38001321409872e-3 | 1.0504886893983151e-2 |
Outlying measurements have moderate (0.12416932281555382%) effect on estimated standard deviation.
mean/OpenCV
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8656938994912843e-3 | 2.8939744034574264e-3 | 2.911414910694208e-3 |
| Standard deviation | 5.003687126111957e-5 | 7.301585346575473e-5 | 9.71384244588492e-5 |
Outlying measurements have moderate (0.1109856354320883%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.