criterion performance measurements
overview
want to understand this report?
joinWordsAppend/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8541154091593034e-4 | 2.3148641079350445e-4 | 3.06873227773805e-4 |
| Standard deviation | 1.0421638501235748e-4 | 1.5938250323009574e-4 | 2.067663956624228e-4 |
Outlying measurements have severe (0.9886567057908393%) effect on estimated standard deviation.
joinWordsAppend/2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.062995196299961e-4 | 4.917508584644438e-4 | 6.05956666601653e-4 |
| Standard deviation | 2.1387553929467061e-4 | 2.860565902914296e-4 | 3.523886332139669e-4 |
Outlying measurements have severe (0.9870982486624238%) effect on estimated standard deviation.
joinWordsAppend/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3770437144265254e-3 | 1.5815654128630903e-3 | 1.8747348387472685e-3 |
| Standard deviation | 5.233478187076274e-4 | 7.155328814282908e-4 | 9.503734664909638e-4 |
Outlying measurements have severe (0.9811634734203182%) effect on estimated standard deviation.
joinWordsAppend/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2777328874601845e-3 | 3.6942320803906876e-3 | 4.171765399178679e-3 |
| Standard deviation | 1.0225321695440572e-3 | 1.2420623604403483e-3 | 1.5434431860164915e-3 |
Outlying measurements have severe (0.9505659434130652%) effect on estimated standard deviation.
joinWordsAppend/20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.627589526029553e-3 | 7.2448365801017645e-3 | 8.131089332910868e-3 |
| Standard deviation | 1.6033263030908323e-3 | 1.9971567360313643e-3 | 2.6044787973180977e-3 |
Outlying measurements have severe (0.931852639798944%) effect on estimated standard deviation.
joinWordsAppend/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6481442744410327e-2 | 1.822844224827975e-2 | 2.0417008753349886e-2 |
| Standard deviation | 3.5952659185266612e-3 | 4.553746217605858e-3 | 6.068310627703861e-3 |
Outlying measurements have severe (0.855148064977569%) effect on estimated standard deviation.
joinWordsConcat/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6721405791139462e-4 | 2.1063306181608125e-4 | 2.91551597618293e-4 |
| Standard deviation | 9.502796847255302e-5 | 1.5230868723377388e-4 | 2.0887263549654328e-4 |
Outlying measurements have severe (0.9890505261274687%) effect on estimated standard deviation.
joinWordsConcat/2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3960654414077486e-4 | 5.281588359252714e-4 | 6.478660142464623e-4 |
| Standard deviation | 2.0435034656064792e-4 | 2.8246613829225654e-4 | 3.573050068225403e-4 |
Outlying measurements have severe (0.9864969693786648%) effect on estimated standard deviation.
joinWordsConcat/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1186697424570302e-3 | 1.2860556621104393e-3 | 1.4683126300039364e-3 |
| Standard deviation | 4.2354827601684105e-4 | 5.341687297720566e-4 | 6.807245316353181e-4 |
Outlying measurements have severe (0.982170090448798%) effect on estimated standard deviation.
joinWordsConcat/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.965007584229878e-3 | 3.3953039154413184e-3 | 3.7923785419808074e-3 |
| Standard deviation | 9.749924058696924e-4 | 1.1711235145651397e-3 | 1.5098871755100359e-3 |
Outlying measurements have severe (0.9507116380866095%) effect on estimated standard deviation.
joinWordsConcat/20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.9879087615401965e-3 | 7.46544512465359e-3 | 7.993212683313867e-3 |
| Standard deviation | 1.08524053198974e-3 | 1.4926891423793264e-3 | 1.959750748141908e-3 |
Outlying measurements have severe (0.8350549859779527%) effect on estimated standard deviation.
joinWordsConcat/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.935312163859856e-2 | 2.2146652439323143e-2 | 2.655322993879109e-2 |
| Standard deviation | 4.411634942821272e-3 | 7.6743760626926e-3 | 1.1976790969375177e-2 |
Outlying measurements have severe (0.9379963649958557%) effect on estimated standard deviation.
joinWordsShowS/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7383643613675388e-4 | 2.1586178350800817e-4 | 2.820081002483283e-4 |
| Standard deviation | 9.810888654457167e-5 | 1.4682966164706186e-4 | 1.9022481689393015e-4 |
Outlying measurements have severe (0.9888919500115926%) effect on estimated standard deviation.
joinWordsShowS/2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.925176589528216e-4 | 4.955427866157917e-4 | 6.153953741466061e-4 |
| Standard deviation | 1.989295131750773e-4 | 2.840023215122429e-4 | 3.7580731694082317e-4 |
Outlying measurements have severe (0.9865851454749284%) effect on estimated standard deviation.
joinWordsShowS/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.207413268030774e-3 | 1.4176030610851373e-3 | 1.6506692239401514e-3 |
| Standard deviation | 4.6304538599439756e-4 | 6.651746744804561e-4 | 8.439202514053304e-4 |
Outlying measurements have severe (0.9818884414352225%) effect on estimated standard deviation.
joinWordsShowS/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5236198417701926e-3 | 2.826509357227495e-3 | 3.1179190577065937e-3 |
| Standard deviation | 6.565588056755684e-4 | 8.529125493519427e-4 | 1.2767997973073845e-3 |
Outlying measurements have severe (0.9540999039684857%) effect on estimated standard deviation.
joinWordsShowS/20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.6602136494558475e-3 | 6.21735438324143e-3 | 6.753788829932034e-3 |
| Standard deviation | 1.1631237338788512e-3 | 1.5278796046392005e-3 | 1.985501455686733e-3 |
Outlying measurements have severe (0.9081062784478496%) effect on estimated standard deviation.
joinWordsShowS/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5119276465802706e-2 | 1.6407831550412144e-2 | 1.7316685191679206e-2 |
| Standard deviation | 2.0585823474311097e-3 | 2.633458759127313e-3 | 3.7946393567434866e-3 |
Outlying measurements have severe (0.7126899150615373%) effect on estimated standard deviation.
joinWordsSeq/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.649913159342484e-4 | 4.077919916155328e-4 | 4.712427460695003e-4 |
| Standard deviation | 9.385428533093818e-5 | 1.4130938877691198e-4 | 1.8446614987317896e-4 |
Outlying measurements have severe (0.9737805323843017%) effect on estimated standard deviation.
joinWordsSeq/2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2716635989662621e-3 | 1.3761970586885022e-3 | 1.5362416244297063e-3 |
| Standard deviation | 2.5823378601149007e-4 | 3.571369251976002e-4 | 5.057373343190085e-4 |
Outlying measurements have severe (0.9446787566598775%) effect on estimated standard deviation.
joinWordsSeq/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.985373640031318e-3 | 4.253717925032016e-3 | 4.70244021734022e-3 |
| Standard deviation | 7.376636524441992e-4 | 9.359517182779927e-4 | 1.2297177255026599e-3 |
Outlying measurements have severe (0.8934645493983563%) effect on estimated standard deviation.
joinWordsSeq/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.853955127316526e-3 | 8.326856658403192e-3 | 8.732281760477866e-3 |
| Standard deviation | 9.41488659532557e-4 | 1.0972431374055489e-3 | 1.284198594545544e-3 |
Outlying measurements have severe (0.6853022355832187%) effect on estimated standard deviation.
joinWordsSeq/20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.95351845007505e-2 | 2.057747358699524e-2 | 2.1965384894486767e-2 |
| Standard deviation | 2.100903801481705e-3 | 2.75913915494484e-3 | 3.4829317668141524e-3 |
Outlying measurements have severe (0.5843566620274605%) effect on estimated standard deviation.
joinWordsSeq/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.9480646817246454e-2 | 5.052658008254906e-2 | 5.1631606757422184e-2 |
| Standard deviation | 1.4364668818599216e-3 | 2.1910865099085356e-3 | 3.0880145349428917e-3 |
Outlying measurements have moderate (0.13871852996781847%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.