Calculates the standard error for simple and percent differences

Source: R/calc_se.R

Calculates the sample standard error for the estimator differences between multiple algorithms on a given instance.

calc_se(
  Xk,
  dif = "simple",
  comparisons = "all.vs.all",
  method = "param",
  boot.R = 999
)

Arguments

Xk

list object where each position contains a vector of observations of algorithm k on a given problem instance.

dif

name of the difference for which the SEs are desired. Accepts "perc" (for percent differences) or "simple" (for simple differences)

comparisons

standard errors to be calculated. Accepts "all.vs.first" (in which cases the first object in algorithms is considered to be the reference algorithm) or "all.vs.all" (if there is no reference and all pairwise SEs are desired).

method

method used to calculate the interval. Accepts "param" (using parametric formulas based on normality of the sampling distribution of the means) or "boot" (for bootstrap).

boot.R

(optional) number of bootstrap resamples (if method == "boot")

Value

a list object containing the following items:

  • Phi.est - estimated values of the statistic of interest for each pair of algorithms of interest (all pairs if comparisons == "all.vs.all", or all pairs containing the first algorithm if comparisons == "all.vs.first").

  • se - standard error estimates

Details

  • If dif == "perc" it returns the standard errors for the sample estimates of pairs \((mu2 - mu1) / mu\), where \(mu1, mu2\) are the means of the populations that generated sample vectors \(x1, x2\), and

  • If dif == "simple" it returns the SE for sample estimator of \((mu2 - mu1)\)

References

  • F. Campelo, F. Takahashi: Sample size estimation for power and accuracy in the experimental comparison of algorithms. Journal of Heuristics 25(2):305-338, 2019.

Author

Felipe Campelo (fcampelo@ufmg.br, f.campelo@aston.ac.uk)

Examples

# three vectors of normally distributed observations
set.seed(1234)
Xk <- list(rnorm(10, 5, 1),  # mean = 5, sd = 1,
           rnorm(20, 10, 2), # mean = 10, sd = 2,
           rnorm(50, 15, 5)) # mean = 15, sd = 3

calc_se(Xk, dif = "simple", comparisons = "all.vs.all", method = "param")
#>   Alg1 Alg2 N1 N2       Phi        SE   r      ropt
#> 1    1    2 10 20 -4.877040 0.5029276 0.5 0.5678141
#> 2    1    3 10 50 -9.302593 0.8235228 0.2 0.1850679
#> 3    2    3 20 50 -4.425553 0.8560417 0.4 0.3259303
calc_se(Xk, dif = "simple", comparisons = "all.vs.all", method = "boot")
#>   Alg1 Alg2 N1 N2       Phi        SE   r      ropt
#> 1    1    2 10 20 -4.873864 0.4949271 0.5 0.5929503
#> 2    1    3 10 50 -9.334720 0.7736555 0.2 0.1742213
#> 3    2    3 20 50 -4.429824 0.8174044 0.4 0.3141521

calc_se(Xk, dif = "perc", comparisons = "all.vs.first", method = "param")
#>   Alg1 Alg2 N1 N2       Phi        SE   r      ropt
#> 1    1    2 10 20 -1.056358 0.1639697 0.5 1.1676293
#> 2    1    3 10 50 -2.014925 0.2635357 0.2 0.5579657
calc_se(Xk, dif = "perc", comparisons = "all.vs.first", method = "boot")
#>   Alg1 Alg2 N1 N2       Phi        SE   r      ropt
#> 1    1    2 10 20 -1.056731 0.1592393 0.5 1.2123248
#> 2    1    3 10 50 -2.025138 0.2630066 0.2 0.5305016

calc_se(Xk, dif = "perc", comparisons = "all.vs.all", method = "param")
#>   Alg1 Alg2 N1 N2        Phi         SE   r      ropt
#> 1    1    2 10 20 -0.5219777 0.05644348 0.5 0.5678141
#> 2    1    3 10 50 -0.9956339 0.09390564 0.2 0.1850679
#> 3    2    3 20 50 -0.4736562 0.09290748 0.4 0.3259303
calc_se(Xk, dif = "perc", comparisons = "all.vs.all", method = "boot")
#>   Alg1 Alg2 N1 N2        Phi         SE   r      ropt
#> 1    1    2 10 20 -0.5237571 0.05301572 0.5 0.5948252
#> 2    1    3 10 50 -0.9964692 0.06634091 0.2 0.1757908
#> 3    2    3 20 50 -0.4754114 0.08128578 0.4 0.3141595