Empirical attainment function (EAF)

Plots the Empiriccal Attainment Function (EAF) given approximation sets of multi-objective stochastic algorithms.

The following description closely follows the excellent introduction into performance assessment of multi-objective optimizers by Knowles et al. [3]. (Evolutionary) multi-objective algorithms are stochastic. Hence, naturally, the result produced by such an algorithm can be described by a probability distribution. This distribution can be characterized by a random set $$ Z = \{z^j \in R^m \,|\, j = 1, \ldots, |Z|\} $$ where the size \(|Z|\) is also random. The attainment function \(\alpha_Z : R^m \to [0,1]\) is defined as $$ \alpha_Z(z) = P(Z \preceq \{z\}) = P\left(z^1 \preceq z \lor \ldots \lor z^{|Z|} \preceq z\right). $$ where \(z^j \preceq z\) denotes that \(z^j\) weakly dominates \(z\). This can be interpreted as the probability to reach goal \(z\) in the sense that there is at least one objective vector in the solution set that weakly dominates, i.e., “attains”, \(z\).

Given \(r\) independent runs of a stochastic multi-objective optimizer and denoting by \(X^i, 1 \leq i \leq r\) the corresponding Pareto-front approximation set, the empirical attainment function (EAF) is defined as $$ \hat{\alpha}_r(z) = \frac{1}{r} \sum_{i=1}^{r} I(X^i \preceq \{z\}) $$ where \(I(\cdot)\) is the indicator function evaluating to \(1\) if and only if the condition is true; \(0\) otherwise. This is a straight-forward estimation from empirical data and simply describes the frequency of attaining \(z\) within \(r\) runs.

The EAF can be used for visualization. Here, we plot the so-called \(k\%\)-attainment surface which splits “the goals that have been attained and the goals that have not been attained with a frequency of at least \(k\) percent” [3].

For further details we refer the reader to the technical report by Knowles, Thiele and Zitzler (reference [3]).

plot_eaf(df, obj.cols, percentiles = c(0, 50, 75, 100))

Arguments

df	[data.frame] Data frame with columns at least those given via parameter obj.cols, “problem” and “algorithm”.
obj.cols	[character(>= 2)] Column names of the objective function values. Default is c("y1", "y2").
percentiles	[numeric] Percentiles of the EAF that will be plotted as attainment surfaces.

Value

A ggplot object.

References

[1] V. Grunert da Fonseca and C. M. Fonseca, The attainment-function approach to stochastic multiobjective optimizer assessment and comparison, in Experimental Methods for the Analysis of Optimization Algorithms (T. Bartz-Beielstein, M. Chiarandini, L. Paquete, and M. Preuss, eds.), ch. 5, pp. 103-130, Springer Berlin Heidelberg, 2010.

[2] Manuel López-Ibáñez, Luís Paquete, and Thomas Stützle. Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization. In T. Bartz-Beielstein, M. Chiarandini, L. Paquete, and M. Preuss, editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 209–222. Springer, Berlin, Germany, 2010. doi: 10.1007/978-3-642-02538-9_9.

[3] Knowles, J. D., Thiele, L. and Zitzler, E. A tutorial on the performance assessment of stochastive multiobjective optimizers. TIK-Report No. 214, Computer Engineering and Networks Laboratory, ETH Zurich, February 2006 (Revised version. First version, January 2005). doi: 10.3929/ethz-b-000023822.

See also

Other multi-objective visualizations: plot_eaf_diff(), plot_heatmap(), plot_pcp(), plot_radar(), plot_scatter2d(), plot_scatter3d()

Examples

if (FALSE) {
data(emoas_on_zdt)
plot_eaf(emoas_on_zdt, obj.cols = c("y1", "y2"))
plot_eaf(emoas_on_zdt[emoas_on_zdt$algorithm == "nsga2", ], obj.cols = c("y1", "y2"))
plot_eaf(emoas_on_zdt[emoas_on_zdt$algorithm == "nsga2", ], obj.cols = c("y1", "y2"),
  percentiles = c(50, 100))
}