Robust Vision Challenge

Joining multiple rankings into one in a fair manner is a non-trivial task. In electorial science the equivalent task of finding one consensus ordering based on many ordered votes has been a central question for many years. The currently most-accepted solution for this problem is the Schulze Proportional Ranking (PR) method [1] for sorted lists of winners. We are using Schulze PR to rank the top ten in each challenge. The software used for this is a fork of the pyvotecore implemenation which you can find here.

[1] Markus Schulze. "A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election method". In Social Choice and Welfare. 2011.

We first rank each method according to a representative subset of metrics per dataset and then rank the resulting ranks to yield one global rank. The metrics and rankings per dataset can be viewed by clicking on "Detailed subrankings". Note that these rankings may differ from the rankings on the original websites as those typically consider a single metric while we are looking for robustness and thus take a number of metrics per dataset into account.




eXTReMe Tracker