Artifact Evaluation

Empirical evidence is important to produce long-lasting impact research. The computation tools that are used to produce or validate the research are not nearly given as much attention as they should. Computations – no matter the research field – are rarely peer reviewed. On the other hand, some researchers spend a vast amount of time building tools that do not receive the recognition they deserve. Building complex tools is often a long process and in the current research world the building effort is rewarded less than one would desire, except for the results that the tool produces. This in turn often leads to code written in a short period of time, that is used for the single purpose of validating the claim of one paper and then abandoned, despite its potential utility to other researchers in the field. To invert this tendency, some communities have started a process called Artifact Evaluation (AE), that rewards well written tools that allows researchers to replicate the experiments presented in papers. The purpose of the AE process is mainly to improve the reproducibility of computational results, and therefore raising the profile of papers and authors. Not secondly, there has been evidence that sharing research data improves the impact of papers in the biosciences.

Following this trend, authors of accepted papers with a computational component will be invited to submit their code and/or their data to an optional repeatability evaluation or artifact evaluation (AE) process.We seek to achieve the benefits of the AE process without disturbing the current process through which ECRTS has generate high-quality programs in the past. Therefore, the current submission, review and acceptance procedure would be completely unaltered by the decision of running an AE process. Only once acceptance decisions are final, the authors of accepted papers will be invited to submit a artifact evaluation (or replication) package. Hence, the repeatability evaluation process has no impact on whether a paper is accepted at ECRTS, and will be entirely optional and up to authors. Moreover, there will be no disclosure of the title and authors of papers which would not pass the repeatability evaluation.

The authors of the papers corresponding to the artifacts which pass the evaluation can decide to use a seal that indicates that the artifact has passed the repeatability test. Artifacts should include two components: (a) a document explaining how to use the artifact and which of the experiments presented in the paper are repeatable (with reference to specific digits, figures and tables in the paper), the system requirements and instructions for installing and using the artifact; (b) the software and any accompanying data. A good how-to to prepare an artifact evaluation package is available online at http://bit.ly/HOWTO-AEC.

We recognize that not all the results are repeatable. In some cases, the execution time of the experiments is too long. In some other cases, one would need a complete infrastructure to execute the tests. We encourage submissions but we will try to repeat only results that are reasonably repeatable with regular computing resources. For the first edition of the process, we argue that the focus should be on (1) replicating the tests that are repeatable, (2) improving the repeatability infrastructure so that more tests would become repeatable in the future.

The artifact submitted is treated as confidential and will not be released. The hope, here, is that since the authors are already working on packaging the material for the submission, they are encouraged to release the artifact as well. This is however not mandatory and it is up to the authors to decide. The evaluation process is single-blind. The evaluation process is non-competitive. The acceptance of the papers has already been decided and the hope is that all the artifacts submitted would pass the evaluation criteria.

If the authors of a paper are willing to submit an artifact, they should package and submit it before May 6th - via email to martina.maggio@control.lth.se. The artifact will be evaluated and a response will be returned to the authors on May 14th, with the image to be used as the seal of approval in case the artifact is positively evaluated. This allows the authors to include the seal of approval before their camera ready deadline on May 17th.

Update
The evaluation process has now terminated and 5 papers received the seal of approval after the process completion. The evaluation criteria have been ultimately simple. The authors provided instructions on how to replicate the results presented in their papers to the members of the Artifact Evaluation Committee, and the committee checked that it was indeed possible to follow the provided instructions and to reproduce the results marked as reproducible. Given that the Artifact Evaluation procedure was only at its first attempt, we aimed for partial reproducibility. The evaluators did not have much time to replicate the results and for some paper the full reproduction would have taken days. We therefore also approved papers where a subset of the results was reproducible.

The papers which passed the evaluation are:

Sanjoy Baruah, Alan Burns, Zhishan Guo; Scheduling mixed-criticality systems to guarantee some service under all non erroneous behaviors
Michael Jacobs, Sebastian Hahn, Sebastian Hack; A framework for the derivation of WCET analyses for multi-core processors
Felipe Cerqueira, Felix M. Stutz, Björn B. Brandenburg; PROSA: A Case for Readable Mechanized Schedulability Analysis
Vincenzo Bonifaci, Björn B. Brandenburg, Gabriele D'Angelo, Alberto Marchetti-Spaccamela; Multiprocessor Real-Time Scheduling with Hierarchical Processor Affinities
Alessandro Biondi, Björn B. Brandenburg; Lightweight Real-Time Synchronization under P-EDF on Symmetric and Asymmetric Multiprocessors

Artifact Evaluation Committee
We would like to thank the evaluators that have coped with the short time allowed for this specific task.

Questions?
Contact Martina Maggio - martina.maggio@control.lth.se