Empirical evidence is important to produce long-lasting impact research. We feel that the tools and experiments that are used to produce or validate research results are not given as much attention as they should. To counteract this tendency,\u00a0 Artifact Evaluation<\/b> (AE) rewards well written tools allowing researchers to replicate the experiments presented in papers. The purpose of the AE process is mainly to improve the reproducibility of computational results.<\/p>\n
ECRTS has been the first real-time systems conference to introduce artifact evaluation<\/b> in 2016, and has continued since then.\u00a0<\/p>\n
Authors of accepted papers<\/strong> with a computational component will be invited to submit<\/b> their code and\/or their data to an optional AE process. We seek to achieve the benefits of the AE process without disturbing the current process through which ECRTS has generated high-quality programs in the past. In particular, the decision to submit or not an artifact has no impact on whether a paper is accepted at ECRTS. Moreover, there will be\u00a0no disclosure of the title and authors of papers which would not pass the repeatability evaluation<\/strong>.<\/p>\n The authors of the papers corresponding to the artifacts which pass the evaluation can decide to use a\u00a0seal that indicates that the artifact has passed the repeatability test, <\/strong>and the artifact will be published in Dagstuhl Artifacts Series<\/a> (DARTS).<\/p>\n We recognize that not all the results are repeatable. For instance, the execution time of the experiments may be too long or a complete infrastructure to execute the tests may be required, but not be available to the evaluators. We encourage submissions but we can only guarantee to repeat experiments that are reasonably repeatable with regular computing resources. Our focus is on: (1) replicating the tests that are repeatable; (2) improving the repeatability infrastructure so that more tests become repeatable in the future.<\/p>\n Artifacts should include two components:<\/p>\n A good how-to to prepare an artifact evaluation package is available at http:\/\/bit.ly\/HOWTO-AEC<\/a>.<\/p>\n The evaluation process is single-blind. It is non-competitive and we hope that all the artifacts submitted can pass the evaluation criteria.<\/p>\n\n\n Special Artifacts<\/strong><\/p>\n\n\n\n If you are not in a position to prepare the artifact as above, or if your artifact requires special libraries, commercial tools (e.g., MATLAB or specific toolboxes), or particular hardware, please contact the AE chair as soon as possible.<\/p>\n\n\n\n Based on previous experience, the biggest hurdle to successful reproducibility is the setup and installation of the necessary libraries and dependencies. Authors are therefore encouraged to prepare a virtual machine (VM) image including their artifact (if possible) and to make it available via HTTP throughout the evaluation process (and, ideally, afterwards). As the basis of the VM image, please choose commonly-used OS versions that have been tested with the virtual machine software and that evaluators are likely to be accustomed to. We encourage authors to use VirtualBox (https:\/\/www.virtualbox.org<\/a>) and save the VM image as an Open Virtual Appliance (OVA) file. To facilitate the preparation of the VM, we suggest using the VM images available Timeline<\/strong><\/p>\n\n\n\n Submission Process<\/strong><\/p>\n\n\n\n Given that your paper has been accepted, we highly encourage you to submit to the ECRTS’23 Artifact Evaluation (AE). The artifact submission deadline is Monday, May 6, 2024, with an abstract due Monday, April 29, 2024.<\/p>\n\n\n\n Authors of artifacts that pass the evaluation will be asked to submit the final artifact version to Dagstuhl Artifacts Series<\/a> (DARTS). The deadline is tba.<\/p>\n\n\n\n Submission Site<\/strong>: https:\/\/easychair.org\/conferences\/?conf=ecrtsae24<\/a><\/p>\n\n\n\n Organizers<\/strong><\/p>\n\n\n\n Artifact Evaluation co-chairs:<\/p>\n\n\n\n Evaluation committee:<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Motivation Empirical evidence is important to produce long-lasting impact research. We feel that the tools and experiments that are used to produce or validate research results are not given as much attention as they should. To counteract this tendency,\u00a0 Artifact Evaluation (AE) rewards well written tools allowing researchers to replicate…<\/p>\nFormatting instructions<\/h5>\n
\n
Recommendation: Use Virtual Machines<\/strong><\/p>\n\n\n\n
at https:\/\/www.osboxes.org<\/a>\/.<\/p>\n\n\n\n\n
<\/li>\n<\/ul>\n\n\n\n\n
KTH Royal Institute of Technology SE<\/li>\n\n\n\n
Davidson College, North Carolina, USA<\/li>\n<\/ul>\n\n\n\n\n