Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions
Fulltext URI
Document type
Additional Information
relationships.isAuthorOf
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Benchmarking via carefully designed competitions makes it possible to provide a common framework for the rigorous comparison of intelligent and autonomous systems; competitions may play the role of scientific experiments while being appealing both to researchers and to the general public thus promoting critical analysis of systems outside the labs. This paper describes our approach to benchmarking domestic service robots through organizing recurrent competitions under the European Robotics League. It details the tools and benchmarks designed to evaluate the performance of robots at task and functionality levels. In particular, the functionality benchmarks for object perception and navigation are described and an overview of the new benchmarks to appear in the league is presented.
Description
Keywords
Citation
URI
Collections
Endorsement
Review
Supplemented By
Referenced By
Number of citations to item: 15
- Yinan Jin, Yixiao Zheng, Junhao Lin, Kaiyu Yang, Shibo Cai, Guanjun Bao, Shahid Hussain, Feng Gao (2026): A Novel Visuo-Tactile Sensor for Contacting Perception and Object Recognition, In: IEEE Sensors Journal 2(26), doi:10.1109/jsen.2025.3637832
- Meysam Basiri, João Pereira, Rui Bettencourt, Enrico Piazza, Emanuel Fernandes, Carlos Azevedo, Pedro Lima (2022): Functionalities, Benchmarking System and Performance Evaluation for a Domestic Service Robot: People Perception, People Following, and Pick and Placing, In: Applied Sciences 10(12), doi:10.3390/app12104819
- Enrico Piazza, Pedro U. Lima, Matteo Matteucci (2022): Performance Models in Robotics With a Use Case on SLAM, In: IEEE Robotics and Automation Letters 2(7), doi:10.1109/lra.2022.3151162
- Meysam Basiri, João Gonçalves, José Rosa, Alberto Vale, Pedro Lima (2021): An autonomous mobile manipulator to build outdoor structures consisting of heterogeneous brick patterns, In: SN Applied Sciences 5(3), doi:10.1007/s42452-021-04506-7
- Miguel Á. González-Santamarta, Francisco J. Rodríguez-Lera, Claudia Álvarez-Aparicio, Ángel M. Guerrero-Higueras, Camino Fernández-Llamas (2020): MERLIN a Cognitive Architecture for Service Robots, In: Applied Sciences 17(10), doi:10.3390/app10175989
- Santosh Thoduka, Nico Hochgeschwender (2021): Benchmarking Robots by Inducing Failures in Competition Scenarios, In: Lecture Notes in Computer Science, doi:10.1007/978-3-030-77820-0_20
- L. Villamar Gómez, J. Miura (2021): Ontology-based knowledge management with verbal interaction for command interpretation and execution by home service robots, In: Robotics and Autonomous Systems, doi:10.1016/j.robot.2021.103763
- Santosh Thoduka, Deebul Nair, Praminda Caleb-Solly, Mauro Dragone, Filippo Cavallo, Nico Hochgeschwender (2024): Trust in Robot Benchmarking and Benchmarking for Trustworthy Robots, In: Studies in Computational Intelligence, doi:10.1007/978-3-031-55817-7_3
- Enrico Piazza, Emanuele Vivani, Matteo Matteucci, Pedro U. Lima (2024): Performance Modelling of Autonomous Mobile Robot Software Components, In: Studies in Computational Intelligence, doi:10.1007/978-3-031-55817-7_5
- Tatsuya Matsushima, Yuki Noguchi, Jumpei Arima, Toshiki Aoki, Yuki Okita, Yuya Ikeda, Koki Ishimoto, Shohei Taniguchi, Yuki Yamashita, Shoichi Seto, Shixiang Shane Gu, Yusuke Iwasawa, Yutaka Matsuo (2022): World robot challenge 2020 – partner robot: a data-driven approach for room tidying with mobile manipulator, In: Advanced Robotics 17-18(36), doi:10.1080/01691864.2022.2114297
- Matthew Studley, Sarah Carter, Francisco J. Perez-Grau, Antidio Viguria Jiménez, Gabriele Ferri, Fausto Ferreira, Deebul Nair, Sven Schneider, Paul G. Plöger, Pedro U. Lima, Meysam Basiri, Gerhard K. Kraetzschmar, Daniele Nardi, Lun Wang, Emanuele Antonioni, Vincenzo Suriani, Luca Iocchi (2023): Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot Competitions, In: Human-Robot Interaction - Perspectives and Applications, doi:10.5772/intechopen.108583
- Meysam Basiri, João Gonçalves, José Rosa, Rui Bettencourt, Alberto Vale, Pedro Lima (2021): A Multipurpose Mobile Manipulator for Autonomous Firefighting and Construction of Outdoor Structures, In: Field Robotics, doi:10.55417/fr.2021004
- Liu Yanmei, Wen Shihua, Chen Zhen, Zhang Dongyu (2020): Research of the Anti- vibration Hammer Resetting Robot Based on Machine Vision, In: 2020 Chinese Control And Decision Conference (CCDC), doi:10.1109/ccdc49329.2020.9164645
- Riccardo Bertoglio, Giulio Fontana, Matteo Matteucci, Davide Facchinetti, Michel Berducat, Daniel Boffety (2021): On the Design of the Agri-Food Competition for Robot Evaluation (ACRE), In: 2021 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), doi:10.1109/icarsc52212.2021.9429792
- Tiago Ribeiro, Fernando Gonçalves, Inês S. Garcia, Gil Lopes, António F. Ribeiro (2021): CHARMIE: A Collaborative Healthcare and Home Service and Assistant Robot for Elderly Care, In: Applied Sciences 16(11), doi:10.3390/app11167248
