Evaluating AI support across the robotics software lifecycle
The ASTIR Robotics Software Engineering Benchmark, or RSEB, provides a structured and reproducible basis for evaluating AI-supported software-engineering tools for intelligent robotics.
General language and coding benchmarks provide limited evidence about performance in robotics. Robotics software must interact with sensors, actuators, control systems, physical environments and safety constraints. Its engineering also involves more than code generation: requirements must remain traceable, architectures consistent, tests representative and deployed systems observable.
RSEB addresses these challenges through benchmark tasks derived from three ASTIR application domains:
- Manufacturing robotics
- Service robotics
- Agricultural unmanned aerial vehicles
The benchmark is designed as a living resource. Its datasets, evaluation artefacts and metrics will be extended as ASTIR technologies progress from component development to integrated industrial validation.
Five benchmark modules
| Module | Benchmark artefacts | Evaluation tasks | Principal metrics |
|---|---|---|---|
| Requirements engineering | Textual and structured requirements with links to related engineering artefacts | Requirements extraction, consistency and completeness checking, comparison of AI-generated and manually authored requirements | Precision, recall, F1 score, consistency-violation rate and traceability coverage |
| System architecture and design | UML and SysML models, architectural diagrams and pattern libraries | Model-to-model transformation, model-to-text transformation and architecture synthesis | Model completeness, syntactic correctness, transformation accuracy and maintainability |
| Implementation and code generation | Industrial C++, Python and ROS 2 code, together with AI-generated equivalents | Code synthesis, refactoring and optimisation for robotics controllers and platforms | Syntactic validity, functional equivalence, reference-test results, runtime efficiency and developer-effort reduction |
| Testing and quality assurance | Unit and integration tests, simulation test cases and coverage information | Test generation, coverage maximisation and defect detection | Test-generation success, code coverage, fault-detection rate and test flakiness |
| Runtime monitoring and maintenance | Runtime logs, anomaly traces and labelled operational events | Anomaly detection, predictive maintenance and log-based reasoning | Detection latency, false-positive and false-negative rates, prediction accuracy and mean time to diagnosis |
Benchmark datasets
The initial benchmark architecture defines three pilot-based datasets.
DS-01: Manufacturing robotics
Source: Manufacturing pilot
Indicative contents: annotated software artefacts, PLC sequences, device lists, component diagrams, cycle-time and material-flow information, runtime logs and safety-check information
Documented formats: JSON and YAML
Indicative size in RSEB v1.0: 2 GB
Licence: Apache 2.0 for releasable resources
Sensitivity: no sensitivity classification recorded in the initial metadata table
Planned access: ASTIR repository and Zenodo
This dataset supports requirements engineering, architecture analysis, traceability and manufacturing-software evaluation.
DS-02: Service robotics
Source: Service-robotics pilot
Indicative contents: ROS 2 packages, API definitions, Gazebo environments, TIAGo-related software, telemetry, runtime logs, test results and developer documentation
Documented formats: ROS bag and CSV
Indicative size in RSEB v1.0: 1.5 GB
Licence: CC BY 4.0 for releasable resources
Sensitivity: low
Planned access: ASTIR repository and Zenodo
This dataset supports software migration, automated testing, integration assessment, runtime monitoring and maintenance.
DS-03: Agricultural UAVs
Source: Agricultural-UAV pilot
Indicative contents: flight-control source code, mission plans, mission-planning scripts, PDDL symbolic plans, test cases, telemetry and selected NDVI or vision-language imagery
Documented formats: C++ and YAML; additional telemetry and imagery formats will be documented with the released resources
Indicative size in RSEB v1.0: 0.9 GB
Licence: Apache 2.0 for releasable resources
Sensitivity: medium
Planned access: ASTIR repository and Zenodo
This dataset supports code generation, symbolic mission planning, testing, anomaly detection and evaluation of vision-language support for UAV engineering.
Models and AI tools
RSEB is intended to evaluate model classes and engineering tools rather than prescribe one particular model. Relevant technologies include:
- Large language models for requirements analysis, code generation, test generation and documentation
- Vision-language models for interpreting UAV imagery and connecting visual observations with mission-planning activities
- Generative-AI systems for architecture synthesis and engineering-artefact transformation
- Anomaly-detection and predictive-maintenance models
- Symbolic planning tools, including PDDL-based mission planning
- Hybrid AI components combining generated outputs with engineering constraints and validation rules
Specific model names, versions, prompts and configurations will be recorded for each benchmark execution. ASTIR has not yet published an approved list of reference models.
How benchmark results are assessed
RSEB combines task-specific technical metrics with wider indicators of engineering value and trustworthy AI. These include:
- Functional correctness and reference-test results
- Software and model quality
- Development-time and engineering-effort reduction
- Runtime and resource efficiency
- Traceability between requirements, models, code and tests
- Hallucination rate and explanation clarity
- Human oversight and accountability
- Cybersecurity and data protection
- Reproducibility across tools, partners and benchmark releases
Results are reported through a common JSON structure and baseline scoring interface. This makes evaluations more comparable and allows results from different ASTIR components to be incorporated into a shared assessment process.
Reproducibility and quality assurance
Benchmark resources are intended to be version-controlled, containerised and accompanied by machine-readable metadata. The supporting environment defined in D1.4 includes:
- Python, ROS and ROS 2
- Docker or Podman containers
- Git-based repositories
- Jupyter Notebooks and PyTest
- Automated CI/CD validation
- JSON-LD metadata
- REST interfaces for data access and reporting
- Semantic versioning and planned Zenodo archiving
Each benchmark artefact undergoes technical and ethical review by at least two consortium partners. Selected tasks are independently repeated to verify that the reported procedures and metrics can be reproduced.
Access and licensing
ASTIR applies a dual-access model:
- Non-sensitive datasets and artefacts will be published under licences such as CC BY 4.0 or Apache 2.0.
- Confidential or commercially sensitive resources will remain restricted to the consortium.
- Public metadata will describe restricted datasets, including their structure, provenance and access conditions.
- Public releases are planned for preservation through Zenodo and the ASTIR GitHub organisation.
The availability of individual resources is subject to technical validation and intellectual-property, cybersecurity and data-protection review.
Release roadmap
| Release | Planned scope |
|---|---|
| RSEB v1.0 | Initial datasets from all three pilots, metadata templates and baseline evaluation scripts |
| RSEB v2.0 | Extended datasets and generated artefacts from ASTIR technology development; refined metrics and additional validation cases |
| RSEB v3.0 | Consolidated and quality-assured benchmark aligned with the final ASTIR evaluations, including persistent publication and complete documentation |
For questions about the benchmark or future releases, please contact the ASTIR consortium through the project website.
The benchmark description is based on ASTIR Deliverable D1.4, “Robotics Software Engineering Benchmark”.