Prof. Dr. Toshiaki Aoki, JAIST

• Abstract

  We are working on practical applications of formal methods to automotive systems. We successfully applied the formal methods to basic software including automotive operating systems such as OSEK/VDX, Classic AUTOSAR, and Adaptive AUTOSAR OSs so far. We would like to extend our target to more modern automotive system platforms, in particular, those for automated driving systems (ADS). The modern automotive system consists of AI for perception and planning, control, and basic software for high-performance computing. Recently, our project titled 'Formal Methods and Verification Tools for Next-generation Automotive System Platforms' which focuses on such modern automotive systems has been accepted by JST/CREST. This project aims at proposing formal methods and verification tools to ensure the safety and reliability of next-generation automotive system platforms. These formal methods and verification tools cover the perception to control functions, and we stick in their practical application to real systems. In the first half of this talk, I would like to introduce the overview of the JST/CREST project.  In the second half of the talk, I would like to focus on our scenario modeling language which is one of techniques being proposed in our project. In practice, the safety of ADS is assessed based on scenarios as shown in ISO 34502. The scenarios represent under-approximation of the whole situation that the system is operated. Thus, it is necessary that the scenarios sufficiently cover important situations; however, it is challenging since the number of the scenarios is huge. We think that one idea to mitigate this problem is to provide the comprehensive and compact representation of the scenarios, which allows us to effectively review it as well as generate scenarios. So far, we proposed a scenario modeling language named CPD (Car Position Diagram) and its scenario generator GCPD based on a SMT solver. Several standards about scenario-base safety analysis have been proposed recently. JAMA (Japan Automobile Manufactures Association) published Automated Driving Safety Evaluation Framework. In this document, cognitive, traffic, and motion disturbances are systematically analyzed, and their variations are exhaustively defined by matrices. Each of elements of the matrices represents a scenario; however, it contains ambiguity which comes from visual notations used in the document. ISO 34502 which is defined based on JAMA's framework uses zone-graphs to visually describe scenarios (Annex E). The zone-graph is more formal than the visual notation used in JAMA's framework; however, there is still room for improvement from my point of view. We are formalizing scenarios appearing in these standards using our CPD/GCPD now. In this talk, I would like to discuss how we should deal with such scenarios based on our modeling language.

Presentation (password necessary)

Dr. Adam Molin, DENSO Automotive GmbH

• Abstract

  Abstract: Verification and Validation (V&V) processes play a vital role in ensuring the safety and reliability of automated driving applications. Scenario-based testing has emerged as an effective approach for identifying critical scenarios that challenge the capabilities of automated driving systems. This presentation aims to explore the methodology of scenario-based testing and its application to automatically find unknown critical test cases and to derive critical influence factors by analyzing data produced by simulation. The influence factors can range from parameters of the dynamic behavior of the actors, the roadway characteristics, or environmental conditions.

  Due to the potentially infinite number of scenario instances and the abundance of influence factors and scenario parameters - even in a constrained operational design domain, uncovering unknown critical scenarios efficiently at a high coverage is still challenging. However, achieving a high coverage of the critical scenarios is essential to validate the safety of the automated system. This talk targets at efficient methods to uncover unknown critical scenarios for automated driving functions using scenario-based testing. Inspired by the ISO 21448 SOTIF standard, we illustrate how elements from the safety domain can be mapped to the activity of critical scenario identification. The mapping is helpful in various ways. It identifies the different sources of the unknown and opens the potential for directing the search by shaping the criticality metrics. Based on these high-level insights, we will derive a workflow for critical scenario identification using simulation-based testing. The workflow takes as input an abstract scenario and a criticality metric. By applying Blackbox-optimization techniques, the algorithm iteratively drives the search within the abstract scenario towards a critical test case related to the criticality metric. The workflow will eventually output critical concrete test cases, that can be used for further analysis or for consideration in the automated driving function development.

  Throughout the presentation, we will share our experiences and best practices that we have been found from various R&D activities on critical scenario identification. As such, we discuss different non-proprietary scenario formats with different levels of abstractions and evaluate them against their utility for identifying unknown critical scenarios. Moreover, we delve into the question of how to find good trade-offs between exploration and exploitation in terms of the design of the scenario space, criticality metric, and the Blackbox optimization search methods in use. For the latter, we show some insights on the nature of these optimization techniques. Finally, we give an outlook on how to increase the effectiveness of critical scenario identification to fully explore the diversity of critical factors, while keeping the search space at a manageable level.

Presentation (password necessary)

Dr. Christian Schyr, AVL Deutschland GmbH

• Abstract

  Zum Nachweis der sicheren Funktion von hochautomatisierten Fahrzeugen werden nach dem aktuellen Stand der Technik fest definierte Szenarienkataloge zum manöverbezogenen Nachweis sowie mehrere Millionen Fahrkilometer umfassende Echtzeitdaten zum statistischen Nachweis herangezogen.
  Zur Entwicklung neuer Fahrzeuge mit Automatisierungslevel 4-5 ist es unabdingbar, eine selektive Erfassung relevanter und kritischer Fahrsituationen, signifikanter Umgebungsdaten sowie der Rohdaten der Fahrzeugsensorik schon während des Fahrbetriebs zu erreichen.
  Diese Daten werden benötigt, um die durch KI getroffenen Entscheidungen validieren, verbessern und reproduzieren zu können, mit dem Ziel somit die notwendige Testabdeckung für zukünftige Funktionalitäten zu erreichen.
  Im Rahmen des Projektes KIsSME wurden wie in Abbildung 1 dargestellt KI-basierte Algorithmen erarbeitet und angewendet, die On-Board-Systeme ertüchtigt, relevante und kritische Szenarien in Echtzeit zu erkennen und hierfür selektiv Rohdaten sowie Szenarienbeschreibungen zu erfassen.
  Die KI-basierten Algorithmen ermöglichen eine inhärente Lernfähigkeit, welche das Erkennen von kritischen Situationen und der zugehörigen relevanten Daten stetig verbessern, um für die Entwicklung von automatisierten Systemen der Level 4-5 die Informationsdichte der für die Erprobung genutzten Daten zu steigern und gleichzeitig die hierfür notwendigen Datenvolumen sowie Aufwände zur Sicherstellung des Datenschutzes signifikant zu reduzieren.
  Im Projekt wurden von den Verbundpartnern insgesamt sieben Demonstratorfahrzeuge eingesetzt. Dabei wurden einerseits die Demonstratorfahrzeuge zur Gewinnung von Testdaten als auch zur Validierung der in den APs erarbeiteten KI-basierten Lösungen verwendet.

Presentation (password necessary)

Ali Shakeri, DLR e.V.

• Abstract

  In the rapidly evolving field of Automated Vehicle (AV) systems, defining an Operational Design Domain (ODD) is essential to safeguard AVs from performing maneuvers outside their capabilities. The concept of ODD has gained attention as standardization committees, including BSI PAS, ISO, and ASAM OpenODD, attempt to standardize the operational domain and ODD specifications. As a member of ASAM OpenODD, I observed that despite all efforts, there are still many misunderstandings and misconceptions surrounding the specification of ODD. These difficulties originate from previous definitions of ODD that relied on ambiguous and unclear terms and the lack of a well-defined definition for OD.
  This presentation reports on the latest status of the ASAM OpenODD standard and presents a critical review of the latest research and standardization efforts in the ODD landscape. It also explains the ambiguity in the current definitions of ODD for AVs and addresses the problems that arise from this vagueness. Finally, it shows how a formal representation of the concepts can help overcome these ambiguities and how a proper specification can be used as a foundation for developing ideas, such as ODD monitoring.

Presentation (password necessary)

Dietmar Kinalzyk - AVL Software and Functions GmbH

• Abstract

  ADAS features in cars are increasing and fully automated vehicles are used in pilot areas. Strong public concerns about their safety have prompted the sector to further safety standards beyond the item-based functional safety standard ISO 26262 [1] to achieve safety compliance. The first SOTIF standard (PAS Public Available Standard) has been published in 2019 and is released as ISO 21448 standard in 2022. It contains many new inputs but the interpretation in safety related handling is still difficult.

  Therefore, other working groups started to have a broad view into the overall safety guidance and requirements.

  SOTIF’s new safety standard approach was triggered by the ADAS and autonomous driving cars interaction with the road and their environment with several unknown and unsafe conditions and scenarios. Therefore, the working group for ISO standard “Road vehicles: Test scenarios for automated driving systems” with actual 5 parts has been started. The first 3 parts are already released.

  Road vehicles: Test scenarios for automated driving systems

  1. ISO 34501:2022 Vocabulary
  2. ISO 34502:2022 Scenario based safety evaluation framework.
  3. ISO 34503:2023 Specification for operational design domain
  4. ISO 34504:2023 FDIS Scenario categorization
  5. ISO 34505:2024 CD Scenario evaluation and test case generation

  This standard compilation explains the working with test scenarios for automated driving systems as enhancement to the SOTIF standard ISO21448. The presentation summarizes the content and outlook of the actual standard in combination with the existing standards for safe automated driving for road vehicles.

Presentation (password necessary)

Felix Brüning, Universität Bremen

• Abstract

  For highly complex systems like partially autonomous vehicles, trains, and aircrafts, conventional model-based testing is usually infeasible, since it would take too much effort to produce one comprehensive reference model reflecting all expected behaviours of the system under test. Scenario-based testing mitigates this problem by using a library of many less complex models or logical specifications, each library element describing the environmental conditions and the associated expected behaviour in a specific (usually parameterised) operational situation.  The downside of the scenario-based approach is that for certification purposes, the completeness of the scenario library has to be verified, in addition to the usual test strength justification that is also required for "conventional" model-based of specification-based testing. In this presentation, we describe promising approaches to the verification of scenario completeness, such that the risk of unidentified scenarios can be statistically quantified. As an interesting additional insight, we explain the similarities between the verification of scenario completeness and the justification of training and verification data set completeness required for the certification of neural networks trained for safety-critical functions like obstacle detection.

Presentation (password necessary)

Christop Glasmacher, ika - RWTH Aachen

• Abstract

  Automated vehicles have the opportunity to disrupt the mobility of the future. However, to introduce those vehicles into the market, their safety has to be ensured. Traditional methods relying solely on real-world testing would necessitate covering billions of kilometers, making them impractical and prohibitively expensive. Scenario based testing has become a promising approach reducing this effort by testing the system under test systematically utilizing scenarios. However, this approach comes with significant challenges: Scenarios to confront a system under test with have to be acquired or defined. Furthermore, since a significant number of scenarios is usually needed to cover an ODD comprehensively, these scenarios have to be easily accessible and manageable.
  To address the issues of acquiring and managing significant amounts of scenarios, a set of methods is developed and implemented into a scenario database. These methods are presented as an end-to-end solution, from processing trajectory data into scenarios of a single common scenario concept to generating purpose specific test scenarios to execute in a simulation. Thereby, the overall concept, specific methods and formats are presented to process large amounts of data for the urban domain automatically and highly systematically. Depending on the use case, from each scenario, different methods are presented to create targeted test scenarios. Furthermore, a solution to store and manage scenarios within a database is presented allowing for an efficient handling and selection of relatively simple as well as complex scenarios. To demonstrate the feasibility and applicability of the presented methods and concepts, a demo is available at https://scenario.center.

Presentation (password necessary)

Dr. Dejan Nickovic, AIT austrian Institute of Technology GmbH

• Abstract

  The automated generation of diverse and complextraining scenarios has been an important ingredient in manycomplex learning tasks. Especially in real-world applicationdomains, such as autonomous driving, auto-curriculum generation is considered vital for obtaining robust and generalpolicies. However, crafting traffic scenarios with multiple,heterogeneous agents is typically considered as a tedious andtime-consuming task, especially in more complex simulationenvironments. In our work, we introduce MATS-Gym, aMulti-Agent Traffic Scenario framework to train agents inCARLA, a high-fidelity driving simulator. MATS-Gym is amulti-agent training framework for autonomous driving thatuses partial scenario specifications to generate traffic scenarioswith variable numbers of agents. This paper unifies various existing approaches to traffic scenario description intoa single training framework and demonstrates how it canbe integrated with techniques from unsupervised environment design to automate the generation of adaptive auto-curricula.

Presentation (password necessary)

Jakob Reckenzaun, SETLabs Research GmbH

• Abstract

  Das Verbundforschungsprojekt ADApproved! erweitert das Sensorikzentrum Roding (Indoor AD-Testhalle) in zwei Richtungen. Zu den bisher verfügbaren Schlechtwettereinflüssen (Nebel, Regen und Gegenlicht), wird Gischt in der Simulation modelliert und in der Halle im Realversuch demonstriert. Zudem erweitert das Projekt die bisherige (manuelle) Funktionalität, um eine automatisierte, validierte Testkette aus Simulation und Sensorikprüffeld. Zusammen mit der Erforschung der Gischt-Phänomene wird die Qualität der Hallentests entscheidend verbessert

  Zur Umsetzung der angestrebten Testkette, soll das bestehende Prüffeld in der Halle um virtuelle Komponenten (Digital Twin) erweitert und automatisiert werden. Die geplante Test-Automatisierung garantiert die Wiederholbarkeit der Tests und deren Auswertung als entscheidende Grundlage für die Zertifizierung. Mit der Anbindung von modellierten Umwelteinflüssen an bestehende Umgebungssimulationen wird eine Erhöhung des Realitätsgrades erreicht. Darüber hinaus wird durch die automatisierte Messdatenerfassung des Prüffeldes eine kontinuierliche Verbesserung des Digital Twins bzw. einzelner Modellfeatures ermöglicht. Zusammen mit diesen technologischen Weiterentwicklungen, sollen Vorschläge für standardisierbare Hallen-Testszenarien und deren zugehörigen KPIs zur Bewertung vorgeschlagen werden.

Presentation (password necessary)