The Science of Testing: An Automotive Perspective

Siddartha Khastgir, Stewart Birrell, Gunwant Dhadyalla, Paul Jennings

Research output: Contribution to journalConference article

Abstract

Increasing automation in the automotive systems has re-focused the industry's attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain. Eighteen experts (each with over 10 years' of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories. The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that "good" scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.

Original languageEnglish
Article number2018-01-1070
JournalSAE Technical Papers
Volume2018-April
DOIs
Publication statusPublished - 3 Apr 2018
Externally publishedYes
Event2018 SAE World Congress Experience, WCX 2018 - Detroit, United States
Duration: 10 Apr 201812 Apr 2018
http://wcx18.org/

Fingerprint

Advanced driver assistance systems
Testing
Supply chains
Industry
Hazards
Automation

ASJC Scopus subject areas

  • Automotive Engineering
  • Safety, Risk, Reliability and Quality
  • Pollution
  • Industrial and Manufacturing Engineering

Cite this

Khastgir, S., Birrell, S., Dhadyalla, G., & Jennings, P. (2018). The Science of Testing: An Automotive Perspective. SAE Technical Papers, 2018-April, [2018-01-1070]. https://doi.org/10.4271/2018-01-1070

The Science of Testing : An Automotive Perspective. / Khastgir, Siddartha; Birrell, Stewart; Dhadyalla, Gunwant; Jennings, Paul.

In: SAE Technical Papers, Vol. 2018-April, 2018-01-1070, 03.04.2018.

Research output: Contribution to journalConference article

Khastgir, S, Birrell, S, Dhadyalla, G & Jennings, P 2018, 'The Science of Testing: An Automotive Perspective' SAE Technical Papers, vol. 2018-April, 2018-01-1070. https://doi.org/10.4271/2018-01-1070
Khastgir S, Birrell S, Dhadyalla G, Jennings P. The Science of Testing: An Automotive Perspective. SAE Technical Papers. 2018 Apr 3;2018-April. 2018-01-1070. https://doi.org/10.4271/2018-01-1070
Khastgir, Siddartha ; Birrell, Stewart ; Dhadyalla, Gunwant ; Jennings, Paul. / The Science of Testing : An Automotive Perspective. In: SAE Technical Papers. 2018 ; Vol. 2018-April.
@article{65691dbd8c7d4fa987ed58ddd624ca73,
title = "The Science of Testing: An Automotive Perspective",
abstract = "Increasing automation in the automotive systems has re-focused the industry's attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain. Eighteen experts (each with over 10 years' of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories. The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that {"}good{"} scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.",
author = "Siddartha Khastgir and Stewart Birrell and Gunwant Dhadyalla and Paul Jennings",
year = "2018",
month = "4",
day = "3",
doi = "10.4271/2018-01-1070",
language = "English",
volume = "2018-April",
journal = "SAE Technical Papers",
issn = "0148-7191",
publisher = "SAE International",

}

TY - JOUR

T1 - The Science of Testing

T2 - An Automotive Perspective

AU - Khastgir, Siddartha

AU - Birrell, Stewart

AU - Dhadyalla, Gunwant

AU - Jennings, Paul

PY - 2018/4/3

Y1 - 2018/4/3

N2 - Increasing automation in the automotive systems has re-focused the industry's attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain. Eighteen experts (each with over 10 years' of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories. The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that "good" scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.

AB - Increasing automation in the automotive systems has re-focused the industry's attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain. Eighteen experts (each with over 10 years' of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories. The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that "good" scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.

UR - http://www.scopus.com/inward/record.url?scp=85045543619&partnerID=8YFLogxK

U2 - 10.4271/2018-01-1070

DO - 10.4271/2018-01-1070

M3 - Conference article

VL - 2018-April

JO - SAE Technical Papers

JF - SAE Technical Papers

SN - 0148-7191

M1 - 2018-01-1070

ER -