In this project, we develop a framework for systematic benchmarking of diagnostic technologies. The diagnosed system is the Electrical Power System (EPS) testbed in the ADAPT lab at NASA Ames Research Center. Our benchmarking approach shows how to empirically generate realistic data sets for diagnostic benchmarking, in the context of ADAPT EPS, and emphasizes the use of standardized vocabularies and protocols which together enable “apples to apples” assessments of the effectiveness of different diagnostic technologies.
The framework introduces a number of specifications including a standardized fault catalog, a common set of metrics, a library of modular and standardized test scenarios, a test protocol, and an evaluation algorithm for processing diagnostic data and calculating the performance metrics. The testing procedure is usually scenario-based, where each scenario may have faults injected into the system. To detect faults, each diagnostic algorithm has access to real-time data from the ADAPT EPS. Moreover, a standardized output scheme is enforced on the diagnostic algorithms to ensure the generation of common data sets for the calculation of metrics. The data from the testbed and this output of the diagnostic system are saved to a database, and the diagnostic algorithm performance is evaluated according to a predefined set of metrics.
The EPS testbed in the ADAPT lab at Ames Research Center.