The Probabilistic Certificate of Correctness Metric for Early Stage Virtual Prototype Verification and Validation

摘要

Today's complex systems have to meet hundreds of top level product acceptance requirements. It is well known (Stoll. 1999) that decisions taken early in the design process tend to have the largest impact on whether these requirements are met or not. However, current system verification and validation practices focus on comparing virtual prototype behaviour with physical prototype behavior, which is not available early in the design process. This significantly increases the risk that errors are found late in the development process, which is the leading cause of program cost overruns (DeWeck 2012). To address this issue, a virtual prototype metric called the Probabilistic Certificate of Correctness (Van der Velden 2012) was developed. This metric computes the probability that the actual physical prototype will meet its benchmark acceptance tests, based on virtual prototype behavior simulations with known confidence and verified model assumptions. To rigorously and efficiently compute PCC it is important to account for all sources of uncertainty in a scalable manner, including model verification and behavior simulation accuracy and precision, manufacturing tolerances, context uncertainty, human factors and confidence in the stochastic sampling itself. The PCC metric was developed as part of the DARPA Adaptive Vehicle Make program (DARPA 2012) and is deployed during the design of the Fast Adaptive Next-Generation Ground Vehicle. To illustrate how PCC is deployed in practice, it was applied it to the verification of a Modelica™ model that computes the fuel efficiency of a Hybrid Electrical Vehicle with a simulated driving cycle. A Modelica™ verification package was developed to enable model authors to verify that the modelled or simulated behavior stays within the intended verification bounds and to capture the effects of simulated behavior uncertainty. We will show that the consistent use of such tools has to potential to address the problem of trust in simulation results through a "correct-by-simulation" verification of complex assemblies of behavior models. An automated simulation framework was used to generate the stochastic dynamic behavior samples that are used to compute the PCC metric. This simulation framework submits hundreds of randomized configuration samples to a HPC cluster for efficient calculation of the PCC metric with a known confidence interval. To address ease of use concerns with respect to this complex stochastic simulation process, we deployed a dynamic simulation process flow whereby PCC metrics could be flexibly defined using spreadsheets for any simulation model. This approach removed the need for the end-user to understand the detailed workings of this complex tool. When requirements are certified with deterministic behavior simulations, t twice as many physical prototype cycles may be needed as compared to a product certified with a high PCC (PCC>0.9). Thus PCC is an effective way to reduce overall development cost as well as decrease time-to-market.