Marin Kobilarov, Johns Hopkins Whiting School of Engineering
Paul Stankiewicz, Johns Hopkins Applied Physics Laboratory

This research applies machine-learning techniques to advance successful performance of autonomy software and assure users of the technology. The research maps and compares failure regions between two versions of autonomy software, enabling test engineers to ensure that more “mature” software releases reduce the failure space and do not produce new, unexpected failure modes.

Researchers first employ adaptive simulation-based testing and dimensionality-reduction techniques, then analyze the two testing sets to identify performance changes that may indicate where a subsequent version of software may increase the failure space over the previous version. This holistic performance testing will complement the traditional practice of unit-based software testing.

This research will deliver software to perform the performance regression and a proof-of-concept of the algorithm methodology to be shared publicly. This approach promises to advance the state of the art in this technology by assuring users and communities that new versions of autonomous systems do not introduce unexpected loss of capabilities.

Figure 1: Performance landscapes for two versions of unmanned surface vehicle (USV) planning software simulated over different test conditions. Two regions (red and pink) are highlighted between the data sets of 3,000 simulations. The red failure region exists in version #1 but is fixed based on software changes made for version #2. The pink failure region, however, does not exist in version #1 but appears as a new failure mode unexpectedly introduced by the software changes in version #2.