Google Play icon

Lockheed Martin Study: NIST Computational Tool Improves Product Testing and Saves Money

Share
Posted June 11, 2015

Lockheed Martin, the global security and aerospace company, estimates that widely used software testing methods developed at the National Institute of Standards and Technology (NIST) can trim test planning and design costs by up to 20 percent, while greatly improving the thoroughness of product and system testing during development.

Using NIST's software testing system, Lockheed Martin reported that it can cut test planning and design costs by up to 20 percent while improving the quality of product testing. The F-16 was one of the products involved in the research. Image credit: DoD photo by Senior Airman Greg L. Davis, U.S. Air Force

Using NIST’s software testing system, Lockheed Martin reported that it can cut test planning and design costs by up to 20 percent while improving the quality of product testing. The F-16 was one of the products involved in the research. Image credit: DoD photo by Senior Airman Greg L. Davis, U.S. Air Force

The projected benefits are based on the results of the company’s two-year pilot study of combinatorial testing, as reported in an article in IEEE Computer.* The study was carried out under a Cooperative Research and Development Agreement (CRADA) with NIST.

In contrast to creating test cases manually, combinatorial testing is a comprehensive “trouble shooting” tool for rooting out faulty code, calculation errors, unanticipated interactions and other causes of software failures. Used during the early stages of product development, the computational tool can detect faults that might otherwise linger until late-stage testing or, perhaps, slip through entirely.

Corrective rework to fix late-detected software errors can wreck project schedules and may be the costliest part of the development process, especially for complex products and systems.

Developed with collaborators from the University of Texas at Arlington, NIST’s Advanced Combinatorial Testing System (ACTS) “uses proven mathematical techniques to greatly reduce the number of tests a company needs to perform to ensure the quality of a product or process,” explains NIST computer scientist Richard Kuhn.

ACTS generates a plan for testing combinations of two to six variables that can interact and cause errors.

While studying software crashes of medical devices and Web browsers, Kuhn and colleagues determined that between 70 and 95 percent of software failures are triggered by interactions between only two variables. Nearly all software failures are triggered by no more than six variables.

“For example, all six-way combinations of 34 switches could be tested with only 522 tests instead of 17 billion for all possible combinations.” said Raghu Kacker, a NIST mathematical statistician.

In one project, combinatorial testing was compared with conventional methods for testing commercial Web browser software. NIST used the method to find 100 percent of flaws using less than 5 percent of the original number of tests.

During Lockheed Martin’s two-year study to assess the merits of the software testing approach, the company chose ACTS as its primary tool, which it supplemented with other combinatorial testing packages. In all, the tool’s utility was evaluated in eight pilot development projects, including an electronic warfare system, the redesign of a fin on a fighter jet, and an evaluation of aircraft engine failure modes.

“Lockheed Martin’s initial estimate is that combinatorial testing and supporting technology can save up to 20 percent in test planning and design costs if used early on in a program and can increase test coverage by 20 to 50 percent,” company and NIST researchers wrote in the Computer article.

“Our experience showed that the combinatorial testing technique is maturing,” says Thomas L. Wissink, director of integration, test, and evaluation at Lockheed Martin. “We are continuing to use combinatorial testing at Lockheed Martin and are planning to extend its use through the company.”

NIST also is building on lessons learned from this collaboration. Knowledge and insights gained are guiding NIST research to further improve measurement and testing capabilities focused on ensuring the quality, safety and reliability of software and systems.

The ACTS tool and tutorial are publicly available.

Source: NIST

Featured news from related categories:

Technology Org App
Google Play icon
83,394 science & technology articles

Most Popular Articles

  1. Bright Fireball Explodes Over Ontario, Meteorite Fragments Might Have Reached the Ground (August 5, 2019)
  2. Why older people smell the way they do? Japanese have even a special word for it (August 4, 2019)
  3. Terraforming the Surface of Mars with Silica Aerogel? (July 23, 2019)
  4. Moisturizers May Be Turning Your Skin Into ‘Swiss Cheese’ (4 days old)
  5. Swarm Autonomy Tested in Second Major DARPA OFFSET Field Experiment (August 8, 2019)

Follow us

Facebook   Twitter   Pinterest   Tumblr   RSS   Newsletter via Email