Patent Issued for Mutation testing in parallel threads (USPTO 11797429): State Farm Mutual Automobile Insurance Company

2023 NOV 14 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States) has been issued patent number 11797429, according to news reporting originating out of Alexandria, Virginia, by NewsRx editors.

The patent’s inventors are Pearson, Andrew L (Normal, IL, US), Shepherd, Nate (Bloomington, IL, US).

This patent was filed on September 27, 2021 and was published online on October 24, 2023.

From the background information supplied by the inventors, news correspondents obtained the following quote: “During development of a software application, software developers can prepare test cases to verify that the software application operates as expected. Such test cases may be associated with unit testing that tests the functionality of a relatively small piece of code, and/or integration testing that tests how multiple pieces of code interact. Because individual test cases may be designed to test one or more relatively small aspects of the overall software application, software developers may create numerous tests associated with the entire software application.

“Software developers may also perform mutation testing to verify that a suite of test cases adequately tests the software application. Mutation testing may involve making a small code change to create an application mutant that includes an intentional error or fault. For example, an application mutant can be created in part by changing an “and” logical operator in a line of code to an “or” logical operator. The suite of test cases can then be executed against the application mutant. If test cases fail due to the code change, the application mutant is considered to be “killed.” This may indicate that the suite of test cases has been properly designed to detect errors in the software application. However, if test cases succeed despite the code changes that introduced faults into the application mutant, such that the application mutant is not “killed,” the success of the test cases may indicate that the suite of test cases has not been designed to fully test errors in the software application. For instance, the suite of test cases may include tests cases that are not actually testing what they are intended to test, and/or may not include test cases that test for errors or situations corresponding to the faults that were introduced into the application mutant. In some examples, a quality level of a suite of test cases can be determined based on a percentage of application mutants that the suite of test cases kill.

“Some tools have been developed that can automatically perform mutation testing by creating application mutants and executing a suite of test cases against the application mutants. However, many such tools rely on initially compiling source code for the software application into bytecode that can be executed by a virtual machine, and then editing lines of the compiled bytecode to create the application mutants. For example, some Java® testing tools are configured to compile Java® source code class files into corresponding Java® bytecode class files, and then edit lines of the already-compiled Java® bytecode class files in various ways to create different application mutants. Accordingly, such tools can create and test application mutants based on changes to already-compiled bytecode that is directly executable by a virtual machine.

“However, such existing testing tools and mutation testing approaches may be inefficient and/or impractical for some types of programming languages. For example, source code for a software application may be written in the Gosu® programming language. Similar to Java®, Gosu® source code class files can be compiled into Java® bytecode class files that can be directly executed by a virtual machine. However, unlike Java®, Gosu® compilers may generate multiple Java® bytecode class files from a single Gosu® source code class file. For instance, compiling one Gosu® source code class file may produce five corresponding Java® bytecode class files.

“A set of multiple compiled Java® bytecode class files associated with a single Gosu® source code class file may include code that is fragmented and/or disassociated with the original structure of the Gosu® source code class file. Accordingly, it may be difficult or impossible to reconstruct the original Gosu® source code class file from a corresponding set of multiple compiled Java® bytecode class files. It may also be difficult or impossible to determine, after compilation, how a change to the source code class file affected code in multiple compiled Java® bytecode class files.

“As noted above, existing mutation testing tools may be configured to generate an application mutant by making a code change to an already-compiled bytecode class file that corresponds to a change that could have been to a corresponding source code class file before compilation. However, because Gosu® compilers may produce multiple bytecode class files for each individual Gosu® source code class file, it may be difficult or impossible to identify changes to one or more of the multiple compiled bytecode class files that would have been equivalent to a small code change to the Gosu® source code class file before compilation. Therefore, it may also be difficult or impossible for existing mutation testing tools to determine which changes to make to bytecode class files to produce application mutants, after Gosu® source code class files have been compiled into the bytecode class files. Accordingly, existing mutation testing tools that rely on modifying bytecode may not be able to perform mutation testing associated with software applications written in Gosu® or other programming languages, such as the Scala programming language, in which compilers produce bytecode that has a different structure or number or class files than the original source code.

“The example systems and methods described herein may be directed toward mitigating or overcoming one or more of the deficiencies described above.”

Supplementing the background information on this patent, NewsRx reporters also obtained the inventors’ summary information for this patent: “Described herein are systems and methods for executing mutation testing via multiple parallel threads. Mutation testing can indicate whether application mutants, created by altering source code of a software application, are successfully “killed” by test cases executed against the application mutants. Different parallel threads may modify individual source code class files, and recompile the modified class files to generate and test different application mutants. Accordingly, different application mutants can be tested concurrently in different parallel threads. The parallel threads can also produce individual mutation test results, which can be aggregated into an aggregated test result report that indicates overall testing metrics associated with the mutation testing across the parallel threads.

“According to a first aspect, a computer-implemented method can include initiating, by one or more processors of one or more computing devices, a plurality of parallel threads. The method can also include identifying, by the one or more processors, and in different threads of the plurality of parallel threads, mutatable source code class files associated with a software application. The method can additionally include modifying, by the one or more processors, and in the different threads, the mutatable source code class files based on at least one mutation configuration. The method can further include compiling, by the one or more processors, and in the different threads, the mutatable source code class files into compiled bytecode. The method can also include executing, by the one or more processors, and in the different threads, a plurality of test cases against application mutants that include the compiled bytecode.

“According to a second aspect, a computing device can include one or more processors and memory. The memory can store application source code, associated with a software application, comprising a plurality of class files. The memory can also store one or more mutation configurations. The memory can further store computer executable instructions associated with a mutation test manager that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations can include initiating a plurality of threads, and distributing the plurality of class files among the plurality of parallel threads. The operations can also include identifying, in the plurality of parallel threads, mutatable class files, of the plurality of class files, based on the one or more mutation configurations. The operations can also include modifying, in the plurality of parallel threads, the mutatable class files based on the one or more mutation configurations. The operations can also include compiling, in the plurality of parallel threads, the mutatable class files into compiled bytecode, and executing, in the plurality of parallel threads, at least a portion of the plurality of test cases against a plurality of application mutants that include the compiled bytecode.

“According to a third aspect, one or more non-transitory computer-readable media can store computer-executable instructions that, when executed by one or more processors, cause the one or more processors to initiate a first parallel thread and a second parallel threads, and to perform, substantially concurrently, first parallel thread operations associated with the first parallel thread and second parallel thread operations associated with the second parallel thread. The first parallel thread operations can include identifying a first mutatable source code class file associated with a software application, modifying the first mutatable source code class file based on at least one mutation configuration, compiling the first mutatable source code class file into one or more first compiled bytecode class files, and executing a plurality of test cases against a first application mutant that includes the one or more first compiled bytecode class files. The second parallel thread operations can include identifying a second mutatable source code class file associated with the software application, modifying the second mutatable source code class file based on the at least one mutation configuration, compiling the second mutatable source code class file into one or more second compiled bytecode class files, and executing the plurality of test cases against a second application mutant that includes the one or more second compiled bytecode class files.”

The claims supplied by the inventors are:

“1. A computer-implemented method, comprising: initiating, by one or more processors of one or more computing devices, a plurality of parallel threads; identifying, by the one or more processors, and in different threads of the plurality of parallel threads, mutatable source code class files associated with a software application; generating, by the one or more processors, and in the different threads, mutated source code class files by modifying the mutatable source code class files based on at least one mutation configuration; compiling, by the one or more processors, and in the different threads, the mutated source code class files into a different amount of compiled bytecode class files; and executing, by the one or more processors, and in the different threads, a plurality of test cases against application mutants that include the compiled bytecode class files.

“2. The computer-implemented method of claim 1, wherein identifying the mutatable source code class files comprises: determining, by the one or more processors, code coverage associated with a plurality of source class files by executing the plurality of test cases against an unmodified version of the software application; and identifying, by the one or more processors, and in the different threads, source class files that: have at least a threshold level of the code coverage; and correspond to the at least one mutation configuration.

“3. The computer-implemented method of claim 1, wherein the application mutants include: previously compiled bytecode class files associated with unmodified source code class files, and the compiled bytecode class files associated with the mutated source code class files.

“4. The computer-implemented method of claim 3, wherein executing the plurality of test cases against the application mutants comprises swapping, in memory associated with the different threads, second previously compiled bytecode class files with the compiled bytecode class files without recompiling the previously compiled bytecode class files.

“5. The computer-implemented method of claim 1, wherein an individual thread, of the different threads, is associated with multiple threads and uses the multiple threads to execute different test cases, of the plurality of test cases, against a same application mutant concurrently.

“6. The computer-implemented method of claim 1, further comprising: logging, by the one or more processors, and in the different threads, mutation test results associated with executing the plurality of test cases against the application mutants; and aggregating, by the one or more processors, the mutation test results into an aggregated test result report.

“7. The computer-implemented method of claim 6, wherein the aggregated test result report includes: summary mutation testing information associated with the different threads; and mutation testing details associated with the mutatable source code class files.

“8. One or more computing devices, comprising: one or more processors; memory storing: application source code, associated with a software application, comprising a plurality of source code class files; a plurality of test cases associated with the software application; one or more mutation configurations; and computer-executable instructions associated with a mutation test manager that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: initiating a plurality of parallel threads; distributing the plurality of source code class files among the plurality of parallel threads; identifying, in the plurality of parallel threads, mutatable source code class files, of the plurality of source code class files, based on the one or more mutation configurations; generating, in the plurality of parallel threads, mutated source code class files by modifying the mutatable source code class files based on the one or more mutation configurations; compiling, in the plurality of parallel threads, the mutated source code class files into a different amount of compiled bytecode class files; and executing, in the plurality of parallel threads, at least a portion of the plurality of test cases against a plurality of application mutants that include the compiled bytecode class files.

“9. The one or more computing devices of claim 8, wherein identifying the mutatable source code class files comprises: identifying source code class files, of the plurality of source code class files, that include at least a threshold percentage of source code lines that are covered by the plurality of test cases; and determining that the source code lines of the source code class files correspond to at least one mutation configuration of the one or more mutation configurations.

“10. The one or more computing devices of claim 8, wherein: the plurality of source code class files is organized into a plurality of class file packages, distributing the plurality of source code class files comprises distributing the plurality of class file packages among the plurality of parallel threads, and identifying the mutatable source code class files comprises identifying at least one mutatable source code class file within a set of class file packages associated with an individual parallel thread.

“11. The one or more computing devices of claim 8, wherein the plurality of application mutants include: previously compiled bytecode class files associated with unmodified source code class files of the plurality of source code class files, and the compiled bytecode class files associated with the mutated source code class files.

“12. The one or more computing devices of claim 11, wherein executing the at least the portion of the plurality of test cases against the plurality of application mutants comprises swapping, in the memory, second previously compiled bytecode class files with the compiled bytecode class files, without recompiling the previously compiled bytecode class files.

“13. The one or more computing devices of claim 8, wherein an individual thread, of the plurality of parallel threads, is associated with multiple threads and is configured to use the multiple threads to execute different test cases, of the plurality of test cases, against a same application mutant concurrently.

“14. The one or more computing devices of claim 8, wherein the operations further comprise: logging a plurality of mutation test results associated with executing the at least the portion of the plurality of test cases against the plurality of application mutants in the plurality of parallel threads; and aggregating the plurality of mutation test results into an aggregated test result report.

“15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to: initiate a first parallel thread and a second parallel thread; and perform, substantially concurrently: first parallel thread operations, associated with the first parallel thread, comprising: identifying a first mutatable source code class file associated with a software application; generating a first mutated source code class file by modifying the first mutatable source code class file based on at least one mutation configuration; compiling the first mutated source code class file into two or more first compiled bytecode class files; and executing a plurality of test cases against a first application mutant that includes the two or more first compiled bytecode class files; and second parallel thread operations, associated with the second parallel thread, comprising: identifying a second mutatable source code class file associated with the software application; generating a second mutated source code class file by modifying the second mutatable source code class file based on the at least one mutation configuration; compiling the second mutated source code class file into two or more second compiled bytecode class files; and executing the plurality of test cases against a second application mutant that includes the two or more second compiled bytecode class files.

“16. The one or more non-transitory computer-readable media of claim 15, wherein identifying the first mutatable source code class file comprises: determining that at least a threshold percentage of source code lines in the first mutatable source code class file is covered by the plurality of test cases; and determining that the source code lines of the first mutatable source code class file correspond to the at least one mutation configuration.

“17. The one or more non-transitory computer-readable media of claim 15, wherein the first application mutant comprises: previously compiled bytecode class files associated with unmodified source code class files associated with the software application; and the two or more first compiled bytecode class files associated with the first mutated source code class file.

“18. The one or more non-transitory computer-readable media of claim 17, wherein executing the plurality of test cases against the first application mutant comprises swapping, in memory associated with the first parallel thread, second previously compiled bytecode class files with the two or more first compiled bytecode class files without recompiling the previously compiled bytecode class files.

“19. The one or more non-transitory computer-readable media of claim 15, wherein the first parallel thread is associated with multiple threads and is configured to use the multiple threads to execute different test cases, of the plurality of test cases, against the first application mutant concurrently.”

There are additional claims. Please visit full patent to read further.

For the URL and additional information on this patent, see: Pearson, Andrew L. Mutation testing in parallel threads. U.S. Patent Number 11797429, filed September 27, 2021, and published online on October 24, 2023. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11797429)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)