Patent Issued for Retrying failed test cases in software testing using parallel threads (USPTO 11720482): State Farm Mutual Automobile Insurance Company

2023 AUG 28 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- From Alexandria, Virginia, NewsRx journalists report that a patent by the inventors Chauhan, Shaktiraj (Normal, IL, US), Shepherd, Nate (Bloomington, IL, US), filed on May 28, 2021, was published online on August 8, 2023.

The patent’s assignee for patent number 11720482 is State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States).

News editors obtained the following quote from the background information supplied by the inventors: “During development of a software application, software developers can prepare tests to verify that the software application operates as expected. Such tests may be associated with unit testing that tests the functionality of a relatively small piece of code, and/or integration testing that tests how multiple pieces of code interact. Because individual tests may be designed to test one or more relatively small aspects of the overall software application, software developers may create numerous tests associated with the entire software application.

“Some tests associated with the software application may build on one another. For instance, a first test may create a record in a database, and a second test may be designed to further access or edit the database record created by the first test. In this situation, a developer may write code for the second test under the assumption that the first test will already have executed and created the record in the database by the time the second test executes. Accordingly, in this situation, the second test may be dependent on data created by the first test.

“In some situations, when a new version of the software application is tested, it can take a relatively long period of time to execute all of the tests and verify that the new version of the software application successfully passes all of the tests. For instance, tests can be written to build on each other in a sequential order as discussed above, and running a large set of tests in a sequence can take a relatively long period of time. As a non-limiting example, tests for a software application may be written in a set of 7000 class files that each include numerous functions, and it may take up to two hours to sequentially execute all of the tests in the 7000 class files.

“The time it takes to sequentially execute a large set of tests can pose challenges and introduce delays during software development, particularly in situations in which multiple developers are working on the same software application. As an example, a team of software developers may generate sixty new builds of the software application in a single day. If each build is tested using a set of 7000 class files that takes up to 120 minutes of server time to execute, the testing may take up to 7200 minutes of server time in a single day.

“As another example, two developers may check out code from a main code branch and independently make changes to the code for the software application. If the first developer submits updated code for testing, it may take two hours to run a full set of test cases on the first developer’s updated code. However, by the time the testing verifies that the first developer’s changes have passed the full set of test cases and can be incorporated back into the main code branch, the second developer may separately have submitted different code changes for testing. The code changes made by the second developer may be incompatible with the changes made by the first developer. Accordingly, even if the second developer’s changes also independently pass the full set of test cases, it may not be possible to incorporate those changes back into the main code branch due to incompatible changes already made by the first developer. If testing of the first developer’s code changes had been performed more quickly, the second developer could have been made aware of the first developer’s changes and accounted for them before the second developer submitted additional code changes for testing.

“Accordingly, it can be desirable to run a full set of tests on updated code for a software application more quickly than described above. One possibility for speeding up the testing process is to run different tests in parallel at the same time. For example, rather than running tests within a full set of 7000 class files in sequence, the tests can be divided into smaller sets that can be run simultaneously in two or more parallel threads. Although executing the full set of tests in sequence may take up to two hours, running different subsets of the tests simultaneously in parallel threads may allow the full set of tests to complete in 30 minutes or less. Developers can thus be notified whether their code changes pass the full set of tests more quickly, and allow code changes that have passed the full set of tests to be merged into a main code branch more quickly. In addition to executing a full set of tests more quickly for a single new build of a software application, running tests in parallel threads can also reduce overall usage of server time when multiple new builds of the software application are tested. For instance, if testing each build of a software application takes up to 120 minutes when tests are executed sequentially in a single thread, it can take up to 7200 minutes of server time to test sixty different builds in a day. If running the tests in parallel reduces the testing time for each build down to 30 minutes as discussed above, testing sixty builds in a day may instead take only 1800 minutes of server time. However, although executing tests in parallel can speed up testing of a software application, in some cases the parallelization itself can cause errors and/or failed tests during testing.

“For example, as discussed above, tests may be written under the assumption that the tests will build on one another in a sequential order. Code changes may pass such tests if the tests are indeed executed sequentially in the originally intended order. However, the same code changes may not pass the same tests if the tests are executed out of order, for instance because related or dependent tests are executed out of order in different parallel threads.

“As an example, multiple tests may attempt to access the same table of a database. The database may be configured to at least briefly lock the table when the table is accessed by a code element, such that the database table cannot be changed by other code elements while the table is locked. If a set of tests are run sequentially in an intended order, a first test may have finished accessing a database table, and the table can thus be unlocked, by the time a second test later attempts to access the database table. The second test may succeed in this scenario due to the unlocked database table. However, if the tests are instead run in parallel as described above, there is a chance that different tests, executing simultaneously in different parallel threads, may attempt to access the same database table at essentially the same time. This can cause database conflicts, table locking errors, application or service errors, and/or failure of one or more of the tests. For instance, if a database table is locked due to a first test accessing the database table, the table may still be locked if a second test, executing in parallel with the first test, also attempts to access the database table. The database table locking can thus cause the second test to fail, even though the second test may have succeeded if it had been executed at a different time when the database table was not locked due to another test executing in parallel.

“As another example, tests may be written such that a first test creates data in a database, and a second test accesses or edits the data that the first test created. If these two tests are executed in sequence, the first test can have successfully created the data in the database by the time the second test attempts to access that data in the database. However, if the tests are instead run in parallel, there is a chance that the second test will execute in one parallel thread earlier than the first test executes in another parallel thread. Accordingly, if the second test executes earlier than the first test due to parallelization, the second test may fail because database data used in the second test has not yet been created by the first test.

“The example systems and methods described herein may be directed toward mitigating or overcoming one or more of the deficiencies described above.”

As a supplement to the background information on this patent, NewsRx correspondents also obtained the inventors’ summary information for this patent: “Described herein are systems and methods for testing a software application using a set of test cases distributed among in parallel threads, including re-executing any test cases that initially fail. Database errors may occur because related test cases execute in different parallel threads in a different order than they would have been executed in a single thread. Such database errors caused by the parallel execution of test cases may cause individual test cases to fail, even if the test cases would have succeeded if executed sequentially in a single thread. However, failed test cases can be tracked and re-executed. Re-executing such failed test cases can increase the chances that the overall set of test cases ultimately succeeds, as many test failures caused by the parallelization itself may not reoccur when the test cases are re-executed at a different time.

“According to a first aspect, a computer-implemented method can include identifying, by one or more processors of a computing device, a plurality of test cases that test interactions between a software application and a database. The method can also include distributing, by the one or more processors, individual test cases of the plurality of test cases among a plurality of parallel threads on the computing device, and executing, by the one or more processors, the plurality of test cases in the plurality of parallel threads. The method can further include determining, by the one or more processors, one or more failed test cases of the plurality of test cases based on the executing. The method can also include generating, by the one or more processors, at least one test failure log. The at least one test failure log can indicate the one or more failed test cases and failure codes corresponding to the one or more failed test cases. The method can also include re-executing, by the one or more processors, the one or more failed test cases. The method can additionally include collecting, by the one or more processors, a plurality of test results associated with the test cases distributed to the plurality of parallel threads. The plurality of test results can indicate whether the plurality of test cases ultimately succeeded during at least one of the executing or the re-executing. The method can also include aggregating, by the one or more processors, the plurality of test results.

“According to a second aspect, a computing device can include one or more processors and memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more computing devices to perform operations. The operations can include identifying a plurality of test cases that test interactions between a software application and a database, distributing individual test cases of the plurality of test cases among a plurality of parallel threads on the computing device, and executing the plurality of test cases in the plurality of parallel threads. The operations can also include determining one or more failed test cases of the plurality of test cases based on the executing, identifying failure codes associated with the one or more failed test cases, and re-executing the one or more failed test cases based on the failure codes. The operations can also include aggregating test results associated with the executing and the re-executing.

“According to a third aspect, one or more non-transitory computer-readable media can store computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations can include identifying a plurality of test cases that test interactions between a software application and a database, distributing individual test cases of the plurality of test cases among a plurality of test sets, and executing different test sets of the plurality of test sets simultaneously in different parallel threads. The operations can also include determining one or more failed test cases of the plurality of test cases based on the executing, generating least one test failure log indicating the one or more failed test cases and failure codes corresponding to the one or more failed test cases, and re-executing the one or more failed test cases. The operations can further include determining whether the plurality of test cases ultimately succeeded, by aggregating a plurality of test results associated with the executing and the re-executing.”

The claims supplied by the inventors are:

“1. A computer-implemented method, comprising: identifying, by one or more processors of a computing device, a plurality of test cases that test interactions between a software application and a database; distributing, by the one or more processors, individual test cases, of the plurality of test cases, among a plurality of parallel threads on the computing device; executing, by the one or more processors, the plurality of test cases in the plurality of parallel threads; identifying, by the one or more processors, failed test cases, of the plurality of test cases, based on the executing; generating, by the one or more processors, at least one test failure log, the at least one test failure log indicating the failed test cases and failure codes corresponding to the failed test cases, wherein the failure codes indicate causes of failures of the failed test cases; determining, by the one or more processors, and based on the failure codes, an execution order for re-execution of the failed test cases, wherein: the execution order is arranged to reduce likelihoods of the causes of the failures, associated with the failure codes, reoccurring during the re-execution of the failed test cases; re-executing, by the one or more processors, the failed test cases based on the execution order; collecting, by the one or more processors, a plurality of test results associated with the test cases distributed to the plurality of parallel threads, wherein the plurality of test results indicates whether the plurality of test cases ultimately succeeded during at least one of the executing or the re-executing; and aggregating, by the one or more processors, the plurality of test results.

“2. The computer-implemented method of claim 1, wherein the plurality of test cases comprises a set of classes, individual classes of the set of classes comprise one or more methods, and the individual test cases are distributed at one or more of a class level or a method level.

“3. The computer-implemented method of claim 1, wherein: the failed test cases comprise: first test cases associated with a first failure code of the failure codes, and second test cases associated with a second failure code of the failure codes, the execution order causes the one or more processors to: re-execute the first test cases substantially immediately following failure of the first test cases based on the first failure code, and re-execute the second test cases on a delayed basis after a period of time following failure of the second test cases based on the second failure code.

“4. The computer-implemented method of claim 1, further comprising: assigning, by the one or more processors, the failed test cases to at least one retry test set; and re-executing, by the one or more processors, the failed test cases in the at least one retry test set based on the execution order.

“5. The computer-implemented method of claim 4, wherein: the at least one retry test set includes a single retry test set, and re-executing the failed test cases comprises executing the failed test cases, in the single retry test set, in sequence in a single thread based on the execution order.

“6. The computer-implemented method of claim 4, wherein: the at least one retry test set includes a plurality of retry test sets associated with the plurality of parallel threads, and the method further comprises assigning, by the one or more processors, the failed test cases among the plurality of retry test sets.

“7. The computer-implemented method of claim 6, wherein determining the execution order comprises determining, by the one or more processors, different execution orders associated with different retry test sets, of the plurality of retry test sets, based on one or more of the failure codes or an original structure of the plurality of test cases.

“8. The computer-implemented method of claim 7, wherein determining the different execution orders comprises identifying, by the one or more processors, relationships between test cases in individual retry test sets, of the plurality of retry test sets, using a machine learning model.

“9. A computing device, comprising: one or more processors; memory storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: identifying a plurality of test cases that test interactions between a software application and a database; distributing individual test cases, of the plurality of test cases, among a plurality of parallel threads on the computing device; executing the plurality of test cases in the plurality of parallel threads; identifying failed test cases, of the plurality of test cases, based on the executing; identifying failure codes associated with the failed test cases, wherein the failure codes indicate causes of failures of the failed test cases; determining, based on the failure codes, an execution order for re-execution of the failed test cases, wherein the execution order is arranged to reduce likelihoods of the causes of the failures, associated with the failure codes, reoccurring during the re-execution of the failed test cases; re-executing the failed test cases based on the execution order; and aggregating test results associated with the executing and the re-executing.

“10. The computing device of claim 9, wherein the plurality of test cases comprises a set of classes, individual classes of the set of classes comprise one or more methods, and the individual test cases are distributed at one or more of a class level or a method level.

“11. The computing device of claim 9, wherein: a failure code, of the failure codes, indicates that a particular failed test case failed due to a record-locking error or a table-locking error associated with the database, and the execution order indicates a substantially immediate re-execution of the particular failed test case, following failure of the particular failed test case, based on the failure code.

“12. The computing device of claim 9, wherein: a failure code, of the failure codes, indicates that a particular failed test case failed due to a missing data error associated with the database, and re-executing the failed test cases comprises: adding the particular failed test case to a test failure log; and re-executing the particular failed test case, based on the execution order, as part of a retry test set that includes other failed test cases.

“13. The computing device of claim 9, wherein re-executing the failed test cases is performed as part of a retry test set executed in a single thread.

“14. The computing device of claim 9, wherein re-executing the failed test cases comprises: distributing the failed test cases among a plurality of retry test sets associated with the plurality of parallel threads; and executing the plurality of retry test sets in the plurality of parallel threads.

“15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: identifying a plurality of test cases that test interactions between a software application and a database; distributing individual test cases of the plurality of test cases among a plurality of test sets; executing different test sets of the plurality of test sets simultaneously in different parallel threads; determining failed test cases, of the plurality of test cases, based on the executing; determining failure codes corresponding to the failed test cases, wherein the failure codes indicate causes of failures of the failed test cases; re-executing first failed test cases, associated with first failure codes of the failure codes, substantially immediately following failure of the first failed test cases; re-executing second failed test cases, associated with second failure codes of the failure codes, on a delayed basis after a period of time following failure of the second failed test cases; and determining whether the plurality of test cases ultimately succeeded, by aggregating a plurality of test results associated with the executing of the different test sets and the re-executing of the first failed test cases and the second failed test cases.

“16. The one or more non-transitory computer-readable media of claim 15, wherein re-executing the second failed test cases comprises: assigning the second failed test cases to at least one retry test set; and re-executing the at least one retry test set.

“17. The one or more non-transitory computer-readable media of claim 16, wherein the operations further comprise determining an execution order, based on the second failure codes, for the second failed test cases in the at least one retry test set.

“18. The one or more non-transitory computer-readable media of claim 16, wherein: the at least one retry test set includes a plurality of retry test sets associated with the different parallel threads, and the operations further comprise assigning the second failed test cases among the plurality of retry test sets.

“19. The computer-implemented method of claim 1, wherein: a failed test case, of the failed test cases, is associated with a failure code, of the failure codes, indicating that a missing data error caused the failed test case to fail; and the execution order is arranged, based on the failure code associated with the missing data error, to delay re-execution of the failed test case for a least a threshold period of time following failure of the failed test case to reduce a likelihood of a reoccurrence of the missing data error during re-execution of the failed test case.”

There are additional claims. Please visit full patent to read further.

For additional information on this patent, see: Chauhan, Shaktiraj. Retrying failed test cases in software testing using parallel threads. U.S. Patent Number 11720482, filed May 28, 2021, and published online on August 8, 2023. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11720482)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)