Patent Issued for Test conflict guard for parallel software testing (USPTO 11816023): State Farm Mutual Automobile Insurance Company

2023 DEC 05 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- From Alexandria, Virginia, NewsRx journalists report that a patent by the inventors Chauhan, Shaktiraj (Normal, IL, US), Shepherd, Nate (Bloomington, IL, US), filed on May 28, 2021, was published online on November 14, 2023.

The patent’s assignee for patent number 11816023 is State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States).

News editors obtained the following quote from the background information supplied by the inventors: “During development of a software application, software developers can prepare tests to verify that the software application operates as expected. Such tests may be associated with unit testing that evaluates the functionality of a relatively small piece of code, and/or integration testing that evaluates how multiple pieces of code interact. Because individual tests may be designed to test one or more relatively small aspects of the overall software application, software developers may create numerous tests associated with the entire software application.

“In some situations, when a new version of the software application is tested, it can take a relatively long period of time to execute all of the tests and verify that the new version of the software application successfully passes all of the tests. As a non-limiting example, tests for a software application may be written in a set of 7000 class files that each include numerous functions, and it may take up to two hours to sequentially execute all of the tests in the 7000 class files.

“The time it takes to sequentially execute a large set of tests can pose challenges and introduce delays during software development, particularly in situations in which multiple developers are working on the same software application. As an example, a team of software developers may generate sixty new builds of the software application in a single day. If each build is tested using a set of 7000 class files that takes up to 120 minutes of server time to execute, the testing may take up to 7200 minutes of server time in a single day.

“As another example, two developers may check out code from a main code branch and independently make changes to the code for the software application. If the first developer submits updated code for testing, it may take two hours to run a set of test cases on the first developer’s updated code. However, by the time the testing verifies that the first developer’s changes have passed the set of test cases and can be incorporated back into the main code branch, the second developer may separately have submitted different code changes for testing. The code changes made by the second developer may be incompatible with the changes made by the first developer. Accordingly, even if the second developer’s changes also independently pass the set of test cases, it may not be possible to incorporate those changes back into the main code branch due to incompatible changes already made by the first developer. If testing of the first developer’s code changes had been performed more quickly, the second developer could have been made aware of the first developer’s changes and accounted for them before the second developer submitted additional code changes for testing.

“Accordingly, it can be desirable to execute tests on updated code for a software application more quickly than described above. One possibility for speeding up the testing process is to run different tests in parallel at the same time. For example, rather than running tests within a set of 7000 class files in sequence, the tests can be divided into smaller sets that can be run simultaneously in two or more parallel threads. Although executing the set of tests in sequence may take up to two hours, running different subsets of the tests simultaneously in parallel threads may allow the set of tests to complete in 30 minutes or less. Developers can thus be notified whether their code changes pass the set of tests more quickly, and allow code changes that have passed the set of tests to be merged into a main code branch more quickly. In addition to executing tests more quickly for a single new build of a software application, running tests in parallel threads can also reduce overall usage of server time when multiple new builds of the software application are tested. For instance, if testing each build of a software application takes up to 120 minutes when tests are executed sequentially in a single thread, it can take up to 7200 minutes of server time to test sixty different builds in a day. If running the tests in parallel reduces the testing time for each build down to 30 minutes as discussed above, testing sixty builds in a day may take only 1800 minutes of server time. However, although parallelization can speed up testing of a software application, in some cases the parallelization itself can cause errors and/or failed tests during testing. In particular, parallelization may lead to database errors or other testing errors if tests are designed to build on one another and/or use the same database data, and such related or dependent tests are executed in different parallel threads instead of sequentially in the same threads.

“As an example, related tests may attempt to access the same table of a database. The database may be configured to at least briefly lock the table when the table is accessed by a code element, such that the database table cannot be changed by other code elements while the table is locked. If a set of tests are run sequentially in an intended order, a first test may have finished accessing a database table, and the table can thus be unlocked, by the time a second test later attempts to access the database table. The second test may succeed in this scenario due to the unlocked database table. However, if the tests are instead run in parallel as described above, there is a chance that different tests, executing simultaneously in different parallel threads, may attempt to access the same database table at essentially the same time. This can cause database conflicts, table locking errors, and/or failure of one or more of the tests. For instance, if a database table is locked due to a first test accessing the database table, the table may still be locked if a second test, executing in parallel with the first test, also attempts to access the database table. The database table locking can thus cause the second test to fail, even though the second test may have succeeded if it had been executed at a different time when the database table was not locked due to another test executing in parallel.

“As another example, tests may be written such that a first test creates data in a database, and a second test accesses or edits the data that the first test created. If these two tests are executed in sequence, the first test can have successfully created the data in the database by the time the second test attempts to access that data in the database. However, if the tests are instead run in parallel, there is a chance that the second test will execute in a parallel thread earlier than the first test executes in another parallel thread. Accordingly, if the second test executes earlier than the first test due to parallelization, the second test may fail because database data used in the second test has not yet been created by the first test.

“The example systems and methods described herein may be directed toward mitigating or overcoming one or more of the deficiencies described above.”

As a supplement to the background information on this patent, NewsRx correspondents also obtained the inventors’ summary information for this patent: “Described herein are systems and methods for testing a software application using a set of independent test cases distributed among different test sets executed in different parallel threads. In particular, a test conflict guard can verify that the test cases within a set of test cases are independent test cases that use distinct data in a database, before permitting a test manager to distribute the independent test cases among different test sets to be executed in parallel. If the test conflict guard instead determines that any of the test cases are dependent and/or related test cases that use the same data in the database, the test conflict guard may block the test manager from distributing the test cases among different test sets and/or executing the test sets in parallel. By ensuring that the test cases distributed among different test sets executed in parallel are independent test cases that do not use the same database data, database and other testing errors that may occur when executing dependent or related test cases in different parallel threads can be avoided. Additionally, executing the independent test cases in parallel, instead of in sequence, can reduce testing times and reduce usages of computing resources.

“According to a first aspect, a computer-implemented method can include identifying, by one or more processors of a computing device, a plurality of test cases associated with a software application. The method can also include determining, by the one or more processors, that the plurality of test cases includes at least two related test cases configured to use common data in a database, and blocking, by the one or more processors, execution of the plurality of test cases based on determining that the plurality of test cases includes the at least two related test cases. The method can further include determining, by one or more processors based on changes to the plurality of test cases, that the plurality of test cases includes independent test cases configured to use distinct data in the database and that the plurality of test cases does not include the at least two related test cases. The method can also include unblocking, by the one or more processors, execution of the plurality of test cases based on determining that the plurality of test cases include independent test cases and does not include the at least two related test cases. The method can also include distributing, by the one or more processors, individual test cases of the plurality of test cases among a plurality of test sets, and executing, by the one or more processors, the plurality of test sets in parallel.

“According to a second aspect, a computing device can include one or more processors and memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more computing devices to perform operations. The operations can include identifying a plurality of test cases associated with a software application. The operations can also include executing a test conflict guard. The test conflict guard can be configured to block execution of the plurality of test cases based on a first determination that at least some of the plurality of test cases are related test cases configured to use common data in a database. The test conflict guard can also be configured to permit execution of the plurality of test cases based on a second determination that the plurality of test cases includes independent test cases configured to use distinct data in the database and does not include the related test cases. The operations can also include executing a test manager. The test manager can be configured to, in response to the second determination of the test conflict guard, distribute individual test cases of the plurality of test cases among a plurality of test sets, and cause execution of the plurality of test sets in parallel.

“According to a third aspect, one or more non-transitory computer-readable media can store computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations can include identifying a plurality of test cases associated with a software application, wherein the plurality of test cases comprises at least one draft test case generated via a programming tool. The operations can also include determining that the plurality of test cases includes at least two related test cases configured to use common data in a database, and causing the programming tool to display a warning message identifying the at least two related test cases. The operations can further include determining, based on changes to the plurality of test cases received in response to the warning message, that the plurality of test cases includes independent test cases configured to use distinct data in the database and does not include the at least two related test cases. The operations can also include distributing, in response to determining that the plurality of test cases includes independent test cases and does not include the at least two related test cases, individual test cases of the plurality of test cases among a plurality of test sets, and executing the plurality of test sets in parallel.”

The claims supplied by the inventors are:

“1. A computer-implemented method, comprising: identifying, by one or more processors of a computing device, a plurality of test cases associated with a software application; determining, by the one or more processors, at a first time, and by analyzing source code associated with the plurality of test cases without executing the plurality of test cases, that the plurality of test cases includes at least two related test cases configured to use common data in a database; blocking, by the one or more processors, initiation of testing comprising execution of the plurality of test cases, in response to determining at the first time that the plurality of test cases includes the at least two related test cases; determining, by the one or more processors, at a second time, and by re-analyzing the source code, without executing the plurality of test cases, in response to at least one change to the source code, that the plurality of test cases: includes independent test cases configured to use distinct data in the database; and no longer includes the at least two related test cases; unblocking, by the one or more processors, the initiation of the testing, wherein the unblocking permits the execution of the plurality of test cases in response to determining that the plurality of test cases includes the independent test cases and no longer includes the at least two related test cases; and initiating, by the one or more processors, and at a third time following the unblocking, the testing by: distributing individual test cases, of the plurality of test cases among a plurality of test sets; and executing the plurality of test sets in parallel.

“2. The computer-implemented method of claim 1, wherein determining that the plurality of test cases includes the at least two related test cases comprises: identifying, by the one or more processors, first source code, in the source code, that is associated with a first test case of the plurality of test cases; determining, by the one or more processors, and by analyzing the first source code, that the first source code references database data comprising one or more of: a table in the database, a record in the database, or a record attribute in the database; identifying, by the one or more processors, second source code in the source code, that is associated with a second test case of the plurality of test cases; and determining, by the one or more processors, and by analyzing the second source code, that the second source code references the database data referenced by the first source code.

“3. The computer-implemented method of claim 1, wherein the plurality of test cases comprises at least one draft test case generated via a programming tool that is configured to at least one of display or edit the source code.

“4. The computer-implemented method of claim 3, wherein blocking the initiation of the testing comprises causing, by the one or more processors, the programming tool to display a warning message identifying at least one portion of the source code associated with the at least two related test cases.

“5. The computer-implemented method of claim 3, wherein the plurality of test cases further comprises a set of pre-existing test cases.

“6. The computer-implemented method of claim 1, wherein: the plurality of test cases comprises a set of classes; individual classes of the set of classes comprise one or more methods; and the one or more processors initiate the testing by distributing the one or more methods among the plurality of test sets at one or more of: a class level, or a method level.

“7. The computer-implemented method of claim 1, further comprising: collecting, by the one or more processors, a plurality of test results associated with the plurality of test sets; and aggregating, by the one or more processors, the plurality of test results.

“8. The computer-implemented method of claim 1, further comprising: generating, by the one or more processors, a local build of the software application based on one or more code changes submitted locally, wherein identifying the plurality of test cases, determining that the plurality of test cases includes the at least two related test cases, blocking the initiation of the testing, determining that the plurality of test cases includes the independent test cases and no longer includes the at least two related test cases, unblocking the initiation of the testing, and initiating the testing are performed as part of a test process for the local build.

“9. The computer-implemented method of claim 1, further comprising: generating, by the one or more processors, a merge build of the software application based on one or more code changes submitted locally and based on a master code branch associated with the software application; wherein identifying the plurality of test cases, determining that the plurality of test cases includes the at least two related test cases, blocking the initiation of the testing, determining that the plurality of test cases includes the independent test cases and no longer includes the at least two related test cases, unblocking the initiation of the testing, and initiating the testing are performed as part of a test process for the merge build.

“10. The computer-implemented method of claim 1, further comprising: detecting, by the one or more processors, and based on analyzing the source code at the first time, at least one instance in the plurality of test cases of an object being instantiated in memory more than once; and causing, by the one or more processors, a programming tool to display a warning message indicating that the object is configured to be instantiated in memory more than once.

“11. One or more computing devices, comprising: one or more processors; memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more computing devices to perform operations comprising: identifying a plurality of test cases associated with a software application; executing a test conflict guard configured to: analyze source code associated with the plurality of test cases, without executing the plurality of test cases; block initiation of testing comprising execution of the plurality of test cases in response to a first determination, based on a first analysis of the source code by the test conflict guard, that at least some of the plurality of test cases are related test cases configured to use common data in a database; and unblock the initiation of the testing in response to a second determination, based on a second analysis of the source code by the test conflict guard, that the plurality of test cases includes independent test cases configured to use distinct data in the database and no longer includes the related test cases, wherein unblocking of the initiation of the testing permits the execution of the plurality of test cases; and executing a test manager configured to, based on the unblocking of the initiation of the testing by test conflict guard, initiate the testing by: distributing individual test cases, of the plurality of test cases, among a plurality of test sets; and causing the execution of the plurality of test sets in parallel.

“12. The one or more computing devices of claim 11, wherein the test conflict guard is a component of the test manager.

“13. The one or more computing devices of claim 11, wherein: the test conflict guard is a component of a programming tool configured to at least one of display or edit the source code; and the test conflict guard is configured to evaluate at least one draft test case, generated via the programming tool, as part of the plurality of test cases.

“14. The one or more computing devices of claim 11, wherein the test conflict guard is configured to evaluate at least one draft test case, received from an external programming tool, as part of the plurality of test cases.

“15. The one or more computing devices of claim 11, wherein the test conflict guard is configured to make the first determination by: identifying first source code, in the source code, associated with a first test case of the plurality of test cases; determining, by analyzing the first source code, that the first source code references database data, the database data comprising one or more of: a table in the database; a record in the database; or a record attribute in the database; identifying second source code, in the source code, associated with a second test case of the plurality of test cases; and determining, by analyzing the second source code, that the second source code references the database data referenced by the first source code.

“16. The one or more computing devices of claim 11, wherein: the plurality of test cases comprises a set of classes; individual classes of the set of classes comprise one or more methods; and the test manager is configured to initiate the testing by distributing the one or more methods among the plurality of test sets at one or more of: a class level, or a method level.”

There are additional claims. Please visit full patent to read further.

For additional information on this patent, see: Chauhan, Shaktiraj. Test conflict guard for parallel software testing. U.S. Patent Number 11816023, filed May 28, 2021, and published online on November 14, 2023. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11816023)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)