Monday, 23 October 2017

Parallel Execution; Speeding up your test execution

A universal truth for all automated tests is that they always need to execute faster, and parallel execution is a way to easily achieve this, and potentially your test run being as fast as your slowest test.

However, in various test framework libraries there is confusion of what parallelism actually is;

Execute the same test multiple times against different browser/os versions

Or..

Execute different tests at the same time against the same browser/os version

While the first statement is useful, it does not speed up execution especially when results are required within a CI/CD process.  The 2nd statement is what you want to achieve.

To be able to achieve parallelism in the future you need to think about it before you even write your first test;


  • Select your test framework library with parallelism in mind; 
    • Does it easily support parallel execution? 
    • Will it collate results across threads? 
    • How much effort do you need to invest to make it execute in parallel?
  • Write each test with the understanding that it could execute the same time as any other test you write; 
    • Do tests share data?
    • Does my test rely on state that cannot be guaranteed? 
    • It may take longer to write the test but the return of parallelism far out weighs the upfront effort!
  • Once you have 2 tests, execute them in parallel.  With overheads of setting up multiple threads, it may be that parallelism of just 2 tests has little if any gain but you will soon have more tests and it is far easier to find issues in parallelism when you have small number of tests then trying for the first time once you have 1000 tests!

Finally, another common point of confusion is how offerings such as Selenium Grid and Saucelabs provide parallelism.  The short answer is they don't! If you have lots of UI tests and have grid with the ability execute many tests at the same time, you still have to make your tests run in parallel to take advantage.

lots of tests + selenium grid != parallelism 

lots of tests + ability to execute in parallel + selenium grid == superfast test execution!

No comments:

Post a Comment