Parallel Execution; Speeding up your test execution

A universal truth for all automated tests is that they always need to execute faster, and parallel execution is a way to easily achieve this, and potentially your test run being as fast as your slowest test.

However, in various test framework libraries there is confusion of what parallelism actually is;

Execute the same test multiple times against different browser/os versions

Or..

Execute different tests at the same time against the same browser/os version

While the first statement is useful, it does not speed up execution especially when results are required within a CI/CD process.  The 2nd statement is what you want to achieve.

To be able to achieve parallelism in the future you need to think about it before you even write your first test;

• Select your test framework library with parallelism in mind; 
• Does it easily support parallel execution? 
• Will it collate results across threads? 
• How much effort do you need to invest to make it execute in parallel?
  
• Write each test with the understanding that it could execute the same time as any other test you write; 
• Do tests share data?
• Does my test rely on state that cannot be guaranteed? 
• It may take longer to write the test but the return of parallelism far out weighs the upfront effort!
  
• Once you have 2 tests, execute them in parallel.  With overheads of setting up multiple threads, it may be that parallelism of just 2 tests has little if any gain but you will soon have more tests and it is far easier to find issues in parallelism when you have small number of tests then trying for the first time once you have 1000 tests!
  

Finally, another common point of confusion is how offerings such as Selenium Grid and Saucelabs provide parallelism.  The short answer is they don't! If you have lots of UI tests and have grid with the ability execute many tests at the same time, you still have to make your tests run in parallel to take advantage.

lots of tests + selenium grid != parallelism 

lots of tests + ability to execute in parallel + selenium grid == superfast test execution!

Page objects....why not Email Objects????

I have been working on the Selenium Webdriver automation of a website which requires all new users  to confirm via an email as part of the registration flow.

This presents a challenge to test automation as we obviously want to verify the registration process post email confirmation and if we do not have a suitable API to create test users, this may be a key process for all tests to use; fortunately in my case there was an API to create users but I was still keen to prove the entire registration flow.

Once we have cracked the problem of generating new accessible email addresses for each tests such as adding "+" to the end of Gmail addresses or using a temporary provider such as Dispotable, there are a few options open to us;

Don't automate

"Hey, we have to accept that not everything can be automated, we still need  manual testers, let them do it"

I just can't let that happen! I want to automate everything ;-)

Automate an email client

"Gmail is a webpage, we can automate a web page, therefore we can automate Gmail"

Well, yes you can BUT you have even less control over this GUI then the one you are testing. How often are you going to have to maintain tests because Google decide to change the layout? Do you really want to test someone elses website? Your tests are going to be flakey enough without adding a 3rd party GUI into the mix!

So..

Having chosen Gmail as my email provider (I just added a GUID to the end of a base email address), I then decided to use the Gmail IMAP API to access.  It was then easy to call a method in my test which polled Gmail API until an email arrived addressed to my new email address.  Once I had this email, I could extract the HTML of the email, but what to do with this HTML????

Initially I did what I subsequently found others had done; extract the URL of the required link using somekind of HTML parser of XPATH engine, then use the Navigate method in Selenium to navigate the browser to this address (see http://www.seleniumtests.com/2011/08/verify-email-confirmation-using.html ).

While this works, it still doesn't quite feel right. What if the link is hidden? What if we want to validate other aspects of the email?

I then stumbled upon something which should have been obvious; I was using Saucelabs and had a couple of problems;

1. Tests would time out if no command was received within 90 seconds

2. The video of the test would look very slow as we would be stuck on one page without any obvious processing going on, and everybody hates slow tests!

To fix this I decided to use Selenium to inject simple HTML (I only know simple HTML!!!) displaying a message on a blank browser so any viewer of the video would know what was going on.  This would then be injected every 30 seconds to keep Saucelabs happy as well.

Then it was either a "DOH!" or 'Eureka" moment; if I can inject my HTML, then I can inject the email's HTML as well.

So that's it, the idea of Email objects was born. Emails are not different from web pages. They are HTML, they have a DOM, they can be processed, they can navigate to other pages. They can have links, they can have buttons, they are just HTML pages.

I had to escape some offending characters which resulted in java script errors, but apart from that it was surprisingly easy and appears to work across platforms and browsers. I created a base email object which had very similar methods and properties to a base page object such as Load(), IsOnEmail(), ClickLink(int idx), WaitForLoad(), etc.  The base email object can then wrap a lot of the boilerplate code as well, so adding a new type of email is very easy

I've only be using Email Object this for a week, so I am not sure of the best approach yet, and would love to hear from others, and if interested I can share code examples.

Here's an example of what a test using email objects looks like;



     .......

     registrationPage.ClickSendEmail();

     var confirmationEmail = new ConfirmationEmail(_driver, "test+hgjhgjgj@gmail.com";

     confirmartionEmail.WaitForEmail();
     
     Assert.IsTrue(confirmationEmail.IsOnEmail());
     
     var registrationFinalPage = confirmationEmail.ClickConfLink();


     ........

Navigation in Page Objects method returning different pages

Page Objects would now appear to be the standard way to approach test automation when using open source tools such as Selenium.

Of the accepted rules, this one can cause problems;

• Methods return other PageObjects

What happens if the login method returns you to different pages dependent on the page you have logged in from or if the login is unsuccessful?

Proposed solutions are to create different methods dependent on the return type;

public HomePage LoginFromHomePage(string email, string password)
{
     .......
     return new HomePage(driver);
}


public PurchaseFlowLoginFromProductPage(string email, string password)
{
     .......
     return new PurchaseFlowPage(driver);
}



public LoginPage UnsuccessfulLogin(string email, string password)
{
     .......
     return new LoginPage(driver);
}






I wasn't happy with this approach, especially if the AUT can return login to lots of different pages.

By using Generics (this example is C# but I assume can be done in Java too), I am able to create one method;

public T Login<T>(string email, string password) 
{
   .....


   return (T)Activator.CreateInstance(typeof(T), base.Driver);
}



I can then write tests like this;

HomePage page = page1.Login<HomePage>("sello", "password");


OR


PurchaseFlowPage page = productPage.Login< PurchaseFlowPage>("sello", "password");


OR


ErrorPage page = page1.Login<ErrorPage>("sello", "");


I think this is a much better way of appraoching page object navigation

Test result trend not appearing in Jenkins for NUnit plugin?

This was driving me crazy for a week. Some jobs were showing the trends graph, others were not.

Eventually, i solved the problem; your job's name must begin with a lower case if you there are any other upper case characters in the first word. Confused? Here's an example;

OvernightBuild_Test (graph will not appear)


overnightBuild_Test (graph will appear)


Overnight_Build_Test (graph will appear)

If you rename, it will appear (including all previous builds)

Lessons from a WATIR to WATIR-Webdriver port - #5 Finally real Grid support!

My port from Watir to Watir-Webdriver has taken a bit of a back seat recently, and instead I have been working on a new project using Selenium-Webdriver (AKA Selenium 2).  I think a prefer Selenium mainly because I am writing in Java and I get compilation errors and better IDE support! Also, who can forget Selenium Grid? Watir has been crying out for a grid, and yes, I know there is "Watirgrid" but let's be honest, it's doesn't hold a candle to Selenium Grid.  To use "watirgrid" you have to re-write tests to take advantage of it.

Although Selenium-Webdriver does not work with Selenium Grid, I have been able to use a pre-beta version of Selenium Grid 2, and it's great. Easy to use, and with minimal effort I achieved things which previously had taken a lot of effort with Watir.

However, back to promoting Watir! With my Selenium Grid 2 up and running, I thought "mmm, if Watir-Webdriver uses the same Webdriver code as Selenium-Webdriver, can I use the grid with Watir?"  The answer is yes!

This great overiew of Watir-Webdriver (http://watirmelon.com/2010/12/14/watir-webdriver-a-detailed-introduction/) mentions running via Remote Webdriver Server.  Although the blog entry refers to the Remote Webdriver Server for headerless testing is can still be used for browsers.

Assuming you had started a locally hosted Remote Webdriver Server on port 4444, a test for firefiox would look something like this:


require "rubygems"
require "watir-webdriver"
require "watir-webdriver/extensions/wait"
b = Watir::Browser.new(:remote, :url => "http://127.0.0.1:4444/wd/hub", :desired_capabilities => :firefox)
b.goto "www.google.com"www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close
Here comes the good part! If I change the URL to a Selenium Grid 2 hub machine, then my test actually runs on the grid. I could have 10 tests running in paralell on my machine, but the browser under test would actually be on different machines for each test.
Looking back on this entry, I really have explained this all quite poorly. In my defence, it is getting late.

Instead of listening to me, try the following;

1. Goto http://selenium-grid.seleniumhq.org/ , understand what a grid gives you and get jealous that it is only for Selenium!

2. Think about how great your Watir tests would be if you had a grid.

3. Get the Selenium 2 Grid source code, build it and set up a simple grid

4. Run a Watir-Webdriver test against your grid

5. Get very excited

Obviously you still need a parallel running solution to help you leverage the grid to the full potential of distributed test runs . As my new project is in Java, I'm using TestNG which comes with "free" multi-threading!

Lessons from a WATIR to WATIR-Webdriver port - #4 When assert_true is no longer assert_true

I wanted to record problems found (and perhaps solved?) while attempting to port a test automation framework from WATIR to WATIR - Webdriver, and a simple blog entry seemed to be the easiest way.


This problem may be just faced by myself and the caused by the versions of TestUnit, Watir & Webdriver I am using, but just in case other people are having the same issue, here it is for google to find!


For the same test, Assert_True was passing when using WATIR, but failing when using Watir-Webdriver.


After tearing my hair out for a while I realised that when using WATIR, assert_true comes from more-assertions.rb in a gem called s4t-utils, where as when I ran the test as WATIR-Webdriver, the assert_true from the testunit gem is used.


The assert_true in TestUnit gem is far stricter in that the value must be boolean true to pass, where as the other assert_true takes advantage of the fact that in Ruby, anything which isn't nil is true.


Personally, I like the version which insists the value is boolean true, so have decided to change the tests to follow this, although obviously I could have overridden it with the less strict version.


As I said, this problem will not affect everyone, perhaps it's only me, but I wanted it recorded!

Lessons from a WATIR to WATIR-Webdriver port - #3 Ruby meets strong typing

I wanted to record problems found (and perhaps solved?) while attempting to port a test automation framework from WATIR to WATIR - Webdriver, and a simple blog entry seemed to be the easiest way.


WATIR is far less strict than Webdriver about the classes you use to access objects in the html, presumably this because Webdriver is written in Java(???) and that is derives its tag_name from teh class itself.


If I have a visible object which looks like a button but in the html is actually a link, then WATIR is far more forgiving.


For example;


 <a href="http://www.google.com" id="goog">Goto Google</a>


Using id, WATIR allows me several ways to access the object, such as;


@browser.button(:id, "goog")
@browser.link(:id, "goog")


However, with Webdriver, I have to use the class define in the html.  Therefore only the following is valid.


@browser.link(:id, "goog")  NB. I believe Link is an alias of "a", therefore "a" is valid too


If you were to use "button", you would get the following error;


Watir::Exception::UnknownObjectException: unable to locate element, using {:tag_name=>"button", :id=>"goog"}


This is obviously not a big problem, but as I said, exposes mistakes which WATIR allowed and requires changes if those mistakes were made