Introducing the Test Automation Fire Triangle

It is a fact that while some test automation projects succeed, a lot, if not the majority, fail. With this in mind, I tried to establish what the common factors are in a successful test automation project, independent of the system under test. Factors which were true whether the automation is User Interface, REST, SOAP, etc.

I identified 3 cornerstones of a successful project, and then realised how it was very similar to the Fire Triangle we all learnt in our science lessons at school.  

A brief recap of the Fire Triangle can be found here (http://en.wikipedia.org/wiki/Fire_triangle ) but the general concept that is that without the elements: heat, fuel and oxygen, then a fire cannot start or continue.  All three elements are as key as each other.

I believe that in Test Automation, these elements can be replaced with; Stability, Maintainability and Scalability.  Each is as key as each other, and without all three, your project will struggle.  When designing or evaluating your test automation framework, all three must be a consideration from the start, even if not immediately an issue, they will eventually be required.  Within a short space of time your test automation framework will become Shelfware.

Stability

Without a reliable way of identifying objects and stable running mechanism;

• Test results are not reliable
• Manual interaction may be required to ensure full runs
• CI is not possible

Maintainability

The framework must adaptable to change. Object identifiers may be updated, xml formats can be updated.  Your framework must be more Agile then the SUT it is testing else;

• Tests will not be able to run when required
• More time will be spent re-engineering the tests than developing the SUT
• Automated tests will not be be part of the development sign off

Scalability

Okay, so you've got a great stable and maintainable framework, you have hundreds, if not thousands of really fantastic tests which will give you immediate feedback of any broken code. Well, when you say immediate, you mean 24 hours!  One machine running all the tests is not a solution going forward.  You need lots of machines, but for this you will need lots of licenses if not open source. Even then you need a mechanism to control and collate test running and results. Without this;

• The framework becomes too expensive (20 x QTP licenses is a lot of money!)
• CI cannot be achieved
• Test runs take too long to be of any use in an Agile project

Conclusion

I am not saying that these 3 elements are the only things required for a successful project, but other decisions are dependent on the SUT itself.  For example;

• Useability (ie. how easy is it to write scripts) - this is dependent on the audience the tool is aimed at.
• Readability - Are you expecting people to read scripts who do not write them?
• Cross-platform - Is the SUT strictly one platform only?

My main message is that the 3 elements in the Test Automation Fire Triangle are key to every test automation project.  Without these, your test automation will fail.  Maybe not immediately if you have the Stability sorted, but as the SUT grows, lack of planning and foresight in Maintainability and Scalability will mean that your Test Framework will struggle to grow with it, and ultimately end up on the Test Automation scrapheap, along with so many of it's friends.