How hard is test automation, really?

One of the places I browse sometimes is sqa.stackexchange.com, the Software Testing and Quality Assurance stack exchange. Recently, I submitted a brief answer to a question about the difficulty in getting started with Selenium that I’d like to expand on here, where I have more space.

The question, for posterity:

How difficult is it to start writing test-scripts for someone who has been coding in Java? I’ve been hearing a lot about test-automation, selenium, test-script and so on but I cant get my hands on code-examples or something that can give me an understanding of the difficulty level?

My java skills are intermediate. I have a strong foundation and know the basic language very well.

This is one of those weird questions I get sometimes about test automation. Sometimes it’s developers wanting to know how hard it could possibly be, and sometimes it’s non-coders who have done manual testing and feel that there must be a better way. Ultimately, though, it’s like asking how hard it is to play the guitar: anyone can pick up a guitar and make some sounds, and with a few hours and youtube you can quickly get a few chords down. If you’re trying to play folk or pop music, you can get away for a long time with just a few chords, as evidenced by an old joke from Friends:

 

But if you want to play lead guitar for Dragonforce, you’re going to need more than a few chords:

So really, the question of how hard it is to learn automation depends heavily on what you do with it.

 

How to get started

I know what the typical Java programmer is really asking. “What library, how do I use it, where are the docs, and give me an example I can mutate until it fits my use case”. After all, using Webdriver should be no different than using, say, Apache Commons libraries, right? It’s just a library, like any other. In that sense, it’s easy to get started. The community seems to prefer maven; if you’re not already using it, I’d suggest installing maven 2 eclipse. Then it’s a simple matter to create a new maven project, and add the following to your pom.xml:

<dependencies>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>2.41.0</version>
    </dependency>
</dependencies>

Then you’re off to the races! Google suggests the following basic example to get you rolling:

package org.openqa.selenium.example;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

public class Example  {
    public static void main(String[] args) {
        // Create a new instance of the html unit driver
        // Notice that the remainder of the code relies on the interface, 
        // not the implementation.
        WebDriver driver = new HtmlUnitDriver();

        // And now use this to visit Google
        driver.get("http://www.google.com");

        // Find the text input element by its name
        WebElement element = driver.findElement(By.name("q"));

        // Enter something to search for
        element.sendKeys("Cheese!");

        // Now submit the form. WebDriver will find the form for us from the element
        element.submit();

        // Check the title of the page
        System.out.println("Page title is: " + driver.getTitle());

        driver.quit();
    }
}

And now you’re cooking with Webdriver! Look at that awesome test. Obviously, you’ll want to change that print to an assert, and HTMLUnitDriver changes to ChromeDriver or some such, and then you replace google.com with your own site, but look how simple it is! Straightforward, ordinary test case. Yup.

 

How to get started…¬†better

There’s some merit to what Phoebe is saying at the beginning of the Friends clip I linked above: In order to learn the guitar, you’ll want to touch it… eventually. But in order learn test automation, you don’t begin by sitting down at a computer and installing maven and playing with Webdriver. That doesn’t teach you test automation at all — it just teaches you Webdriver.

Test automation is a discipline much like software development: it can be beautiful, more art than science, or it can be ugly, a spaghetti mess of unmaintainable code. Frequently it is both. Perhaps there are people who can start from Google’s example above and produce clean, elegant code that is easy to maintain, but I am not one of them. For me, software engineering of any kind begins with basic architecture.

Consider your website. How is it structured? What pages are there, what is the typical workflow for the user, what alternate paths are there? Consider your body of existing tests. How are they grouped? What common elements are there to all tests? To a suite? To a few edge cases? Is the way they’re organized the optimal organization, or simply how things happened over time? Are the tests still valid? Do they pass as written, or do you apply a good deal of domain knowledge to transform the test on paper to one that will pass?

The secret to maintainable test code is the same as the secret to maintainable code in general: good planning, DRY code, and clear separation of duties. Instead of making each test an island, I follow two main practices for every test automation project I undertake, regardless if it’s desktop or web: I create a framework to abstract the implementation details, and I use a base test class to abstract common elements like test setup.

Framework

My first functional test frameworks were for desktop and ATM software; I’d never heard of the PageObject concept, but I learned by example how much easier the tests were to read if you had a detailed framework. This also allows for separation of concerns with regards to staff: let those who are good at programming work on the framework, and those who have a good eye for test cases work on the test code itself by calling framework methods. The framework should contain everything needed to interact with the software under test, packaged up in clear, easily read methods like mainWindow.openHelp() or fileDialog.chooseFile(filename). Your test author shouldn’t have to know about things like button names or window handles or xpath locators in order to write a test; they should use something very close to human language, so it’s easy to verify that the automated test matches the manual test case.

In Webdriver, this is accomplished by using PageObjects. In essence, a PageObject is a class that represents a single page or portion of a page in your web application, akin to a class representing a screen or a dialog in desktop automation. There is a PageFactory class that can automatically create elements in your PageObject on the fly when they are required based on annotations; I suggest reading through the documentation, because this is a really neat little time-saver for creating your framework. I also like to create a helper method that will take in a url and return the PageObject that that URL represents, therefore allowing the test to figure out when it’s ended up on the wrong page by mistake.

 

Base test

I could go on for a few paragraphs about why you need a base class for your tests to inherit, but instead, I think I’ll show you an example of a base test:

package com.dealertire.V5Framework;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.log4j.MDC;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
import org.openqa.selenium.Alert;
import org.openqa.selenium.UnhandledAlertException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;


/**
 * The base class for all V5 testing.
 * @author bgreen
 */
public class BaseTest {
    /**WebDriver instance*/
    protected WebDriver driver;
    
    private String remoteHost;
    
    /** The testing environment variables, for global access */
    public static Environment testEnvironment;
    
    /** The browser we're using, just in case you need to know that */
    protected String browserName;
        
    //Logging
    /** The current logger instance, so you can log details of test execution.*/
    protected static Logger logger;
    
    //Pages snipped for brevity	
    
    /**
     * Default constructor. Initializes the logger, sets the remote host, and sets up the test environment.
     */
    public BaseTest() {
        logger = LogManager.getLogger(this.getClass().getSimpleName());
    	MDC.put("module", this.getClass().getSimpleName());

    	
    	//TODO: Read from command-line arguments
    	remoteHost = "[snipped]";
    	browserName = "Internet Explorer";
    	testEnvironment = new Environment(EnvironmentLevel.DEMO,
    											EnvironmentLevel.DEMO,
    											"[snipped demo URL]");
    	

    }
    
    
    /**
     * Set up before each test. 
     * This will create the connection to the RemoteWebDriver instance and request a driver.
     * It will then delete the cookies and navigate to the home page. 
     * @throws IOException if there's an issue with the config file
     */
    @Before
    public void setUpWebDriver() throws IOException {
    	MDC.put("Browser", browserName);
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setBrowserName(browserName);
        capabilities.setCapability("unexpectedAlertBehaviour", "ignore");
        
        logger.info("Connecting to remote host at " + remoteHost);
        
        // Create a new instance of the driver
        try {
            driver = new RemoteWebDriver(new URL(remoteHost), capabilities);
        } catch (MalformedURLException e) {
            logger.fatal("Error creating remote web driver: " + e.getLocalizedMessage());
            throw e;
        }
        
        
        //Correct context
        MDC.put("Browser", browserName + " " + ((RemoteWebDriver) driver).getCapabilities().getVersion());
        
        //Put that cookie down!
        driver.manage().deleteAllCookies();
    }
    
    /**
     * Runs after every test.
     * This will close the browser window and exit the remote web driver session.
     */
    @After
    public void tearDown() {
        if (driver != null) {
            try{
                driver.close();
                driver.quit();
            } catch (UnhandledAlertException e) {
                Alert alert = driver.switchTo().alert();
                alert.dismiss();
                driver.switchTo().defaultContent();
                tearDown();//Retry
            } catch  (WebDriverException e) {
                if (e.getMessage().equalsIgnoreCase("Not yet implemented")) {
                    return;
                } else if (e.getCause() instanceof java.lang.UnsupportedOperationException) {
                    return;
                } else {
                    logger.fatal("Error closing remote web driver session: " + e.getLocalizedMessage());
                    throw e;
                }
            }
        }
    }
    
    /** 
     * Logging
     */
    @Rule 
    public TestWatcher watchman = new TestWatcher() {
        @Override
        protected void succeeded(Description description) {
            logger.info("Test "+ description.getMethodName()  +" succeeded.");
        }

        @Override
        protected void failed(Throwable e, Description description) {
            logger.error("Test "+ description.getMethodName() + " failed with \"" + e.getMessage() + "\".");
        }

        @Override
        protected void starting(Description description) {
            logger.info("Beginning Test " + description.getMethodName());
        }

    };
}

Do you really want every test to be handling its own basic logging, parameter initialization, remote webdriver negotiations, and other cruft? Or do you want it to just contain the steps needed to execute the test? And this is a pretty simple setup; I’ve got others with dozens of lines of configuration file parsing, fallbacks, and so on.

Running and Reporting

How is your test going to get run? From inside Eclipse, manually, every time? Or do you need to plug into a CI system, or a test manager? How are the results reported? Are you going to make an excel spreadsheet? Or do you have to enter the results into a management system? Does that system have an API? If you’re using Sauce Labs, they have some library functions that can make it easy to report the build, suite, and result, but you do have to write that in, and it belongs somewhere. What does the application flow for your test execution look like? How does the person running the test configure it? What data does it need, will that change, and how is it stored? Where will the application under test live while it’s being tested? All these things might influence how you write your test framework, so they have to be considered when you’re working on it.

Conclusion

Ultimately, everything we do in SQA depends a lot on our gut instincts, a good feel for the domain we’re working in, and the education we’ve received, whether formal or informal. You can’t rely on just instinct for very long; even the best instincts can only be improved by learning new techniques and dedication to learning the craft of automated testing. After all, if you’ve worked as a developer, I doubt the first program you ever wrote was as good as the tenth or the hundredth, right? The same for testing frameworks. There’s always something more to learn. I encourage you, no matter what your job title is, to challenge yourself to learn something new at every opportunity.

One thought on “How hard is test automation, really?

Leave a Reply

Your email address will not be published. Required fields are marked *