Teatime: What kinds of testing should I do?

Welcome back to Teatime! This is a weekly feature in which we sip tea and discuss some topic related to quality. Feel free to bring your tea and join in with questions in the comments section.

Tea of the week: It’s been a stressful (and cold!) week as I pre-write this, so I’m chilling out with some Toasted Caramel Rooibos from Sub Rosa Teas


Today’s topic: What kind of testing should I do?

For the first QA-content-filled teatime, I wanted to start at the beginning and touch briefly on what kinds of testing there are, and when to use each of them. This is not intended to be an exhaustive list, more of a gentle overview while we sip tea and meditate on our own projects and the testing each one needs.

Things to consider

When you’re putting together a test plan, and you’re considering performing various types of tests, you should consider the following:

  • What is the purpose of this kind of test?
  • In what situations is this test most suited?
  • What differentiates this test type from others?
  • Why are we performing this test?

Testing without purpose is not so much testing as playing; if there’s no business need, and no reason to perform the test, all you’re doing is wasting your time. Furthermore, if you already have your needs covered, adding more types of tests won’t help anything, by definition: your needs are already covered, and further testing is just wasting time.

The Testing Quadrants

test quadrants

I came across a version of this diagram in the book Continuous Delivery by Jez Humble and David Farley, and I really liked it, so I’ve recreated my own version here. There are two axes represented above, both equally important. On the horizontal axis, testing can either support development efforts, which is to say, it can provide input into the ongoing effort of building the software to correct the course by small degrees; or, it can critique a finished product, producing a feedback loop for development of future enhancements and/or future products. On the vertical axis, tests can be developer-facing, giving feedback to the development team on the project, or they can be user-facing, giving feedback to the BA from a user’s perspective of the software.

Therefore, in this model, acceptance tests are user-oriented tests; do not write acceptance tests for things that are invisible to the user! On the other hand, they support development, so we want to run them as early as possible to tighten the feedback loop so the devs can course-correct. Which argues for an Agile approach 🙂

Unit tests are similar to Acceptance Tests in that they support development and so should be written and executed as early as possible,  but they are developer-facing, so they should absolutely test for things that only developers can see, like method signatures and code organization.

Exploratory tests are like Acceptance Tests in that they are user-facing, and thus should only focus on things the end user can see; however, they are a critique, intended to find things when the product is in a stable, “release candidate” state. They serve to prove to the business that what we built meets their expectations, not to aid developers in building it.

And finally, Nonfunctional acceptance tests allow us to critique the product from a development standpoint: now that we know it works (because of Unit tests), we need to see if it’s performant, secure, et cetera.

What about regression testing?

You may have noticed that regression testing isn’t in any of the four quadrants. It doesn’t really fit into this graph; it sort of lies orthogonal to the graph, or envelops the entire thing. Regression testing is simply the act of verifying that requirements which were previously tested to work have  not been broken since the test was run. Without qualifiers, we typically mean Functional Regression Testing, which is simply running old acceptance or unit tests over again to ensure that the functionality was not broken. You can also, however, perform nonfunctional regression testing, say, to verify that software that was previously fast has not gotten slower, or software that previous installed on Windows still does after being enhanced to run on Linux.  Exploratory regression testing would be exploratory testing of areas that have not changed in the most recent version.

So what kinds should I do?

All of them 🙂

Honestly, only you can answer the question of what types of testing are right for your business needs, your product, your development team, your release cycle, et cetera. Hopefully, however, you have some ideas now that you’ve paused to consider the types available and what goals they fulfill. So, you tell me: what kinds of testing will you do?

New Year’s Resolutions: 2015

My job title says that I work in SQA: Software Quality Assurance, or maybe Quality Analysis if you want to get pedantic, since we don’t actually assure quality so much as kermitflail when it’s not present.

The tests are failing!

But what is quality? How do I know when something is quality or not? I’ve been pondering the nuances this month, being as it is the first month of the year and the time when everyone tries to lay out their goals. Where do I have authority to advise, and when am I overstepping my bounds?

One of my coworkers went to Velocity last year, and came back fired up about performance and Real User Metrics. I can’t find a single definition of quality that doesn’t include performance. If I can make changes that can get metrics in front of him so he can see the realtime impact of his changes, is that SQA?

Our promotions process is slow and buggy and prone to errors. If I can get a system in place that automatically runs unit tests after a code promotion, is that Quality? What if it does linting, checking the style of the code? What if it simplifies the process of making branches to move code onto our demo servers in the first place? Where does Quality become Process? Or is there even a distinction?

Our database development team has trouble keeping their sandboxes in sync. If I poke my hands into their Subversion practices to turn deploying a new sandbox into a half-hour routine maintenance task instead of an all-day chore, is that Quality? Why or why not? Cite your sources.

Ultimately, my goals this year aren’t around improving the codebase. I’m not a developer. I don’t fix anything. What I can do, where I can do the most help, is around helping other people streamline their daily tasks so that they have the energy to make things better. If it takes someone less time to move code, they’ll be less afraid to make fixes that improve the quality. If there’s a safety net of tests, they’ll be able to do some refactors that have been on the wishlist for years. The best way to improve our codebase is to apply grease to the wheels until they turn smoothly and efficiently.

So that’s my new year’s resolution for 2015. Maybe I’ve been watching too much Emma Approved, but maybe I really can help people make their lives better.


How hard is test automation, really?

One of the places I browse sometimes is sqa.stackexchange.com, the Software Testing and Quality Assurance stack exchange. Recently, I submitted a brief answer to a question about the difficulty in getting started with Selenium that I’d like to expand on here, where I have more space.

The question, for posterity:

How difficult is it to start writing test-scripts for someone who has been coding in Java? I’ve been hearing a lot about test-automation, selenium, test-script and so on but I cant get my hands on code-examples or something that can give me an understanding of the difficulty level?

My java skills are intermediate. I have a strong foundation and know the basic language very well.

Continue reading “How hard is test automation, really?”