Teatime: Improving websites, with science!

Welcome back to Teatime! This is a weekly feature in which we sip tea and discuss some topic related to quality. Feel free to bring your tea and join in with questions in the comments section.

Tea of the week: All the Lemon! by The Tea Dude. I know I usually recommend black teas, but I’ve got a sore throat this week, and lemon tisane hits the spot perfectly. 

Today’s Topic: Improving websites, with science!

“What should I do for teatime this week?” I asked my fellow Sock Devs.

“A/B Testing! With cookies!” It was so off the wall, I had to do it. You might want to grab some cookies to go with your tea this week 😉


So let’s say we have a website.


It’s a very good website; it sells cookies, and cookies are amazing. Everyone wants your cookies. Well, your boss’ cookies. You don’t bake the cookies; you’re the IT guy, keeping the registers working and the website up and running. It’s a small business, so there’s just you that does “the computer things”.

The website takes orders and puts them in a queue; every morning, the baker makes the custom orders, boxes them up, and sends them fedex, marking the orders done on their tablet. They then bake the day’s cookies for the shop, open the place, and a series of bored teenagers try to make the stupid Square dongle work on the tablet so people can buy cookies in person. And it works, but not well enough.

For some reason, the online business isn’t taking off the way they said it should. Mrs Smith, the baker, asks you, “Do we need to call up the Googles and ask them to make the website better?” You assure the baker – she’s a sweet older lady, she doesn’t know much about computers – that your SEO is fine, really, your site is at the top if you search for “Akron Cookies rainbows”, and that’s the best search term. Analytics say a bunch of people are showing up on the site, but there’s not a lot of orders. Maybe the website needs an update?

Conversion Rate

What she’s really worried about isn’t the SEO, it’s the Conversion Rate. The conversion rate is simply a percentage that indicates how many people that land on your website go on to “do the thing” – in this case, buy cookies. It could be signing up for a cookie newsletter, or leaving a review on Yelp, or whatever, but today we’re worried about sales, because when your sales are low, nothing else matters.

Well, you don’t know much about improving conversion rates. You don’t know much about baking cookies, either, or what kinds of behaviors drive people to buy cookies online. I mean, the problem could be anything, right? Maybe people just don’t like cookies anymore? No, no, Mrs Smith assures you that people buy cookies when they’re in the store. Her physical conversion rate is like, 80%. It’s something about the website.

The Scientific Method

Well, you don’t know much about conversion optimization. And you don’t know that much about websites. But you do know something about science. You aced your science fair in middle school, and you got all the way through AP bio in high school before you decided to do websites fulltime instead of becoming a world-famous botanist. Surely the scientific method could help you here, right?

So you dig out your composition book and turn to a new page. We’re ready to begin doing science now. How did it start….

Ask a Question

Oh right. The first thing you need to do is ask a question. As scientists, we are always making observations about the world around us; we don’t start digging into something, however, until we have a question. Why does that flower look that way? Why does paper fall slower than bowling balls if acceleration due to gravity is a constant? What happens if I bake cookies at twice the temperature for half the time?

So what’s our question? How about, “Why aren’t people buying cookies online?”

Gather Data

Next, we need to do some research. If we don’t have good, concrete data about the world around us, it’s going to be impossible to answer the questions. We’re not forging ahead into a realm of experimental physics here; we’re looking at something tangible and concrete in the world, so we can measure what it’s like currently.

So you go and you pull data from google analytics for a week or two, tracking where people go when they’re on the site. It’s not very exciting data; you have a bunch of visitors, but most of them seem to be hitting the home page, scrolling down for a bit, then leaving. Why do they leave? What’s going on here? Now we can dig into the specifics.

Construct Hypothesis

So here’s your cookie homepage:

You hit the site, pretending you’ve never been here before. Maybe you don’t realize you can buy cookies here? I mean, it says cookies, but there’s sky behind it, maybe this is like, a cookie fansite? Or a dictionary page?

It’s time to make a hypothesis, a concrete statement of fact that can be proven or disproven by the data you can collect. One format for a hypothesis is the following madlib: “If I _____, then ________ will happen.”

So you use this template and jot down your hypothesis in your composition notebook:


“If I change the home page picture, then more people will buy cookies.”

Conduct Experiment

Alright, time to get down, dirty, and dangerous! Wait, that’s botany. We’re dealing with websites here. But still, time for science! We’re going to change the home page, and see if conversions go up. But, wait a minute, Mrs Smith was just telling you about her new ad campaign that’s sure to “fix the Googles problem”. And it’s almost October and that’s National Cookie Month, and people tend to buy more cookies when they hear about that on the radio. So how will you know it works? The data’s going to be contaminated by all these outside factors changing! You can’t very well expect Mrs Smith to put her advertising efforts on hold for the purity of your data, and you can’t ask time to stop for you. So what can we do?

A/B Testing


This is where A/B testing comes in. The idea is, you show the new homepage header to half the incoming visitors. That way, the influx of new visitors doesn’t impact your data, because you’re comparing results over the same time period rather than across separate months. The data remains pure and clean, and you can track this specific variable without any outside interference.

Since this is a simple website and you’re using Google Analytics, this is pretty simple to set up. You FTP upload an alternate version of the home page, one with the cookies header. Then you go into google analytics, add an experiment, and put in both URLs. You put some tracking code onto each version of the home page and re-upload them, and voila, Google handles the rest. From there, half the visitors are routed to one version of the home page, and the other half to the other version. Google tracks how many of them go on to buy cookies, and starts giving you data pretty quickly.

Analyze Data

Now in Analytics you can see the results: a breakdown of how many conversions you’re getting on each version of the homepage on each day. And when the website goes down because the power goes out to the bakery and Mrs Smith doesn’t understand what the cloud is or why she should use it when there’s a perfectly good server sitting in the broom closet, that doesn’t affect the data either, because you’re getting 0 conversions for those six hours with both variations. So it all works out!

Act on the Results


Now that you have your results, you can act on it. The cookie homepage is definitely bringing in more sales; you stop the experiment, and re-upload the cookie header version as index.php so it shows up for everyone. The orders start pouring in, so much so that Mrs Smith is able to hire another baker to help handle just home deliveries. She’s so happy, she offers to design a line of cookies with your name on it! You thank her politely, but honestly, it was your interest in botany that saved the website, so maybe cookies that look like plants?


So she agrees! And you all live happily ever after… at least, until the conversion rate tops out and you start learning about SEO in earnest. Then you get to start all over again from the top.


Cookie Monster copyright Sesame Street. Sample website created with Mobirise website generation software.

Teatime: What kinds of testing should I do?

Welcome back to Teatime! This is a weekly feature in which we sip tea and discuss some topic related to quality. Feel free to bring your tea and join in with questions in the comments section.

Tea of the week: It’s been a stressful (and cold!) week as I pre-write this, so I’m chilling out with some Toasted Caramel Rooibos from Sub Rosa Teas


Today’s topic: What kind of testing should I do?

For the first QA-content-filled teatime, I wanted to start at the beginning and touch briefly on what kinds of testing there are, and when to use each of them. This is not intended to be an exhaustive list, more of a gentle overview while we sip tea and meditate on our own projects and the testing each one needs.

Things to consider

When you’re putting together a test plan, and you’re considering performing various types of tests, you should consider the following:

  • What is the purpose of this kind of test?
  • In what situations is this test most suited?
  • What differentiates this test type from others?
  • Why are we performing this test?

Testing without purpose is not so much testing as playing; if there’s no business need, and no reason to perform the test, all you’re doing is wasting your time. Furthermore, if you already have your needs covered, adding more types of tests won’t help anything, by definition: your needs are already covered, and further testing is just wasting time.

The Testing Quadrants

test quadrants

I came across a version of this diagram in the book Continuous Delivery by Jez Humble and David Farley, and I really liked it, so I’ve recreated my own version here. There are two axes represented above, both equally important. On the horizontal axis, testing can either support development efforts, which is to say, it can provide input into the ongoing effort of building the software to correct the course by small degrees; or, it can critique a finished product, producing a feedback loop for development of future enhancements and/or future products. On the vertical axis, tests can be developer-facing, giving feedback to the development team on the project, or they can be user-facing, giving feedback to the BA from a user’s perspective of the software.

Therefore, in this model, acceptance tests are user-oriented tests; do not write acceptance tests for things that are invisible to the user! On the other hand, they support development, so we want to run them as early as possible to tighten the feedback loop so the devs can course-correct. Which argues for an Agile approach 🙂

Unit tests are similar to Acceptance Tests in that they support development and so should be written and executed as early as possible,  but they are developer-facing, so they should absolutely test for things that only developers can see, like method signatures and code organization.

Exploratory tests are like Acceptance Tests in that they are user-facing, and thus should only focus on things the end user can see; however, they are a critique, intended to find things when the product is in a stable, “release candidate” state. They serve to prove to the business that what we built meets their expectations, not to aid developers in building it.

And finally, Nonfunctional acceptance tests allow us to critique the product from a development standpoint: now that we know it works (because of Unit tests), we need to see if it’s performant, secure, et cetera.

What about regression testing?

You may have noticed that regression testing isn’t in any of the four quadrants. It doesn’t really fit into this graph; it sort of lies orthogonal to the graph, or envelops the entire thing. Regression testing is simply the act of verifying that requirements which were previously tested to work have  not been broken since the test was run. Without qualifiers, we typically mean Functional Regression Testing, which is simply running old acceptance or unit tests over again to ensure that the functionality was not broken. You can also, however, perform nonfunctional regression testing, say, to verify that software that was previously fast has not gotten slower, or software that previous installed on Windows still does after being enhanced to run on Linux.  Exploratory regression testing would be exploratory testing of areas that have not changed in the most recent version.

So what kinds should I do?

All of them 🙂

Honestly, only you can answer the question of what types of testing are right for your business needs, your product, your development team, your release cycle, et cetera. Hopefully, however, you have some ideas now that you’ve paused to consider the types available and what goals they fulfill. So, you tell me: what kinds of testing will you do?