That's the buffet table.: Pathway Exploratory Testing

Mittwoch, 26. Juli 2017

Pathway Exploratory Testing

Katrina Clokie has a very good section on her blog she calls “pathways”, which are curated link lists and training exercises you can use to get started with certain software testing aspects. Katrina offers these pathways for Security Testing, Mobile Testing and many other topics. You should definitively check them out.

When a colleague asked me to send him an introduction into Exploratory Testing a while ago I created a list of useful blog posts I think are helpful to get started with Exploratory Testing. His answer was something along the lines “oh my, this is a complete pathway”. I think it was not, but it was not that far off from being one.

Therefore I added some more links and thought of some exercises to evolve this into a proper pathway. In the meantime, more people asked me for information about exploratory testing so from now on I can answer with a link and an offer for a joined coffee.

Just as Katrina's pathways, this is a rabbit hole: A lot of the articles below link to other articles, which will forward you to even more sources. I hope this will be a fun ride for you.

STEP 1: What is Exploratory Testing?

The questions often begin with what Exploratory Testing actually is. And perhaps, more importantly, why would you use it in the first place? Sometimes people have some negative bias as if exploratory testing is just toying with the software and can only be an addition to test case based approaches. Sometimes people seemingly have heard great things and now want in on this cool "new" method. Here are a few articles, that describe what Exploratory Testing is and how you can use it in your project:

Exercise: If you are currently not using exploratory testing techniques, but are testing based on test cases, watch your next test execution very carefully: Are you really performing only the specified steps and checking the specified expected results or are you doing more? Are you looking “left and right”? How do you tell people what you saw when you moved away from the script? How do you remember it yourself? Do you take notes that go beyond the test steps of your test cases? Are your cases/steps answering questions about risks?

STEP 2: How do I manage my Exploratory Testing?

Session-Based Test Management is the most popular method to introduce Exploratory Testing in projects. It helps to make Exploratory Testing manageable and reportable, especially towards people outside of the testing team. Michael Bolton used the metaphor of putting the clouds, that are bursts of performed Exploratory Testing, into boxes so you can count those boxes.

Here are some articles about and experience reports on Session-Based Test Management:

If for whatever reason you cannot make Session-Based Test Management work James Lyndsay provides a lot of other ways to manage your Exploratory Testing:

STEP 3: How do I find test charters?

A topic, that troubles a lot of people starting out with Exploratory Testing and Session-Based Test Management is generating test charters in comparison to generating test cases. Most of them are used to derive test cases from any form of specification document, which verifies that the software works “as specified", which is not necessarily “as intended”.

But how to create missions for exploration in addition, or instead of, test cases?

A good idea is to think about the goal you want to achieve with your respective testing mission, Simons examples for different charter types can help you with this. I listed some more models and texts, which can help you in finding test charters:

Types of Sessions by Simon P. Schrijver

Session Based Testing in Agile/DevOps Environments by Lalit Bhamare

Exercise: Chose one of the above approaches and write down down three test charters for the software you are currently working on. You can use the charter template provided in the link from STEP 2.

STEP 4: How do I come up with test ideas?

Once you have your testing mission for a session down, the interesting part is having ideas of what to actually test to serve your mission. I have seen testers, who suffered from some form of writer's block when they started their first test sessions. Especially when they were not involved with designing test cases and only executed them in the scripted testing world they knew.

Fortunately, there are a lot of methods and tools out there to help you come up with test ideas. I just listed a few down there, which you will hear or read often when deeper exploring the Exploratory Testing world, or which help me specifically. If you really, really want to go deep into this topic follow the link to Erik Brickarp's text below and be blown away.

Testing Mnemonics by Lynn McKee
Mindmap Collection by TestInsane
Testing Tours by Microsoft
Difference between oracles and heuristics by Katrina Clokie
Heuristics & Oracles by Rajesh Mathur
Heuristics: Test Heuristics Cheat Sheet by Elizabeth Hendrickson
Oracles: FEW HICCUPPS by Michael Bolton
How to come up with test ideas by Erik Brickarp
Critical Thinking for Testers by Michael Bolton and MoT-Podcast "Getting Critical" with Michael Bolton and Mark Tomlinson
TestSphere Card Deck by Beren van Daele
Rubber Duck Debugging

Exercise: Remember the test charters from the last STEP? Now it is time you perform these test sessions. As a first step, you don’t need to test the whole 90 minutes as described in the linked introduction tests, start with 30 minutes if you like to.

STEP 5: I've got to the end of my session. Now what?

After one or more sessions a Debriefing has to take place. I met several people, who skipped on these and asked: "Do I really need that Debriefing?". Short answer: yes. Long answer: Although a lot of people find it tedious to add yet another meeting to their schedule, the Debriefing is still absolutely crucial. It helps to identify new test charters, spread the testing results across teams, identify holes in documentation and processes, improve the overall testing. Regular Debriefings also help the testers to develop their testing skills. All in all, Debriefings belong to the important meetings you really should attend. Here are some helpful links how you can structure a debriefing:

Exercise: A Debriefing is usually a meeting and you should not do it alone and by yourself. I still want to ask you that you take your time and reflect on the test sessions you performed during the last exercise. Write down answers to the following questions:
What happened during testing? What did you find out? Were there things, you wanted to test but couldn’t and why couldn’t you? What is left to test in maybe another session or did you find interesting new session ideas? What were they? And what were you feeling during the session as a tester or potential user of your software? Was something annoying you or were you positively surprised?

STEP 6: Are there tools, that can help me?

The most obvious tools for an Explorer is that of a classical Explorer: A notebook and a pen to write down what she finds during exploration. Still a lot of people ask about software tools.
There are tools, that help with Session-Based Test Management, for example by aggregating the session metadata in the TASK BREAKDOWN section. Since taking notes is a crucial part during a test session I choose a note-taking tool, Evernote, for my projects. This solution works best for me at the moment, especially with the Mac Client App.

I want to emphasise one tool in particular and that is TestBuddy: TestBuddy is still under development and is being designed specifically for Exploratory Testing and note taking by people, who really love this style of testing. The prototypes I saw look very promising. The link below will bring you to a “waiting list”. Please get in contact with the folks at Qeek, they are eagerly waiting for your feedback and insights.

An Exploratory Testers Notebook by Michael Bolton
http://www.satisfice.com/sbtm/sessions.exe by James Bach
Session Tester by Jonathan Kohl
TestRail by gurock
Evernote by Evernote Inc.
testnote.io by Matt Obee
Exploratory Testing Chrome Extension by Francisco Moreno Sanz
Rapid Reporter by Shmuel Gershon
SBTM Template in Spreadsheet by Altom
TestBuddy by Qeek (Simon Tomes & Rajit Singh) - in development

STEP 7: How do I document my testing?

In an environment that heavily uses scripted testing and test cases, testers usually document their testing by ticking off the steps of a test case as either "passed" or "failed". A test session in Session-Based Test Management does not work that way instead it reflects much more how a detective, a journalist or scientists take notes during an investigation or experiment. A lot of testers switching to Session-Based Test Management are quite surprised at the amount of writing they have to do during test execution and they struggle in finding the right balance.

My personal belief is that you do not write more documentation than you do when using test cases because those have to be written, too. Test cases tend to be heavily documented, a lot of people just don’t connect this to their test execution since they wrote them weeks before not close to or even during testing as they do with a test session.

Another thing I want to add is that in test case based approaches testers often don’t correctly document their actual testing, for example by ticking all test steps in a test case because they “kind of did this” although they skipped several steps due to routine. They often don’t write down strange things (not bugs!) outside of the current test case scope and things get lost.

It's important to find a healthy balance when it comes to documenting your sessions. Here is a list of ideas or experience reports and don’t worry: Even experienced testers constantly question the way they take note as the last three links prove.

Exercise: Alan's 10 experiments in the last link are a great way to get started. How about giving it a go?

STEP 8: Will Exploratory Testing pass an audit?

I often hear people writing Exploratory Testing off as a just playing with the software, a non-structured testing approach, that does not survive the scrutiny of an audit. This is not true at all. Testers have been using Exploratory Testing techniques in heavily restricted environments. Here are a few links, which can help you to report your Exploratory Testing beyond a single session and to make it audit-prone:

Exercise: Try to come up with a low tech testing dashboard for your application and discuss it with your team members.

Books & Resources

In addition to all these blog posts, pdfs or online articles there some books and a series of videos about Exploratory Testing, which I recommend to you: