The Nature of Exploratory Testing
The Nature of Exploratory Testing, outline
Cem Kaner, James Bach, 2004
http://www.testingeducation.org/a/nature.pdf
Exploratory testing is not a testing technique. It's a way of thinking about testing.
ET is simultaneously:
- Learn about the product
- Learn about the market
- Learn about the ways the product could fail
- Learn about the weaknesses of the product
- Learn about how to test the product
- Test the product
- Report the problems
- Advocate for repairs
- Develop new tests based on what you have learned so far.
In ET, tests are designed and executed at the same time, and they often are not recorded.
ET emphasizes adaptability and learning.
Software testing poses several core challenges to the skilled practitioner/manager. The same challenges apply to exploratory testers:
- Learning (How do we get to know the program?)
Early coverage heuristics: capability before reliability, single features before combinations, visible before hidden, common use cases before unusual
Active reading: Look for
- Affordances: Ways the product can be used.
- Dimensions & Variables: Product space and what changes within it.
- Relationships & Interactions: functions that cooperate or interfere.
- Navigation: Where things are and how to get to them.
- Benefits: What the product is good for, when it has no bugs.
- Consistencies: Fulfillment of logical, factual, and cultural expectations.
- Oracles: Specific mechanisms or principles by which you can spot bugs.
- Bugs and Risks: Specific problems and potential problems that matter.
- History: Where the product has been, and how it came to be.
- Operations: It's users and the conditions under which it will be used.
- Complexities & Challenges: Discover the hardest things to test.
- Attitudes: What your (testers’) clients care about; what they want from you.
- Capabilties: What the testing staff are good at; how they have excelled in the past.
- Resources: Discover tools and information that might help you test.
Active reading: ask specific questions, summarize and reorganize info, describe what you read to others, integrate new with already known.
Learning from available specifications: stakeholder/designer/programmer intent, user expectations. Predictions about the product can be turned into tests
Ambiguity analysis: interpretation of specifications, viewpoints or standards of regulators/auditors, customers, developers.
How is this different from traditional test planning? We create and execute tests as we read. The intent of the research is development of personal insight
Attacks: An attack is a stereotyped class of tests, optimized around a specific type of error. "How to Break Software" by Professor Whittaker
Failure mode lists/Bug taxonomies: "Testing Computer Software" listed 480 common bugs.
Risk catalogs
Scenario development
Models: architecture diagrams, state-based, dataflow
- Visibility (How to see below the surface?)
- Control (How to set internal data values?)
- Risk / selection (Which are the best tests to run?)
- Execution (What’s the most efficient way to run the tests?)
- Logistics (What environment is needed to support test execution?)
- The oracle problem (How do we tell if a test result is correct?)
- Reporting (How can we replicate a failure and report it effectively?)
- Documentation (What test documentation do we need?)
- Measurement (What metrics are appropriate?)
- Stopping (How to decide when to stop testing?)