A reply to the Oracle problem

Cem Kaner has written a post  “The Oracle Problem and the Teaching of Software Testing”.

My comment to that is waiting for moderation. I’m publishing it here, as he is probably too busy to handle all the comments on his blog.

My comment:

I agree that the Hoffman list is better suited for automation, mostly due to being more focused.
But I can not agree that Bach/Bolton list is not helpful for designing tests.

For the exam question I would start with asking what was changed, why the new version – before thinking about specific oracles for test automation. I cannot test everything, and even If there will be additional manual testing I would still run into deadlines. Thus I would need to narrow my testing scope. For that I would use the Bach/Bolton consistency heuristics, in order to understand the change and its impact on the product, its claims, purpose, etc. And then I would go into automating my tests and using the Hoffman list do help me design them.
And whenever I find a possible problem I can then use both lists to figure out if I have a problem, in the product or in my oracles.

Using either list effectively means that it needs to be updated according to project specifics. The more detailed and focused the list of heuristics is the more it needs to be adapted to the current project.

Bach/Bolton list is more generic, thus needing thinking testers to use them with their full potential. They need to expand the heuristics to bring them to the level for using in daily testing, be it manual or automation.

Advertisements

3 thoughts on “A reply to the Oracle problem

  1. I understand your reasoning, and I think your comment reflects James Bach’s point of view to a large degree. But I’m writing as a teacher who has gained a lot of experience with this material. I am evaluating it as instructional material. My responsibility is to decide what is instructionally effective and what is not instructionally effective, and to replace the stuff that doesn’t work with other stuff that does.

    It is easy to blame the students–to say they are not “thinking testers” or that they are lazy or uncreative– for not being inspired to use these materials “with their full potential.” But I’ve worked with a lot of very bright, very hardworking students. They range from young university students who have no work experience all the way through very experienced practitioners.

    When these students (including the ones I consider my best students) make the same problems again and again, or miss the same points again and again, my conclusion cannot be that they are not “thinking testers.” My conclusion has to be that it is time for me to do something else.

    • Although my teaching experience is limited, I have seen similar problems with students getting fascinated with one aspect of testing exercises, finding the ‘hidden’ context, for example. And they tend to spend most of their effort doing that rather than focusing on the actual issue. Couple of reasons for that is, I think, that it seems new, encourages creative thinking, and does not seem to be too much actual structured work.

      I don’t think it’s about being the “thinking tester” or not, but more about how we go about teaching testing. In addition to making sure people understand oracles and heuristics, we need to construct exercises/questions that help them to learn how to apply them to actual work. Which oracles are useful in this specific context? How to apply “consistency with claims” heuristics to state based testing? Etc. I’m getting a feeling that the RST course is a bit introductory and for experienced/bright/hardworking/thinking testers there needs to be a ‘advanced’ course. Where the focus is on test design on a more sophisticated level with detailed, longer exercises. I don’t know if BBST course covers that, as I have not had time to do it yet.

      In conclusion, I agree, that this is not students problem, but the way we teach testing.
      My replay was mostly driven from not agreeing with the sentence: “…do find them helpful for evaluating test results, I don’t find them helpful for designing tests.”
      That, I feel, is taking the “replacing stuff that does not work” to the extreme.

      • Well, you have your experience, I have mine. Mine is pretty consistent, over several years and many students. My summary is that, in my experience, the consistency heuristics are too general to be useful for designing tests. You can call that “extreme.” I call it a summary of what happened.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s