Thursday, October 27, 2016

Key Points to Software Testing

This post will cover various points as an overview to software testing. Because this is an overview it may seem scatter brained, but I promise ever point is relevant and import in software testing

  • It is impossible to test everything
  • The challenge is deciding what to test and what not test
  • The nature of testing is an information gathering process
    • information includes things such as reliability and quality of the system being tested
  • Testing does not improve quality, it provides information that allows for the opportunity of improvement
  • Testing Maturity of a company:
    • Level 0: Debugging is sufficient
    • Level 1: Purpose of testing is to show that the software works
    • Level 2: Purpose of testing is to show that the software doesn't work
    • Level 3: Purpose of testing is to reduce the risk of bad things happening
    • Level 4: Improve development competence to prevent problems
  • Developers are inherently not good testers of their code
    • too close to it
    • typically have the same blind spots as when they were writing it
    • developers are optimists
  • Developers must test their own code
    • Through systematic techniques and discipline
    • Getting help such as code inspections
  • Not all bugs are created equal
  • Factors that influence severity:
    • frequency
    • severity of the consequences of the bug
  • Black box testing only finds failures
    • Advatages:
      • Tests are based on UX
      • Can focus on test cases that
        • Emphasize areas likely to have problems
        • Emphasize test user most likely to see
        • Helps make testing efficient 
      • No Knowledge of system under test(SUT) implementation needed
      • can test non-functional requirements
    • Disadvantages:
      • No formal way to know when you are finished
      • Cannot be sure of how much of the system has been tested
      • Easy to miss error cases (sunny day testing)
      • Requirements are slippery (may change, not ready, not exist)
  • White box testing finds errors, faults and failures
    • testing with knowledge of the innards of the software
    • test cases are determined by the code
    • General, robust approach to coverage testing
  • Equivalence Class Testing
    • You can create sets of input which have equivalent behavior
    • All possible inputs show up in one of the created classes
    • IF you execute one test case with data from an equivalent class, that should cover all the data in that class
  • Boundary Value Testing
    • testing the boundaries of the requirements
    • boundary value testing is logical outgrowth of equivalence class testing (extension of ECT)
    • very useful in testing edge cases (behavior typically changes significantly on or at the boundary)
    • Steps to create Boundary tests:
      • Identify equivalence classes
      • Identify the boundaries of each of the equivalence class
      • For each boundary write one case for one unit below, on the boundary and one below
  • Requirements Testing
    • For each requirement, write one or more test case to validate that requirement 
    • not well structured, not systematic
  • State-Transition Testing
    • Diagram the states of the system
    • Write cases to cover all states and transitions
    • Levels of test cases:
      • Level 1: visit each state at least once. This is consider weak coverage
      • Level 2: Exercise each transition at least once. Considered strong coverage
      • Level 3: Trace all possible paths through the state graph. This is the strongest. However, it is impossible to achieve if there are loops present.
    • General practice is level 2
    • You can Augment by adding additional trips through loops, where deemed important
    • Not useful if there is not external state information
  • Cyclomatic Complexity:
    • C = edges - nodes +2
    • C = closed region +1
    • C = predicates +1
  • Basis Paths
    • Each basis path covers at least one edge that no other basis path covers
    • create one test case for each basis path
    • Because a basis path set covers all nodes and edges, it guarantees both branch and statement coverage
    • Steps to generating a Basis Path Set:
      1. Pick a "baseline" path. It should typical path rather than the 'edge case'
      2. For the second path, change the outcome of the first decision node and keep as many other decisions the same
      3. To generate the third path vary the second decision node instead of the first
      4. Continue varying each decision node, until you reach the bottom of the graph
    • Test Cases from a Basis Path
      • Examine decisions and create input that will fire the decisions as you want them
  • Loop Testing
    • Advantages
      • tests areas most likely have errors
      • can keep the number of loop tests lows
    • Disadvantages
      • can miss special cases
      • nested loops can be a problem
      • input can be difficult to generate
  • Code Inspections
    • How it should work
      • code is written and compiled, but not tested
      • Reviewers receive source code
      • reviews study the code
      • Meeting: reviewers discuss code, generally line by line
      • code author is present at the meeting
      • Faults are recorded
    • You should be looking for
      • visible bugs
      • faults
      • bad design
      • bad code
      • comments
    • Intent: Improve the immediate quality and the future maintainability of the code
  • Configuration Testing
    • Testing combinations of components
    • Testing software compatibility with different configurations of software and hardware

I hope this list with brief explanations helps you in some way, shape, or form.


  1. I think I'm missing something in this post - is there a unifying concept to this list that I'm not seeing? can I do anything with it? I'm trying to figure out what sort of value should I aim to get from this list (is it a checklist to go over at certain occasions? points for a further discussion?) and I'm getting confused.
    Also, much in this list reminds me the opinions (and mistakes) that can be found in the ISTQB CTFL syllabus, why repeat this list again?

    1. I just came up with this list going through a number of software testing books and other materials. There is no overarching theme. It is just a ltst.