|
|
|
|
Main
Page by Topic |
||
|
C. Statistics |
||
Quality
Assurance
This page shares
some thoughts and considerations with regard to Quality Assurance for CTRM
Systems.
Outline
1) Definition
of ‘QA’ (Quality Assurance)
2) Half of
Quality Assurance Is Assurance
3) Setting the
QA Team Up To Fail. And How to Avoid
That.
4) Does
Regression Testing Find Bugs?
5) Other
Topics
1) Definition of ‘QA’ (Quality Assurance)
In the context
of this post, QA, i.e., ‘Quality Assurance’ refers to testing software.
1.1) Typically
the process goes like this:
1.1.1)
Software team develops software.
1.1.2) Then
that software is ‘built’ or put into some compiled or ready-to-go form.
1.1.3) Then
that compiled software is passed off to another team to check for bugs. That team is called the ‘QA Team’.
1.2) So what
we are *not* talking about here, in this post, is quality in the sense of:
1.2.1) Having
a good software design in the first place, meaning designs that are functional,
fit for purpose.
1.2.2) Coding
that is well done and up to appropriate standards of comments and readability.
1.2.3) GUIs/
screens that are easy to use.
These are, of
course, super important. Some of these
themes/topics are discussed in prior blog posts and some will be covered in
future ones.
2) Half of Quality Assurance Is Assurance
One of the
things I like to say is that ‘Half of Quality Assurance’ is ‘Assurance’. I mean that in two senses.
2.1) First…
there are two words in ‘Quality Assurance’.
One of out two is half.
2.2) Second,
people may forget or not realize how important the ‘Assurance’ part is.
2.3) So what
do we mean by ‘Assurance’ in this case?
We mean that people, for example, need to have a comfort level that if
QA Testing passes, they’ll be able to do their jobs, i.e., the system will work
for their business.
2.4) Another
thing to consider for the ‘Assurance’ side of QA is the idea of being very
transparent in terms of what you do. For
example, publishing the QA Test Cases.
In other words, users of CTRM software, though really the IT group,
should be able to view, e.g., perhaps online, the test cases that are run in
the ‘suite’ of test cases that are run for each new build of the CTRM software.
In summary, if
you just try to focus on ‘quality’, and forget the ‘assurance’ part, you won’t
be doing all you can or should.
3) Setting the QA Team Up To Fail. And How to Avoid That.
3.1) If you
define the mission of your QA team as ‘find/catch all bugs before the software
goes out the door’, then your team is bound to fail. There is no way to catch them all.
In fact, I recall
reading some theory that basically says that in order to catch all of the bugs
in one system, you need a more complicated system. Can you imagine the monster of a system
needed to test all variations of one of the big CTRM systems? And how many test cases would you need? 100,000?
3.2) So what
can you do?
You can
redefine the QA Team’s mission from ‘find bugs’ to this:
‘The QA will
execute a suite of test cases quickly and efficiently’.
3.3) For the
‘quickly’ part: If a QA team goes from
taking 5 days to execute a suite/set of test cases to just 3 days, that is an
improvement and something management would want. Maybe the improvement would be do to
automated testing or maybe another reason.
In either
case, the firm can either decrease the time it takes to get a new release out
the door. Or another possibility is to
increase the number of test cases that can be run. E.g., instead of 10,000, maybe run 15,000.
3.4) For the
‘efficiency’ part, this means that the QA team runs the test correctly.
So the idea is
that if a bug is found *and* there was a test case *already* that should have
caught the bug, and the QA team ran that test case and didn’t find the bug,
then the QA team is rightfully to blame.
i.e., in this case, they ‘failed’ to find a bug that they should have.
On the other
hand, if a bug is found, i.e., by the client after the software went out the
door *and* there was no test case that could have caught it, then that is *not*
the QA team’s fault. Why? They can’t be expected to have test cases to
cover all possible scope. Meaning that
simply isn’t realistic.
In summary:
The QA team should be held responsible to execute their suite of test cases
quickly (speedy is good) and efficiently (meaning effectively). Whether or not a bug is found that is
*outside* the scope/coverage of their suite of test cases isn’t something that
they should be held accountable for.
Because it is bound to happen, and setting up a team to constantly fail
isn’t a good approach.
3.5) So what
to do after a bug is found, meaning by a customer after the software goes out
the door?
What a firm
should do is to have a policy of requiring the QA team to create a new test
case that would have caught the issue.
This is, as
the expression goes, ‘closing the barn door after the horses have already
left’. However, think of this: If a firm created the test cases in this
method, i.e., reactively, then the number of test cases created in such a
manner will be the most in the areas that are most buggy. This makes sense from a risk adjusted point
of view with regard to testing.
Of course, the
QA team or related, should continue to create ‘proactive’ test cases, i.e., at
least one for each new feature added to the CTRM system.
4) Does Regression Testing Find Bugs?
Answer:
Yes. No surprise there.
But you might
be surprised with how little that happens, relative to total bugs caught. From my experience only about 10% of the bugs
caught by the QA Team or Clients are things that could have been caught by
regression testing.
For regression
testing to catch a bug, something would have had to work at some point. And then gets broken over time, i.e.,
unexpectedly. Such that a regression
test, which was perhaps automated, detects a difference (a ‘failure’) and
reports it as a potential issue, which a human, doing failure analysis on the
discrepancy, confirms.
From my
experience, the other 90% of the bugs/issues are things that were broken from
‘Day 1’, i.e., from the time the developer finished coding.
These are
things that can’t be caught by regression testing or automated testing. Unless you want to be generous in your
definition, as these bugs, i.e., ‘day 1 bugs’ would be caught when a QA person
was creating the regression/automated test case in the first place.
5) Other Topics
This is a list
of some additional QA related topics, perhaps for additional commentary in the
future:
5.1) Automatic
versus Manual Testing
5.2) Thoughts
and considerations on different types of QA testing, such as:
5.2.1)
Regression testing, i.e., testing before and after, e.g., for a ‘like-for-list’
upgrade
5.2.2) Unit
Testing, i.e., testing some feature, such as a report, on its own
5.2.3) Day in
the life testing, i.e., end-to-end testing of all of the steps in a typical day
5.2.4)
Integration testing, i.e., testing interfaces.
Introduction to
CTRM
Click on this
link for a great introduction to CTRM software: Introduction to CTRM Software