Home

About Us

PnL Explained Professionals FAQ

Site Map

Glossary

Membership

Contact Us

           

Main Page by Topic

A. PnL Explained

B. CTRM Software

C. Statistics

 

 

Quality Assurance

 

 

This page shares some thoughts and considerations with regard to Quality Assurance for CTRM Systems.

 

Outline

1) Definition of ‘QA’ (Quality Assurance)

2) Half of Quality Assurance Is Assurance

3) Setting the QA Team Up To Fail.  And How to Avoid That.

4) Does Regression Testing Find Bugs?

5) Other Topics

 

1) Definition of ‘QA’ (Quality Assurance)

In the context of this post, QA, i.e., ‘Quality Assurance’ refers to testing software.

 

1.1) Typically the process goes like this:

1.1.1) Software team develops software. 

1.1.2) Then that software is ‘built’ or put into some compiled or ready-to-go form.

1.1.3) Then that compiled software is passed off to another team to check for bugs.  That team is called the ‘QA Team’.

 

1.2) So what we are *not* talking about here, in this post, is quality in the sense of:

1.2.1) Having a good software design in the first place, meaning designs that are functional, fit for purpose.

1.2.2) Coding that is well done and up to appropriate standards of comments and readability.

1.2.3) GUIs/ screens that are easy to use.

These are, of course, super important.  Some of these themes/topics are discussed in prior blog posts and some will be covered in future ones.

 

2) Half of Quality Assurance Is Assurance

 

One of the things I like to say is that ‘Half of Quality Assurance’ is ‘Assurance’.  I mean that in two senses.

 

2.1) First… there are two words in ‘Quality Assurance’.  One of out two is half.

 

2.2) Second, people may forget or not realize how important the ‘Assurance’ part is. 

 

2.3) So what do we mean by ‘Assurance’ in this case?  We mean that people, for example, need to have a comfort level that if QA Testing passes, they’ll be able to do their jobs, i.e., the system will work for their business. 

 

2.4) Another thing to consider for the ‘Assurance’ side of QA is the idea of being very transparent in terms of what you do.  For example, publishing the QA Test Cases.  In other words, users of CTRM software, though really the IT group, should be able to view, e.g., perhaps online, the test cases that are run in the ‘suite’ of test cases that are run for each new build of the CTRM software.

 

In summary, if you just try to focus on ‘quality’, and forget the ‘assurance’ part, you won’t be doing all you can or should.

 

3) Setting the QA Team Up To Fail.  And How to Avoid That.

 

3.1) If you define the mission of your QA team as ‘find/catch all bugs before the software goes out the door’, then your team is bound to fail.  There is no way to catch them all.

 

In fact, I recall reading some theory that basically says that in order to catch all of the bugs in one system, you need a more complicated system.  Can you imagine the monster of a system needed to test all variations of one of the big CTRM systems?  And how many test cases would you need?  100,000?

 

3.2) So what can you do?

You can redefine the QA Team’s mission from ‘find bugs’ to this:

‘The QA will execute a suite of test cases quickly and efficiently’.

 

3.3) For the ‘quickly’ part:  If a QA team goes from taking 5 days to execute a suite/set of test cases to just 3 days, that is an improvement and something management would want.  Maybe the improvement would be do to automated testing or maybe another reason. 

 

In either case, the firm can either decrease the time it takes to get a new release out the door.  Or another possibility is to increase the number of test cases that can be run.  E.g., instead of 10,000, maybe run 15,000.

 

3.4) For the ‘efficiency’ part, this means that the QA team runs the test correctly.

So the idea is that if a bug is found *and* there was a test case *already* that should have caught the bug, and the QA team ran that test case and didn’t find the bug, then the QA team is rightfully to blame.  i.e., in this case, they ‘failed’ to find a bug that they should have.

 

On the other hand, if a bug is found, i.e., by the client after the software went out the door *and* there was no test case that could have caught it, then that is *not* the QA team’s fault.  Why?  They can’t be expected to have test cases to cover all possible scope.   Meaning that simply isn’t realistic. 

 

In summary: The QA team should be held responsible to execute their suite of test cases quickly (speedy is good) and efficiently (meaning effectively).  Whether or not a bug is found that is *outside* the scope/coverage of their suite of test cases isn’t something that they should be held accountable for.  Because it is bound to happen, and setting up a team to constantly fail isn’t a good approach. 

 

3.5) So what to do after a bug is found, meaning by a customer after the software goes out the door?

What a firm should do is to have a policy of requiring the QA team to create a new test case that would have caught the issue. 

 

This is, as the expression goes, ‘closing the barn door after the horses have already left’.  However, think of this:  If a firm created the test cases in this method, i.e., reactively, then the number of test cases created in such a manner will be the most in the areas that are most buggy.  This makes sense from a risk adjusted point of view with regard to testing.

 

Of course, the QA team or related, should continue to create ‘proactive’ test cases, i.e., at least one for each new feature added to the CTRM system.

 

4) Does Regression Testing Find Bugs?

 

Answer: Yes.  No surprise there.

 

But you might be surprised with how little that happens, relative to total bugs caught.  From my experience only about 10% of the bugs caught by the QA Team or Clients are things that could have been caught by regression testing.

 

For regression testing to catch a bug, something would have had to work at some point.  And then gets broken over time, i.e., unexpectedly.  Such that a regression test, which was perhaps automated, detects a difference (a ‘failure’) and reports it as a potential issue, which a human, doing failure analysis on the discrepancy, confirms.

 

From my experience, the other 90% of the bugs/issues are things that were broken from ‘Day 1’, i.e., from the time the developer finished coding.

 

These are things that can’t be caught by regression testing or automated testing.  Unless you want to be generous in your definition, as these bugs, i.e., ‘day 1 bugs’ would be caught when a QA person was creating the regression/automated test case in the first place.

 

 

5) Other Topics

 

This is a list of some additional QA related topics, perhaps for additional commentary in the future:

5.1) Automatic versus Manual Testing

5.2) Thoughts and considerations on different types of QA testing, such as:

5.2.1) Regression testing, i.e., testing before and after, e.g., for a ‘like-for-list’ upgrade

5.2.2) Unit Testing, i.e., testing some feature, such as a report, on its own

5.2.3) Day in the life testing, i.e., end-to-end testing of all of the steps in a typical day

5.2.4) Integration testing, i.e., testing interfaces.

 

 

Introduction to CTRM

Click on this link for a great introduction to CTRM software: Introduction to CTRM Software

 

 

Home

Site Map

Contact Us