Risk Analysis for Software testing

Today I like to write about Risk Analysis.

I think Risk Analysis is quite complicated and sophisticated process in any kind of project. We, test engineers, have been exposed to this term projects after projects and I think we already know what it is and what the benefits are.

I would define risk analysis as a process of identifying the risks in the project and prioritize them with severity. So if you work through this priority work items, your project will become less and less risky and even if you are not completely finished the work your project will be less likely to fail. No surprise there.

Now, what's really important questions are "how do you identify all the risks involved with the project?" and "how do you decide which one is riskier than others?"

Here is my practice of risk analysis.

First, I try to find all the risks involves with the project I'm working on into three buckets.

1. Business and entire stack perspective Risks
2. Development Process risk
3. Testing process risk.

For #1, good source for risk assessment would be your PM. Ask her what she think it is the most important and risky part of the feature. And ask what are the impacts when each feature or use case does not work. Then you will find business perspectives of risk.

For #2, good source for risk assessment would be your developer. Ask her to draw how each components and classes interact each other. Ask what part of component in the application has complicated logic or what the dependencies are to find out point of failure. Or just ask her if you were a tester, what would you focus on testing? You will get good feedback from your dev.

For #3, you are a good source for risk assessment. You know your framework and you understands the risk of each business perspective and development perspective. You need to be able to come up with testing process risk.

Now you combine all of your risk analysis data and prioritize them and set some intensity level on each test case.

You can find many different articles about risk analysis. I found these articles very useful.

Heuristic Risk based testing by James Bach
sqa tester.com article

Challeges in SOA test automation

Today, I like to mention about testing a service in SOA world.

I don't think I need to explain what is SOA or what are the benefits of Service Oriented Architecture. You can find good articles and papers online.

So let's talk about service testing. I think there are several different kinds of service in SOA world. One of them would be bottom layer service which is normally a wrapper for database. Another one would be sort of middle layer service which consumes other bottom layer services.

In my opinion, normally the motivation for the bottom layer services is that many different parts of the system accessing this one giant database and managing this DB is getting out of control or already out of control. And the solution that architects and business people sit down and come up with is creating a wrapper service for that DB and do all kind of optimization and throttling.

So when we develop this wrapper(bottom layer service) for DB, normally the business logic is already implemented and used in the system. Now, it sounds simple to move this business logic to one place, but the reality is not as simple as it sounds.

First, going through legacy code and figuring out the business logic is really complicated work. What is available in legacy code might not be available in wrapper service. There're lots of corner cases handled in different kinds of conditions, and so forth.

Second, combining these different business logic to common functionality in the service is another challenge. You cannot simply copy the business logic of legacy code to the wrapper service. Sometimes, you need to come up with some new logic that handles DB more efficient or new logic that returns better sets of data.

Third, the testings is another challenge. So if the new service has some new logic that handles DB more efficient and accurate than existing legacy code, then output of the service might not be the same result as current system produces.

Do you see the challenge here? Now you don't have something to assert to. Existing system inputs and outputs are not the same as new wrapper service returns. And your new service might have logic flaw. So you need to make that the new service is going through the logic as designed and returns correct results. And also test the integrity of the result. If you have billions of data in your DB, it's just impossible to handle every inputs and outputs in your test case.

I'll try to find out best approach for this kind of testing challenge. What do you think? I'll post what I think in next blog.

Assertion in test automation

I think we(software workers) love to use the expression "it depends" a lot when we explain things. And it sounds right. I thought of one topic "When and where to put assertions", but I think the answer covers most of stuff is "it depends." But I really don't want that expression on this topic.

Let's talk about assertions. I think assertions has different meanings in test code compared to dev code. I don't think dev want to use assertion in production code. We want to handle error case more gracefully than failing the execution. I guess dev can use assert while developing application, then disable it when it goes to production.

Test code cannot exist without assertion. That's what we(test engineers) do. So I've been thinking about best practice for assertion usage.

Let me throw some questions that I had.

"Do I need to use only one assertion per test cases?"

"Too many assertion in one test case is bad?"

"Does many assertions in one test case means too many logic is executed in one test case?"

"Is that mean I should have smaller tests?"

"Does helper classes need to have assertions?"

"If assertion is in helper class, then does it mean helper class is part of the test?"

What's your answer? "IT DEPENDS???"

I don't think number of assertions in one test case really does not matter. What it really matters is that one test case is verifying one logical thing. It could be multiple things to check to pass the test or there are multiple steps need to be taken to execute the test case.

But one important thing to remember is that assertion should stay in test procedure code not in helper class code. Because helper class, as its name says, helps test execution. Assertions should be executed before or after helper code executes and keep the generic nature of helpers.

No need to add complexity to your test code

Today, I like to write about over re-factoring in test automation code.

This is not true for every tester who writes test automation code. I've seen this from last two companies that I worked for. But there are interesting view to it.

First, I like to mention about redundant code. What is redundant code? I think one example would be piece of code that contains the same logic spreading around the system or framework. Yes, when some logic change required we need to find all the occurrence of that logic and update them. If you miss one that will become a bug.

Second, what is the purpose of refactoring? I think refactoring is a process of change redundant codes to easy to manage code, which will need combining the same or almost same logic code that spread around the system into one function or class. Or taking the repeating same code out and put them in one method or a class. I think there are many many principles or rules about refactoring(I think there are several books only talk about refactoring). You will find more information about refactoring by google it, bing it or read it from book.

The point I'm trying to make here is sometimes refactoring affects adversely to your test automation. We know that duplicate codes are bad and redundant codes are bad. Refactoring is good because we make change only one place and that will applies for all.

Now, let's take a look at test automation code.

I think there are two different types of code in test automation.

One is test procedure code which states and executes the step to be taken for one test case. The other is helper codes that used by test procedure code.

The problem I'm seeing most of the time is the test procedure code. As you implement more and more test cases, you realized that several test cases are only different in value passed or some small different conditions. So you are seeing duplicate codes from your test cases. And you refactor them out. And you found more things in common and refactor more.

Now what I see from test case method is some thing like this.

public void testCase1()

{

runTest(true, false, false);

}

public void testCase2()

{

runTest(true, true, true)

}

What's the problem with these codes?

First, you added complexity to your test cases. The 'runTest(boolean, boolean , boolean) method contains unnecessary complexity due to the refactoring. The runTest method will have if-else statements for first, second and third boolean values. Sometimes, I see that runTest() kind of method takes "enum" values. Adding more and more complexity. Second, refactoring made the code harder to understand. It is really hard to know what testCase1 and testCase2 do from just by looking at runTest() methods. You need to go inside of runTest() method and go over if-else and switch statements to figure out that testCase1 and 2 execute. If a new tester wants to add new tests, he/she has to spend time and energy to go over unnecessary complexity. It is really waste of time.

So then what is my suggestion?

Do not refactor code to handle a little variation. Refactor only for exact same repeating code with meaningful method name.

Business Driven Testing

dfad