Skills required for a test engineer

Today, I like to talk about the skills need for a test engineer.

1. Inference


This is the most important skill you need for being a test engineer. This make software testing more like Art than Science. It requires creativity, intuition, experience, analytical reasoning and extra. What can you infer from requirements or spec? I would say that spec is explicit requirements for the project. However there are always implicit requirements for the given project. For a simple example, "how would you test a soda pop vending machine?" Beside putting money and getting the right soda pop, there are a lot of things you need to consider. Like, how to check the temperature of the soda pop, how to make sure to notify office when power is out on vending machine, how to notify office when soda pop is out, how to make vending machine energy saving, on and on and on.


2. Prioritization

This the second most important skill you need. After you come up with all the possible test cases for the given project, you should be able to put each test case into priority bucket. Risk analysis would be a good tool for you. Based on the risk (what and how big is the impact of this test case failing), you can categorize all the test cases in to priority bucket. As we all know, we have limited time and budget, we need to make conscious decision not to test based on the risk.


3. Troubleshooting

This is another core skill for test engineer. Test engineer should be very good at finding root cause of the problem. Can you eliminate the noise related to the problem? Can you find out the pattern from certain condition? Time is clicking. When you have production issue, how good your troubleshooting skill is directly affecting company's loss

4. Simple and robust coding skill

Now days, software is very complex and large scale. We need to use program to test program. There are a lot of testing tools that helps testing. But fundamentally, there is no other test team that test your testing application. You need to beware of the fact when you write automation code. Besides unit testing, you need to be able to write simple and robust code. Less if statements in your code, less switch statements, avoiding nested if statements. These are all helping you not to make automation bugs.

5. Big picture in mind

If you are a test engineer, you should have better understanding of the system than developer. Developers are normally focused on what they are developing. You need to have end to end mind, system overall operation. Help developer understand how the feature they're developing fits with existing application.


6. Multi-tasking or rapid switching one task to another

Multi-tasking/switching one task to another is another skill you need. People notify testers for so many different problems they're having. Lab environment, live production issue, project testing,and etc. You normally handle several projects at a time and you should be ready for handling production issue, support issue and etc. This can be very frustrating if you are not used to it.

Having business layer in automation

Today I like to write about having business logic layer in test automation.

Now days, I guess test automation is not something new in software testing. Whether your are testing desktop application, web application or SOA application, test automation plays an important role.

I've been writing quite a bit of automation code through out my software testing career. Of course, there are multiple different ways to write automation and it depends highly on the context of application.

What I've seen so far on writing test automation is pretty straightforward. You design your automation based on two separate purpose. Test execution code and helpers.

Test execution code basically represents the steps you need to go through to execute one test case. Normally, test execution code is a method starts with testBlahBlah...(). Helpers are more generic classes or functions that help test execution code. Things like until classes. String Helpers, DB Handlers, File Readers and etc.

I think these separation works out pretty well. I don't see much of problem with this way. Helper classes provide good flexibility to test execution code. Basically, you can mix and match helper classes based on any need of each test case.

And I thought of this business layer between test execution code and helper codes in automation.

Now if you see your one test execution code closely, you can easily tell that it contains three major part. The first one is setting up some situation and cleaning up or setting back to initial state for your test. The second one is executing what you're testing. (i.e. click submit button on your web page, sending request to the service, calling a function or etc. The last one is some sort of verification.

So, I normally have these three business logic classes in my automation, which means test execution code interact with these three business logic classes to do its operation.


So the automation flow is "Test Execution code" -> "Business logic lay class" -> "Helper classes".

The benefit you get from this design is pretty interesting. First, Test execution code can interact with the business logic layer without knowing helper classes, which means test execution code can call business logic layer class with sort of natural language. For example, we want to test user type something on discussion board goes to data base correctly. Now test execution code will do something like this.

public void testUserInputOnDiscussionBoard()
{
MyBusinessLogicClassForDiscussionBoard blc = new MyBusinessLogicClassForDiscussionBOard();
blc.login();
blc.goToDiscussionPage();
blc.typeRandomStringAndSubmit();

MyVerifyingClass vc = new MyVerifyingClass();
vc.verifyingDiscussionBoardInput(blc.getUserId(), blc.getRandomString());

blc.logOut();
}

This is pretty simplified version of it. But my point is that test execution code should not worry about how to login to the page, how to put some text on the page, or even how to check whether db is updated or not.

Second, I can keep the helper classes more generic. Since the business logic class will know what DB to use or what credential to use or what helpers classes to use,helper classes will not have business logic other than doing its own thing. DB Helper will only expose server name and DB name. And execute query code. You do not have to create some child class of some generic helper class.

Third, modifying test case is much easier. Since the business logic stays in one logical place, you can easily find places to change.

Risk Analysis for Software testing

Today I like to write about Risk Analysis.

I think Risk Analysis is quite complicated and sophisticated process in any kind of project. We, test engineers, have been exposed to this term projects after projects and I think we already know what it is and what the benefits are.

I would define risk analysis as a process of identifying the risks in the project and prioritize them with severity. So if you work through this priority work items, your project will become less and less risky and even if you are not completely finished the work your project will be less likely to fail. No surprise there.

Now, what's really important questions are "how do you identify all the risks involved with the project?" and "how do you decide which one is riskier than others?"

Here is my practice of risk analysis.

First, I try to find all the risks involves with the project I'm working on into three buckets.

1. Business and entire stack perspective Risks
2. Development Process risk
3. Testing process risk.

For #1, good source for risk assessment would be your PM. Ask her what she think it is the most important and risky part of the feature. And ask what are the impacts when each feature or use case does not work. Then you will find business perspectives of risk.

For #2, good source for risk assessment would be your developer. Ask her to draw how each components and classes interact each other. Ask what part of component in the application has complicated logic or what the dependencies are to find out point of failure. Or just ask her if you were a tester, what would you focus on testing? You will get good feedback from your dev.

For #3, you are a good source for risk assessment. You know your framework and you understands the risk of each business perspective and development perspective. You need to be able to come up with testing process risk.

Now you combine all of your risk analysis data and prioritize them and set some intensity level on each test case.

You can find many different articles about risk analysis. I found these articles very useful.

Heuristic Risk based testing by James Bach
sqa tester.com article

Challeges in SOA test automation

Today, I like to mention about testing a service in SOA world.

I don't think I need to explain what is SOA or what are the benefits of Service Oriented Architecture. You can find good articles and papers online.

So let's talk about service testing. I think there are several different kinds of service in SOA world. One of them would be bottom layer service which is normally a wrapper for database. Another one would be sort of middle layer service which consumes other bottom layer services.

In my opinion, normally the motivation for the bottom layer services is that many different parts of the system accessing this one giant database and managing this DB is getting out of control or already out of control. And the solution that architects and business people sit down and come up with is creating a wrapper service for that DB and do all kind of optimization and throttling.

So when we develop this wrapper(bottom layer service) for DB, normally the business logic is already implemented and used in the system. Now, it sounds simple to move this business logic to one place, but the reality is not as simple as it sounds.

First, going through legacy code and figuring out the business logic is really complicated work. What is available in legacy code might not be available in wrapper service. There're lots of corner cases handled in different kinds of conditions, and so forth.

Second, combining these different business logic to common functionality in the service is another challenge. You cannot simply copy the business logic of legacy code to the wrapper service. Sometimes, you need to come up with some new logic that handles DB more efficient or new logic that returns better sets of data.

Third, the testings is another challenge. So if the new service has some new logic that handles DB more efficient and accurate than existing legacy code, then output of the service might not be the same result as current system produces.

Do you see the challenge here? Now you don't have something to assert to. Existing system inputs and outputs are not the same as new wrapper service returns. And your new service might have logic flaw. So you need to make that the new service is going through the logic as designed and returns correct results. And also test the integrity of the result. If you have billions of data in your DB, it's just impossible to handle every inputs and outputs in your test case.

I'll try to find out best approach for this kind of testing challenge. What do you think? I'll post what I think in next blog.

Assertion in test automation

I think we(software workers) love to use the expression "it depends" a lot when we explain things. And it sounds right. I thought of one topic "When and where to put assertions", but I think the answer covers most of stuff is "it depends." But I really don't want that expression on this topic.

Let's talk about assertions. I think assertions has different meanings in test code compared to dev code. I don't think dev want to use assertion in production code. We want to handle error case more gracefully than failing the execution. I guess dev can use assert while developing application, then disable it when it goes to production.

Test code cannot exist without assertion. That's what we(test engineers) do. So I've been thinking about best practice for assertion usage.

Let me throw some questions that I had.

"Do I need to use only one assertion per test cases?"

"Too many assertion in one test case is bad?"

"Does many assertions in one test case means too many logic is executed in one test case?"

"Is that mean I should have smaller tests?"

"Does helper classes need to have assertions?"

"If assertion is in helper class, then does it mean helper class is part of the test?"

What's your answer? "IT DEPENDS???"

I don't think number of assertions in one test case really does not matter. What it really matters is that one test case is verifying one logical thing. It could be multiple things to check to pass the test or there are multiple steps need to be taken to execute the test case.

But one important thing to remember is that assertion should stay in test procedure code not in helper class code. Because helper class, as its name says, helps test execution. Assertions should be executed before or after helper code executes and keep the generic nature of helpers.