Assertion in test automation

I think we(software workers) love to use the expression "it depends" a lot when we explain things. And it sounds right. I thought of one topic "When and where to put assertions", but I think the answer covers most of stuff is "it depends." But I really don't want that expression on this topic.

Let's talk about assertions. I think assertions has different meanings in test code compared to dev code. I don't think dev want to use assertion in production code. We want to handle error case more gracefully than failing the execution. I guess dev can use assert while developing application, then disable it when it goes to production.

Test code cannot exist without assertion. That's what we(test engineers) do. So I've been thinking about best practice for assertion usage.

Let me throw some questions that I had.

"Do I need to use only one assertion per test cases?"

"Too many assertion in one test case is bad?"

"Does many assertions in one test case means too many logic is executed in one test case?"

"Is that mean I should have smaller tests?"

"Does helper classes need to have assertions?"

"If assertion is in helper class, then does it mean helper class is part of the test?"

What's your answer? "IT DEPENDS???"

I don't think number of assertions in one test case really does not matter. What it really matters is that one test case is verifying one logical thing. It could be multiple things to check to pass the test or there are multiple steps need to be taken to execute the test case.

But one important thing to remember is that assertion should stay in test procedure code not in helper class code. Because helper class, as its name says, helps test execution. Assertions should be executed before or after helper code executes and keep the generic nature of helpers.

No need to add complexity to your test code

Today, I like to write about over re-factoring in test automation code.

This is not true for every tester who writes test automation code. I've seen this from last two companies that I worked for. But there are interesting view to it.

First, I like to mention about redundant code. What is redundant code? I think one example would be piece of code that contains the same logic spreading around the system or framework. Yes, when some logic change required we need to find all the occurrence of that logic and update them. If you miss one that will become a bug.

Second, what is the purpose of refactoring? I think refactoring is a process of change redundant codes to easy to manage code, which will need combining the same or almost same logic code that spread around the system into one function or class. Or taking the repeating same code out and put them in one method or a class. I think there are many many principles or rules about refactoring(I think there are several books only talk about refactoring). You will find more information about refactoring by google it, bing it or read it from book.

The point I'm trying to make here is sometimes refactoring affects adversely to your test automation. We know that duplicate codes are bad and redundant codes are bad. Refactoring is good because we make change only one place and that will applies for all.

Now, let's take a look at test automation code.

I think there are two different types of code in test automation.

One is test procedure code which states and executes the step to be taken for one test case. The other is helper codes that used by test procedure code.

The problem I'm seeing most of the time is the test procedure code. As you implement more and more test cases, you realized that several test cases are only different in value passed or some small different conditions. So you are seeing duplicate codes from your test cases. And you refactor them out. And you found more things in common and refactor more.

Now what I see from test case method is some thing like this.

public void testCase1()

{

runTest(true, false, false);

}

public void testCase2()

{

runTest(true, true, true)

}

What's the problem with these codes?

First, you added complexity to your test cases. The 'runTest(boolean, boolean , boolean) method contains unnecessary complexity due to the refactoring. The runTest method will have if-else statements for first, second and third boolean values. Sometimes, I see that runTest() kind of method takes "enum" values. Adding more and more complexity. Second, refactoring made the code harder to understand. It is really hard to know what testCase1 and testCase2 do from just by looking at runTest() methods. You need to go inside of runTest() method and go over if-else and switch statements to figure out that testCase1 and 2 execute. If a new tester wants to add new tests, he/she has to spend time and energy to go over unnecessary complexity. It is really waste of time.

So then what is my suggestion?

Do not refactor code to handle a little variation. Refactor only for exact same repeating code with meaningful method name.

Business Driven Testing

dfad

Test automation purpose and trade-offs

Today, I like to write about misconception and purpose of test automation.

Now days, test engineers' programming capability is crucial in the software industry. I guess there is already an occupation like a Test Architect. And considering scale of the project, we cannot test applications without test automation. I guess I can safely say that test engineers' programming skill are better than like 5-6 years ago.

What is concerning to me is that as test engineers writing more codes, some of them are loosing focus. Test engineers understands design patterns, code quality, OOP concepts etc. I think that is very good thing happened in our test community. However, I really want to say that test automation framework code is not the same as production code. Test automation framework exist to "TEST" applications.

Here are some trade-off for test automation

• Simpleness over performance: One of the biggest difference between test automation code and production code is that there is a team to test production code while there is no other team to test test automation code. So then, we, test engineers, are responsible for testing our test automation framework as well. There are two things we can do about this: unit testing and simple design. You can read a lot about unit testing by searching online. I want to focus one simple design. Simple can be pretty relative term, but I like to define simple as many separate helpers doing one thing. I also want to define simple as avoiding a long chain of dependencies or deep hierarchy. Simple also can mean you do not spend too much time on debugging. Simple means no multiple nested if statements as well. So your test framework will consist of test procedure codes and bunch of small helpers. So you can easily find where the test failed and why the test failed. The usage of many small helpers will not be very efficient or fast. However, this simplicity reduces time for debugging and time for fixing test code bug.

• Readability over duplicate code: Everyone knows that duplicate codes are bad. No one goes against avoiding duplicate code. We also believe that refactoring is virtuous thing in software development. Why duplicate codes are bad? Yes, of course, if you have the same logic of codes written in many places in the application, it will be hell to find all the usage and update them. That's why we want to refactor them and change the codes in one place. I think avoiding duplicate codes principle applies also when we are writing test helper classes. Now, let's closely look at test procedure codes. In one TestCase class, there are several test methods representing testcase1-1, testcase1-2 and testcase1-3. Let's say, each test case differs in inputs. Then codes for input are different, but rest of the codes are pretty much identical. I can see the duplicate codes in three methods. Again, we need to be very careful about refactoring these duplicate codes. Each test method contains test procedure codes, which shows steps to be taken(code-wise) to verify failure or success of that specific use(test) case. This steps should not be over refactored. Refactoring means one more stack trace to debug. And also hides some steps from each test method(since it will be refactored to somewhare outside of the method). Let's not refactor code for the sake of refactoring. 6 months later, it will really hard for you to figure out what you did. And, it will be even harder for other testers to read your test case or add new test case.

Mix and match over inheritance for test automation

Today, I like to talk about test automation design.

First, I like to ask you, test engineers. How much effort does project team put on test automation design? Project team meaning Dev,Test,PM and even architect. Here is a similar question. How much effort does project team put on dev design and code review vs test automation design and test code review? Does your company architect spend time on test automation design?

Is it too obvious question? I think the project team should focus more on dev design and implementation because that is what eventually the company deliver to the customers.

What I'm trying to say here is that since the focus is on dev side, testing side design and implementation tend to be sole responsibility of test engineers. Therefore, you and I, test engineers, should be able to come up with robust and reliable test automation. Which means design of test automation should be well thought through.

Let me cut the intro part here and get to the point. The test automation design.

1. Test Runner. What is the responsibility of test runner? Java world JUnit or TestNG in .Net world NUnit. I believe test runner should be responsible for running tests and only running. Test runner by itself should not have functionality other than executing your test code. I've seen many cases where the test runner extends several crafted base classes. I'm not trying to reference the book like "Gang of 4 design pattern" says that favour composition over inheritance. (It's a great book by the way. And you probably read that several times already). Just simply look at how much flexibility we are loosing by inheriting other base class. Before inheriting the base class, test runner was like "I can run anything!" After inheriting base class, it is like "I'm specialized in running xxx tests". As the project grow, your test runner becomes more and more specialized in running xxx tests.

2. Power of mix and match. I think we can take out commonly used functionality in base classes and put them in feature specific common folder. As the project grow, you will have more classes to put in common folder. But test runner will still like "I can run anything!" We, test engineers, can use any class that makes sense for running test cases. If we find more generic class in feature specific common folder, we can move them up to project level common folder.

3. never say "never/always". When you prepare test automation, try not to convince yourself by saying when I run tests "yyy" will always have to run. You can put "yyy" in setup method in test runner. And "yyy" can be changed and "yyy" may not be used 6 months from now.