Assertion in test automation

I think we(software workers) love to use the expression "it depends" a lot when we explain things. And it sounds right. I thought of one topic "When and where to put assertions", but I think the answer covers most of stuff is "it depends." But I really don't want that expression on this topic.

Let's talk about assertions. I think assertions has different meanings in test code compared to dev code. I don't think dev want to use assertion in production code. We want to handle error case more gracefully than failing the execution. I guess dev can use assert while developing application, then disable it when it goes to production.

Test code cannot exist without assertion. That's what we(test engineers) do. So I've been thinking about best practice for assertion usage.

Let me throw some questions that I had.

"Do I need to use only one assertion per test cases?"

"Too many assertion in one test case is bad?"

"Does many assertions in one test case means too many logic is executed in one test case?"

"Is that mean I should have smaller tests?"

"Does helper classes need to have assertions?"

"If assertion is in helper class, then does it mean helper class is part of the test?"

What's your answer? "IT DEPENDS???"

I don't think number of assertions in one test case really does not matter. What it really matters is that one test case is verifying one logical thing. It could be multiple things to check to pass the test or there are multiple steps need to be taken to execute the test case.

But one important thing to remember is that assertion should stay in test procedure code not in helper class code. Because helper class, as its name says, helps test execution. Assertions should be executed before or after helper code executes and keep the generic nature of helpers.

No need to add complexity to your test code

Today, I like to write about over re-factoring in test automation code.

This is not true for every tester who writes test automation code. I've seen this from last two companies that I worked for. But there are interesting view to it.

First, I like to mention about redundant code. What is redundant code? I think one example would be piece of code that contains the same logic spreading around the system or framework. Yes, when some logic change required we need to find all the occurrence of that logic and update them. If you miss one that will become a bug.

Second, what is the purpose of refactoring? I think refactoring is a process of change redundant codes to easy to manage code, which will need combining the same or almost same logic code that spread around the system into one function or class. Or taking the repeating same code out and put them in one method or a class. I think there are many many principles or rules about refactoring(I think there are several books only talk about refactoring). You will find more information about refactoring by google it, bing it or read it from book.

The point I'm trying to make here is sometimes refactoring affects adversely to your test automation. We know that duplicate codes are bad and redundant codes are bad. Refactoring is good because we make change only one place and that will applies for all.

Now, let's take a look at test automation code.

I think there are two different types of code in test automation.

One is test procedure code which states and executes the step to be taken for one test case. The other is helper codes that used by test procedure code.

The problem I'm seeing most of the time is the test procedure code. As you implement more and more test cases, you realized that several test cases are only different in value passed or some small different conditions. So you are seeing duplicate codes from your test cases. And you refactor them out. And you found more things in common and refactor more.

Now what I see from test case method is some thing like this.

public void testCase1()

{

runTest(true, false, false);

}

public void testCase2()

{

runTest(true, true, true)

}

What's the problem with these codes?

First, you added complexity to your test cases. The 'runTest(boolean, boolean , boolean) method contains unnecessary complexity due to the refactoring. The runTest method will have if-else statements for first, second and third boolean values. Sometimes, I see that runTest() kind of method takes "enum" values. Adding more and more complexity. Second, refactoring made the code harder to understand. It is really hard to know what testCase1 and testCase2 do from just by looking at runTest() methods. You need to go inside of runTest() method and go over if-else and switch statements to figure out that testCase1 and 2 execute. If a new tester wants to add new tests, he/she has to spend time and energy to go over unnecessary complexity. It is really waste of time.

So then what is my suggestion?

Do not refactor code to handle a little variation. Refactor only for exact same repeating code with meaningful method name.

Business Driven Testing

dfad

Test automation purpose and trade-offs

Today, I like to write about misconception and purpose of test automation.

Now days, test engineers' programming capability is crucial in the software industry. I guess there is already an occupation like a Test Architect. And considering scale of the project, we cannot test applications without test automation. I guess I can safely say that test engineers' programming skill are better than like 5-6 years ago.

What is concerning to me is that as test engineers writing more codes, some of them are loosing focus. Test engineers understands design patterns, code quality, OOP concepts etc. I think that is very good thing happened in our test community. However, I really want to say that test automation framework code is not the same as production code. Test automation framework exist to "TEST" applications.

Here are some trade-off for test automation

• Simpleness over performance: One of the biggest difference between test automation code and production code is that there is a team to test production code while there is no other team to test test automation code. So then, we, test engineers, are responsible for testing our test automation framework as well. There are two things we can do about this: unit testing and simple design. You can read a lot about unit testing by searching online. I want to focus one simple design. Simple can be pretty relative term, but I like to define simple as many separate helpers doing one thing. I also want to define simple as avoiding a long chain of dependencies or deep hierarchy. Simple also can mean you do not spend too much time on debugging. Simple means no multiple nested if statements as well. So your test framework will consist of test procedure codes and bunch of small helpers. So you can easily find where the test failed and why the test failed. The usage of many small helpers will not be very efficient or fast. However, this simplicity reduces time for debugging and time for fixing test code bug.

• Readability over duplicate code: Everyone knows that duplicate codes are bad. No one goes against avoiding duplicate code. We also believe that refactoring is virtuous thing in software development. Why duplicate codes are bad? Yes, of course, if you have the same logic of codes written in many places in the application, it will be hell to find all the usage and update them. That's why we want to refactor them and change the codes in one place. I think avoiding duplicate codes principle applies also when we are writing test helper classes. Now, let's closely look at test procedure codes. In one TestCase class, there are several test methods representing testcase1-1, testcase1-2 and testcase1-3. Let's say, each test case differs in inputs. Then codes for input are different, but rest of the codes are pretty much identical. I can see the duplicate codes in three methods. Again, we need to be very careful about refactoring these duplicate codes. Each test method contains test procedure codes, which shows steps to be taken(code-wise) to verify failure or success of that specific use(test) case. This steps should not be over refactored. Refactoring means one more stack trace to debug. And also hides some steps from each test method(since it will be refactored to somewhare outside of the method). Let's not refactor code for the sake of refactoring. 6 months later, it will really hard for you to figure out what you did. And, it will be even harder for other testers to read your test case or add new test case.

Mix and match over inheritance for test automation

Today, I like to talk about test automation design.

First, I like to ask you, test engineers. How much effort does project team put on test automation design? Project team meaning Dev,Test,PM and even architect. Here is a similar question. How much effort does project team put on dev design and code review vs test automation design and test code review? Does your company architect spend time on test automation design?

Is it too obvious question? I think the project team should focus more on dev design and implementation because that is what eventually the company deliver to the customers.

What I'm trying to say here is that since the focus is on dev side, testing side design and implementation tend to be sole responsibility of test engineers. Therefore, you and I, test engineers, should be able to come up with robust and reliable test automation. Which means design of test automation should be well thought through.

Let me cut the intro part here and get to the point. The test automation design.

1. Test Runner. What is the responsibility of test runner? Java world JUnit or TestNG in .Net world NUnit. I believe test runner should be responsible for running tests and only running. Test runner by itself should not have functionality other than executing your test code. I've seen many cases where the test runner extends several crafted base classes. I'm not trying to reference the book like "Gang of 4 design pattern" says that favour composition over inheritance. (It's a great book by the way. And you probably read that several times already). Just simply look at how much flexibility we are loosing by inheriting other base class. Before inheriting the base class, test runner was like "I can run anything!" After inheriting base class, it is like "I'm specialized in running xxx tests". As the project grow, your test runner becomes more and more specialized in running xxx tests.

2. Power of mix and match. I think we can take out commonly used functionality in base classes and put them in feature specific common folder. As the project grow, you will have more classes to put in common folder. But test runner will still like "I can run anything!" We, test engineers, can use any class that makes sense for running test cases. If we find more generic class in feature specific common folder, we can move them up to project level common folder.

3. never say "never/always". When you prepare test automation, try not to convince yourself by saying when I run tests "yyy" will always have to run. You can put "yyy" in setup method in test runner. And "yyy" can be changed and "yyy" may not be used 6 months from now.

Things to consider for a good test automation

1. Keep the linear dependencies.

• This is not only for test framework but also development code as well. Code changes are inevitable, linear dependencies help minimize the side effect of code changes. Loop dependencies cause development hell. Here is a good lecture about why loop dependencies lead slow down the project. Link
  

2. Make robust and simple backbone of test framework strategy.

• This is pretty obvious statement I guess. The thing that I want to point out is that we need to sit down and think about framework design before starting to write test code. How test runner will interact with application and how test runner will prepare test data need to be thought through and determined. As we draw the overall design of the framework, we can see the dependencies and think about possible design patterns or strategy we cam apply.
  

3. Use object for multiple test data values.

• This is more like a tip. If test data is being passed throughout the tests and it contains several values (id, name, data etc.), encapsulate test data by using an object. This will help add and remove test data details easily and overwriting toString() method help you log your test data
  

4. Find proper category of test cases as you go

• This is from my personal experience. This might be not a problem for others. Normally you will have test suites and test cases. And test suites contains list of related test cases. Test case is normally one class(java, cs , etc) And you name the class (i.e. LoginPageTests.java or something). As you implement the test cases and as you add more test cases, you will find more specific test cases under that test case class. Let say for LoginPageTest.java you find javascript related test cases and non-java script related cases(this is just an example). What I did was creating a new class called LoginPageJavascriptTest.java. Since LoginPageTest.java was in version control for a while and I did not want to rename them for some other reasons. So my LoginPageTest.java is non-javascript test cases. This is not very cohesive class name for its purpose. The proper name would be LoginPageNonJavascriptTest.java. What I'm trying to say here is that as soon as you realize that non-cohesive name, step back and think about what is proper category for your class. What would be the more appropriate category for test suites and test cases. DO NOT STUCK with your class name you created 4 weeks ago. Be flexible of naming.
  

5. Re-factor implementation logic not test case steps.

• This is pretty challenging part for myself as well. I can see the duplicate code and have strong urge to re-factor them. This is mainly for readability. Your code is owned by your company and other test engineers will see or work with your code. Normally one test case has steps like set up, test procedures, assertion and clean up. I guess set-up and tear down(clean up) can be refactored. But if you refactor test procedures and assertion part, it will be hard for others to follow your steps or even harder for finding flaws in your test procedures.
  

6. Use helper method instead of hard-coded value

Test Engineer Value

Today I like to write about the role of test engineers in software industry and value they are bringing to the software development process.

First, let me ask, what kind of value does a test engineer bring to a software company? Why software company needs test engineers for their projects? I guess the role of test engineers has been changed along with evolution of software industry. Normally people say test engineers are responsible for the quality of the product, and I believe that statement.

Do we need test engineers because developers are not perfect and they make mistakes? I think that is part of the test engineers' role. I guess we had time like that. Old days, we called developers "programmers" and they are responsible for 70 to 80% of the software development work. At the end of the project, "testers" are basically verifying their work. And there were a lot of 'hot fixes' after that. The industry has found a lot of problems about that approach in terms of cost, development pace or even success of the project.

Now, we know that test engineers are required to do more than verifying software based on spec or trying different inputs from end to end testing. I think the core requirements for test engineers are clear understanding of functionality and business value of the software, good knowledge of risk and vulnerabilities of the context of the software and strong capability of test case execution.

It sounds pretty obvious, but let's go over one by one.

Clear understanding of functionality and business value of the software

This involves with test engineers participating on spec reviews and designing of architecture of the software. Test engineers represents the end user but that is not all about test engineers. Test engineers are internal people of the software. They need to understand the architecture of the software and the interactions among components. If you join a new company and want to test, first thing you need to do would be understanding the architecture and components of the software. After that, they need to think about business value of the software. We need to be able to look at software from business perspective. Now, we know the software inside and out. We understand how the software works and how it is used.

Good knowledge of risk and vulnerabilities of the context of the software

This has a lot to do with your expertise or experience. Every software is different. I think understanding of vulnerabilities of the software you are testing is really a crucial part of your job as a test engineer. What is the risk of developing client application? What is the risk of developing web/network application? What is the risk of developing mobile application? (My plan for next blogs is focusing on risk and vulnerabilities of different context of software.) Test engineers can execute a lot of test cases on a software, but it is hard to evaluate what they are doing is exactly what software testing needed for that particular context. Software quality comes from extensive testing on risky and vulnerable part of the software. Understanding of the context of the software is worth emphasizing.

Strong capability of test case execution

This is pretty personal matters. Software development business or any kind of business has time and budget constraints. If your organization has enough resources and can afford them, that would be great. I'm trying not to talk about details of manual testing and test automation. Good test engineers need to come up with best method to execute the test cases. Some test engineers are obsessed with test automation. Some test engineers are not bothering to consider automation. We need to be able to choose best method for executing test cases and should be able to handle them. If you are not good at test automation, go back to library and study it. Understand code quality and maintainability of the software. Study about design patterns and best practices. If you think you are obsessed with automation, sit back and think about how can you use intelligent human brains and senses. There are absolutely many places where manual testing work better than automation. I remember Jame Whittaker at Google mentioning 'User experience code and infrastructure code.' Basically, manual testing is better fit for sophisticated user experience code testing. And test automation is better fit for internal of the software. I agree with his point. But in practice we cannot separate them strictly I guess. But spend some time think about which method would be better for a particular test execution rather than going with your favorite method.

Women in Software Testing

This is a bit strange, but I want to write something I find interesting. I have not work for many different companies, but each of my test team manager was a woman. And I've seen many women speakers who talked about software testing in seminars and conferences. I do not have scientific evidence about it, but I feel that women are pretty strong in software testing field.

Why do I feel that way?

I want to talk about gut feeling or six sense part of the software testing. This might has a lot to do with experience. Have you experience something like worries or uncomfortable feeling while you are reading a spec. Not that the spec is poorly written, but you get some feeling just by looking at the spec. Or while you are testing something, do you get some kind feeling that this is not going to work? This might be a little different from the smells of the code that software development engineers calls when they see others' code.

Here is my experience. Marketing team come up with really good plan for our users. We are going to send customized progress report based on user activities on our site. PM come up with nice spec about the benefits and plan for this. The very first question I got from our lady test engineer: "There are about ten millions of users and more than ten million activities. We need to join massive size of tables to generate info for customized email. What's our plan for that?"

Here is another experience. I asked my wife who is a senior civil engineer. I asked her, "How would you test soda vending machine? What are the things you want to make sure to sell soda vending machines?" Her first answer; I'll check to make sure the temperature soda machine keeps certain degree. I don't like warm soda.

People say men tend to be logical and women tends to be emotional. I think we need both in our software testing. Man-only testing team is not as good as men and women team. Maybe it is not about gender. I get different ideas from different guy test engineers. This is one of the reason why our team does test case reviews. Just like developers do code reviews. We like to hear other people's perspectives. And I remember reading interesting James Whittaker's blog post: test engineer types. Let me find that link. Here

I am admitting that I'm using pretty logical approach to test applications. Software testing is also a creative thinking process. Should I develop my creative emotional thinking part or just depend that part on our women test engineers?

Bye.

Programming by intention in automated test case part2

Hello,

In part1, I explained what is programming by intention technique is and showed one example. Now, we can move to automated test cases.

As you might guess, automated test cases are set of instructions. There are steps to go through until it reaches assertions. By using programming by intention technique, we can represent each step with a private method. Maybe making all the steps with private methods will not be possible, but you can make distinction pretty clearly. One benefit comes from doing so is that you can extract private methods that are used many places in your test framework and make them helpers or utility methods. Therefore, you can call helper methods instead of the function you defined in your test case, which make re-factoring pretty convenient for you.

Let's take a look at one simple test case in web application. You are testing a user changing his/her profile info in your application (birth day), and you are using some sort api that handles browser navigation(i.e. selenium).

Let me write how code looks like without using the technique

public void testProfileBirthdayChange(){

// creating a new selenium client instance
Selenium selenium = new DefaultSelenium(...,..,.,); selenium.start();
selenium.open("/");

// login
selenium.type(....email..);
selenium.type(.. password..);
selenium.click(...login button..);
selenium.waitForPageToLoad(...);

// go to profile page and change birthday
selenium.open(..profile page..);
selenium.click(.. profile change button); selenium.waitForPageToLoad(...);
String year = "1960";
String month = "09";
String day = "29";
selenium.type(.. year of birthday);
selenium.type(.. month of birthday);
selenium.type(.. day of birthday);
selenium.click(..save button);
selenium.waitForPageToLoad(...);

// move to home page again and come back to profile page selenium.open("/");
selenium.click(..profile page link);

// assert for changed birthday
assertTrue(selenium.getValueOf(birth year).equals(year)); assertTrue(selenium.getValueOf(birth month).equals(month)); assertTrue(selenium.getValueOf(birth day).equals(day)); }

This example is pretty easy and clear enough for you to change later. Let's see how this simple test case becomes simpler.

public void testProfileBirthdayChange(){
Selenium selenium = SeleniumHelper.getDefaultSelnium();

// log-in
login(selenium);

// go to profile page and change the birthday
String year = "1960";
String month = "09";
String day = "29";
changeBirthdayOnProfilePage(selenium, year, month, day);

// move to home page again and come back to profile page
goHomepageAndComebackToProfilePage(selenium);

// assert for changed birthday
assertTrue(selenium.getValueOf(birth year).equals(year));
assertTrue(selenium.getValueOf(birth month).equals(month));
assertTrue(selenium.getValueOf(birth day).equals(day));
}

Then you can go implement the private methods. You can create birthday random as well. The point is you can easily see the each step more clearly than previous one. And also some private methods might be used again in other test cases. You can extract login() method to be helper method because most of the web application requires login. If you have more and more steps to do, this technique help you keep track of your steps more easily. Also you can apply development change more flexible way. If just simple buttons or id name changes, you do not have to touch your steps, you can only change the implementation. If step changes, you can easily find where the new step needs to go because long implementation for each steps are in the private method.

Programming by intention in automated test case part1

Today, I like to write a little bit about coding. Specifically about writing automated test cases like in test frameworks.

There is a technique called programming by intention. I came across this term for the first time from netobjectives.com. I'm not sure it is a well-known term or not. But, the idea about this technique is pretty strong and effective. There are many benefits you can get by doing this. I'll mention about these benefits later.

My understanding of this technique is to lay out the steps or process first and finish implementation later. This keeps your thinking process and coding staying on track. It is like writing a pseudo code, but it is writing actual codes. I guess the best way to explain this technique is using example.

Let's say you are writing a function called 'whatTypeOfTriangleIsThis(int a, int b, int c)' and this will return some numeric value representing certain type of triangle. If you write this in one function, you will have to use a lot of if statements. And you will keep thinking about cases to cover. But soon yon can figure out how to group them and handle them in each group, that's the idea. Let me lay out the steps

1. Are inputs valid? If not -1

2. Can inputs make triangle? If not -1

3. Is this equilateral? If yes 1

4. Is this isosceles? If yes 2

5. Is this scalene? If yes 3

I could write this function like

public int whatTypeOfTriangleIsThis(int a, int b, int c){

if(!areInputsValid(a,b,c) || !canInputsMakeTriangle(a,b,c))

return -1;

if(isThisEquilateral(a,b,c))

return 1;

else if(isThisIsosceles(a,b,c,))

return 2;

else return 3;

}

Now I go back to each private method and implement the rest.

I'll comeback for more explanation.

Bye.

Software Testing - mindset

I've been working as a test engineers for a few years. And if someone ask me "what is the most important thing for test engineers have to have?", I would say it is the mindset. What differentiates test engineers from development engineers? What are the things we(test engineers) are specialized in?

If you cannot answer those questions, you need to read the rest.

To answer those questions, I need to talk about bugs. What is a bug? Who writes a bug? We all know that there is no bug free software. Yes, we are humans and we make mistakes. Developers cannot write bug free code.

From my experience, there are two kinds of bugs. The first one is the mistake developers make while they are writing codes. He/she coded incorrectly and the code does not work the way it should. The other one comes when they miss some possible situations. He/she did not handle null case or did not handle really big input or did not handle negative value and so on.

The first is normally straight forward. We can easily get all the possible permutations or possible expectations from the spec or the context. I think the latter is the part we(test engineers) are specialized in. Possible situation is not about inputs for some functions. It's about considering all the possibilities that will affect the application. We are so good at figuring out what those pieces of code or thousands lines of code will face. We are also good at prioritizing the situation in terms of business perspective and software development perspective.

A good developer is capable of writing good code in any kind of application, environment or language.

A good test engineer is capable of coming up with good possible situations whether it is one method, components, client applications, web applications or enterprise applications.

Going back to mindset.
Test engineer's job is not making sure the application works as spec states. It's about understanding the aspects of the development process and figuring out important situations that the application will come across.

A string Permutations

This might be a good example of recursion.

So one example question would be:
Print all permutations of a given string. (i.e. "abc" would print "abc", "acb", "bac", "bca", "cab", and "cba")

I start thinking about patterns in terms of recursion.

1. There should be a base case and a recursive case.
2. For recursive case, I need to call the same function with sub version of original string.
3. Let's call the function name "printPermutation()".
4. The pattern I came up with is the following

• Go through the String from index 0 to index length -1
• And then take the character at index i and call printPermutation() with rest of the string
• Let's take a look at example string "abc"
• My logic is when index is at 'a' and call printPermutation() with "bc" and when index is at 'b' call printPermuation() with "ac" and so on.
• So the string will eventually become size of 1 then print all and return (this seems like base case)
• Then when the function takes size of 1 string, it needs to know what is the parent string is.(sort of path the function is taken)
• OK... So I wrote printPermutation() with two parameters: printPermutation(String current, String parent);

5. Now I have the base case:

if(current.length() == 1){
System.out.println(parent+current);
return;
}

6. Oh now, I need a simple function takes an index and a string. It will remove the character(index) from given string.

public static String removeCharAt(int index, String str) {
return str.substring(0, index) + str.substring(index + 1);
}

7. then recursive part would look like

for(int i =0; i<current.length(); i++)
printPermutation(removeCharAt(i, current), parent + current.charAt(i));
}

8. Following is my solution

public static void printPermutation(String current, String parent){
if(current.length() == 1){
System.out.println(parent+current);
return;
}

for(int i =0; i printPermutation(removeCharAt(i, current), parent + current.charAt(i));
}

}

public static String removeCharAt(int index, String str) {
return str.substring(0, index) + str.substring(index + 1);
}


What do you think?