Why do you like software testing?

Do you like software testing? Why do you like software testing?

This is my first interview question. I've heard various answers on this question. Of course, the answer will not greatly affect hiring decision but I think it's been hard to come by candidates who really impressed me on this question.

You test engineers. Why do you like software testing?
It will be a same question as to ask to musician why he/she like music. Or ask teachers and professors why they like teaching.

One of the greatest value of America, at least I believe, is that the culture here encourages people do what they like to do. You've seen movies like one become successful by chasing their dreams. Sports player, musician, writer, chef, teacher, and etc.

Is there any kid who wants to be the best test engineer? Not even kid. Will there be any college student major in Computer Science wants to be the best software test engineer? Even myself did not know much about software testing. I started as junior developer and if I did not meet the software test architect who got me into software testing, I might not be in software testing field.

One of the famous test engineers, James Whitaker, (I should say used to be a test engineer), preaching about software testing as dying profession. He is proud of running his org without test engineers. (calling SDETs developers). He's got his own reasons and many of his point make sense.

Anyway, let me address my reason why I like testing.

I like software testing because it is NOT straightforward. It's not like you need to do 100 things for small applications and 10,000 things for bigger applications and done with it. And it's not like you're good if you have a certain number of unit tests, functional tests ,integration tests and system tests. It's not about the quantity; it is actually about the quality. There are so many things to consider when we're testing something. Different aspects of problems that the application solves, technology that is used, business expectations, user behavior, priorities, threats, performance, dead line date, project team, process, and etc. Finding best possible testing strategy and execution around this environment is great fun. And every feature, every story(in agile) provides different testing problems.

I like software testing because of its exploring nature. I like the thinking process of how it(new feature or project) might not work in the system and problem that comes as a consequence. I love to apply the Systems Thinking on this. Many people say going over test cases are boring. But coming up with test cases (written or not) or trying something with some suspicion about the application is great fun. "I try this because I've seen similar things before. I try this because from my understanding this new thing might cause problem on certain features that are already exist." Understanding the interactions in the system and explore the possibilities of unexpected interaction issue is really a beauty of testing. And this process start from spec reviews (sprint planning in agile) and I get to know more and more and exploring goes to many interesting directions.

I like software testing because I have many different options to execute tests and there are tons of tools out there to choose from. I would say this is more of test execution fun. This is not about writing automation code I would say. Coming up with best test execution strategy is great fun. What kind of tool is useful for data driven testing? How about model based testing? What's best test automation strategy for agile process? Experience in several different test framework and tools. Quick experiment on new tools. What if we just choose to go with manual for testing this feature? How can we represent our test results and communicate with the team? This requires endless learning. Learn to design good test framework,  learn new languages and new tools, learn how to represent test results, learn how to optimize test execution time, learn, learn and learn.

How about you? Why do you like software testing?

Systems thinking and software testing

Today I like to write a little bit about Systems Thinking and how it is related to software testing.

I've got the word Systems Thinking from James Bach. I saw one of his youtube video and he mentioned about reading Introduction to Systems Thinking. So I google it and youtube it.

Systems thinking (Software testing 101)

More to come but now I want to share the video that I watched more than 20 times.

Dr. Ackoff's Systems thinking presentation at TED

A good test automation framework

Today, I like to mention a little about test automation frameworks.

If you're looking at any kind of job posting (company career site, monster.com, and etc) for a software test engineer now, you will see almost all the posting mention some level of experiences in test automation framework as a requirement. And actually I do use test framework for my testing and it is very useful tool that resolves several limitations of manual testing.

So I thought it is worthwhile to put my thought about what a good test automation framework is in my blog. First, let me scope the area of discussion since test automation framework is a pretty big topic. I'm going to set aside the discussion around pros and cons of test automation in software testing since that's not my main point of this article. And I'm talking about your work related test automation framework not the generic programming language specific test framework. For example, my test automation framework is built on top of JUnit or TestNG. JUnit and TestNG are also test automation frameworks, but those frameworks do not have the specific business logic of my testing.

OK. Let's start. What is test automation frameworks?  Wikipedia mentions "A test automation framework is a set of assumptions, concepts and tools that provide support for automated software testing." My simple definition is "a test automation framework is a program that helps testers execute and validate test cases(test scenarios). So it is a program that helps software testing. The meaning of "help" will be saving time, being more efficient in testing process, gathering useful data easily after test execution and etc compared to the manual human test execution. But it is a program, which means someone has to design, implement and maintain. This brings a very important ground rule of test automation. Benefit of using test automation should always be greater than the effort of designing, implementing and maintaining the test automation. Otherwise there is no point using test automation beside you being cool. :)

I believe having a good test automation framework is to maximize the benefits of test automation and minimize the effort of implementing and maintaining it. So let me address the characteristics of a good test automation framework.

1. Have a clear strategy and everyone should be on the same page. This is the most important and most effective way to maximize the benefit of using the test automation framework. There are many different ways to design and implement test automation frameworks. And it is REALLY hard to come up with one solution that will satisfy every test engineers, which means when a test architect or experienced SDET come up with a framework, he/she should explain the strategy of the test automation and help everyone in the team understand how it works and how to use it. Once everyone is on the same page (know how it works and how to use), implementing test cases (scenarios) is much faster and maintaining the framework is much easier. By following a certain rule or instruction, test writers are less distracted and he/she can understand others code very easily. And when it comes to a situation where the current approach does not fit, it is much easier to notice and notify test automation designer (architect or SDET) to resolve the issue with proper solution.
2. Favor object composition over class hierarchy. This is one advice from Gang of four design pattern. One of the common mistake with test framework is that some framework base class provide too many functionality and become giant class (a few thousand lines of code). This approach works fine at the beginning but it does not scale very well and becomes very complex code base as number of tests grows to handle various test conditions, error scenarios and exceptional cases. One key characteristics of good test framework is to provide flexibility to test writers. Test writers should be able to choose classes or modules they need to execute one test case. Just Mix and Match for their tests.
3. Separation between test execution and validation. This is more about validation code being independent from test execution. It also means validation code by itself does not contain any context about test cases. What this allows to the framework is that you can package one logical chunk of  validation into one class. It will takes some parameters as source of validation. You call validate() method and it will simply return true or false. No decision making process here. For example, if error expecting flag is set do this. if some other case do that. NO. JUST DO THE VALIDATION. What does this allow test writers? You gather source of validation during test execution and pick and construct validation instances you need from pool of validation classes. DO NOT let the validation code to be workflow. LET IT BE collection of appropriate validations for each test case. This promotes great re-usability of the code and less complex code.
4. Build the test framework to support the way test execution steps are defined. The ideal framework, in my opinion, is that defined (written or thought) test execution steps are transformed to test automation code with 1:1 mapping. This seems pretty easy to do but this requires thoughtful design to expose natural language like method names and encapsulate well structured implementation details. One way to think about this is that you're providing framework for test writers who are not as technical as you. This promotes great readability of the code as well. 
5. Be cautious about re-factoring codes. Re-factoring is useful work. It's a good process that turns spaghetti codes to well-structured code. But often times, test automation writers get confused by this concept where he/she does not understand the difference between test automation code and development code. If you look closely on test automation code, there are two big parts: one is test execution steps related part and the other is test automation framework part that supports test execution step code. Re-factoring should largely applied in the latter part of automation code. Here is the common problem that I've seen. At first, test cases that a tester is implementing is very similar beside some inputs, config, or environment, so he re-factor the common piece in one method. Looks good. On the next iteration or release, he adds more test cases. He notices a little variation of the execution step or input, so he add some conditionals (if/switch statement) to the common method. As more iteration or releases come by, the common method becomes more and more complex. Complexity is not the only problem. His thought process of executing test steps is restricted by existing code base and he is trying to fit his thought on that code path. Also it is very hard for other testers to read and follow the logic. Refactoring test execution steps code can be very dangerous because it restricts testers thought process, makes test automation code less flexible, and makes readability of the code very bad and it hurts other test writers to take over. I admit that it is hard process because sometime it is hard to draw a line between test step code and supporting code. The key is to have test execution steps to be flexible and to have supporting code to be well structured.
6. Pinpointing the failure is more important than handling error gracefully. Do not be scared of null pointer exception. Automation code is not production code. I don't think this needs a lot of explanations
7. Minimize bootstrap code. Setup methods like @BeforeClass in JUnit or some sort of prep code runs before the test run is useful. But it also prevent your framework from being flexible. And it also make the framework very heavy. Sometimes you just want to check a simple validation that was used in test automation for other purpose. Well modularized code does not need much bootstrap code. You should be able to run your test in main method if your automation framework is well modularized.

If you want more fundamentals around test automation check this one.  Thoughts around test automation framework
Here is  part2 with more detailed explanation

How to innovate in software testing team

1. Eagerness to be better. An old couple who run hot-dog stand in market place story.
2. Trying to find solution or ideas against ultimate goal rather than improvement of current process. Broccoli and cupcake story. 
3. Steal like an artist. Use other academic approach. Car sales man. Writers.