Systems thinking and test automation

Today, I like to share some of thoughts around test automation related to Systems Thinking.

You as an SDET, it is very important to make your team understand what your test automation strategy is. When it is clear, your team can support you and align their work with your automation strategy.

OK. let's start. To be sure, I am specifically writing about functional test automation here. And, in fact, I consider test automation as regression tool not as testing activity. I like James Bach's term 'checking' to describe automation. However, I do believe that test automation as regression tool has great value.

My test automation strategy is based on 'Systems Thinking'. Dr. Russel Ackoff came up with this idea and there there has been lots of conference around this.  I posted his video on my blog, so you can watch his video.

There are two principles that I like to mention from his talk.

First, when it comes to the system, as a whole, it creates a unique behavior or property that none of its parts have.

Second, System's unique behavior or property is created by the interaction among individual components(parts) NOT by sum of individual parts behavior.

As Dr. Ackoff explained, let me take a CAR for example.
What is CAR's unique behavior/property? That is carrying you(person) one place to another. It is only unique as a system. Let's take a look at each parts of the car. Engine, transmission, door, battery and so on. None of them can carry you one place to another. But that unique property cannot be created without parts. More specifically, the interactions between parts creates that property. You can easily understand that simply having the parts without interaction is not contributing to produce a unique system's property.

The point I'm applying to my test automation strategy is the INTERACTION.

Here is one more car example. Let's say, your car stopped while you're driving. And let's assume, it is because the engine is stopped. Yes. it is obvious that broken engine caused car to stop. But if you think a slight different perspective, car actually stopped because no force or thrust was delivered to the wheel. In other words, the interaction was broken.

And now, let's say your car is stopped, but you don't know what caused it. What do modern car manufactures do to identify issue?  Of course they are using some sort of sensors and chips. And it alerts when any sensor detects something that is not normal or when certain criteria they set is not met. And then mechanics are going after actual root causes based on that information.

I think SDETs job is to put sensors all over the code using automation. Sensors that detects all the small functionalities  that comes from interaction between components and modules. Sensors that detects component/sub-component behaviors that comes from interactions of classes and methods (unit tests - this is normally written by dev).

What this thorough automation provides a simple value. Make easy/straight bugs to be found easily and hard/mysterious bugs to be found hard.  Sounds simple, but this will give tremendous trust to your work and your test automation.

Testers are like caddies in some sense

Today, I like to describe testers with some odd(?) metaphor.

I think testers are like caddies in some sense. More specifically, the guy who walks with Tiger Woods or Phil Mickelson on the field of Professional Golf Tournament. The caddies of PGA tours. 

Why? I'm going to explain why.. 

First off, Caddy is an occupation. Golf player is an occupation as well. Caddie might not get attention as much as the players, but still he/she plays very important part of professional golf tournament. It sounds like testers in software development, doesn't it?

Here is one thought on being testers. I've seen so many testers who complains a lot about the company not treating them as important as developers. Being treated like second class citizens or getting paid less. Again it's an occupation. You choose to do it. If you care about being treated like developers, then you should be a developer. If you do not enjoy testing, you should not do testing. Perception is really hard to change. If you care what others' think about you, well..I don't know. I think you should be yourself.

 However, I truly respect those testers who has been fighting for the importance of testing in software industry over the years.  

OK.. let's start.

Caddies understand the game of golf as much as players on the field. They know general information about the game like the golf course, golf rules, difficulties of each holes, distance of each hole, location of hazard, trees around the hole, bunkers and roughs. And they also know what affects each shot. They understand things that affects the shot like winds, uphill, down hill, rough, bunker, reading greens, soft or hard green, wet grass or dry grass and etc. Coming back to testers. They should also know about general information about the product/service they're testing. What kind of product it is (web, mobile, desktop and etc.), who are the users, what kind of technology the team using, what features are important for the business, what development process is used (agile, kanban or waterfall) and so on. And they also understand what affects the developments like software architecture, code review, check-in process, bug triage, regular build process, regression strategy, reporting mechanism and so on. This is really basic stuff. Good caddies and good testers take seriously about the basics of their work.

Caddies provide good information for each shot players make. You may have seen this on TV. PGA golf players always discusses with a caddy before he/she makes shot. Caddies and players both check the distance, hole location, hole surroundings,  ball location, wind, grass, slope, curves, trees and so on. That brief conversation before the shot is the result of quick analysis. Testers are similar in that sense. Test plan/strategy should show the analysis of the feature development. Whether it is formal or informal, testers should analyze the feature that is going to be developed and discuss with developers. Here are some examples. User impact, complexity of the component, impact to existing code, risk, timeline, algorithms or implementation plan, testability and so on. This requires a lot of studying and experiments. Testers should continuously learn how to do better analysis and improve.

Caddies know players more than players know themselves. Professional caddies know his/her players so well. Strength and weakness. Ups and downs of emotion. Distance of each club. Fade and hook tendencies. And they know how to communicate with those things with players. Sometimes caddies insist on certain club because they can predict what his/her player will hit. This is really important part for testers as well. Testing is not just having a list of things and execute them. Knowing the context is the key. That's why I love the context driven school guys. Testing never starts from nothing at the beginning. Product has history. It has business environment and developers. Testers should try to understand developers' coding style and thought pattern. It really requires a close attention to each developer, but it really pays off. Thought patterns are amazingly accurate to predict. Sometimes you can even guess what a developer would  implement this particular class or functions. Bug trends as well. Who makes what kind of bugs can be understood by testers.

Well. I'll stop here.. But there is actually a big difference between caddies and testers. Actual testing part. Testing performed by testers are not there in caddie's work. I'll address that later..

               

A good test automation framework - Part 2

I previously wrote about test automation framework with details related to coding. (I would call it part 1)

Today, I like to mention a little bit more technical explanations of a good test automation framework.

I guess you (test engineers) have worked on test automation framework that was written by someone else and also worked on designing it from scratch. Have you feel some fuzzy or itchy feeling about the test automation you're working on? Something seems to be wrong but hard to explain? Does it take hours just to add a new test case? Do you feel test execution is heavy? I have the same feeling as well. I've been seriously thinking about that and I came up with following technical details. I don't think it is just because of  different coding style nor different thinking process. Hopefully, this blog actually scratch those itchy feelings. :)

1. Goal setting: This is about initial design of test automation framework. You need to set two goals. If you can satisfy both goals from framework user (testers average 2-3 years on automation), I would say your design is successful. The first goal is the framework should be easy to understand, easy to use and easy to implement. And the second goal is you need to design test framework in a way that volume of the code increases but complexity of the code does not. This seems to general and yes it depends on the context of your application. But if you write test automation framework from scratch, please do not simply start with drawing diagrams. Think about consumer of your framework. And also how the framework will grow and not become collection of giant and complex code base.

2. Do not re-factor test execution step code. This is for the user of test framework. I mentioned from my first blog to be cautious about it. Now I change my mind. Again, Do not re-factor test executions step code.  It really has many side effects.

First, your coding is restricted by re-factored code. You framework API provide several functionality and normally those are the components that supports your test execution steps. Your test execution code should be just mix and match of those components. Once you re-factor the execution steps, you are losing your flexibility of test case implementation. It's really silly to be proud of test execution step code  to be one function and it takes 6-7 parameters. Also, you lose the readability of test execution steps. Now how can you replace your test case document with your code? It's so hard to read. DRY (Do not Repeat Yourself) does not applied to test execution step code.

Second, you are adding unnecessary complexity by re-factoring code. I have a good metaphor to back up this argument. Let's say you want to test a car and you have test system that control every part of the car. But that system provides only one button for control. With one button click you need to come up with a way to start the engine, step on gas, break, changing gear, steering wheel control, lights, radio and etc. How would you control that with one button. Yes, you need to come up with some sort of Morse Code to differentiate the operations. "bip bip wait wait bippppp bip wait bippppp" that means turn the head light on. How about you have buttons for each operations. Engine button, break button, accelerator button, light button, changing gear button and etc.

Third, you don't gain much anyways. In test automation framework, what part of the codes are most frequently and most likely added and modified? It is test executions step code. Test cases are keep added and modified as the application grows sprint by sprint. Let's say you have 8 distinct test case which exercise different aspects of the application. Let's say test case 1 and 2 have 70% of the same code. And test case 2 and 3 share 60% of code a slight different way. Test case 3 and 4 shares 65% of code but also slightly different way and so on. It looks like a lot of codes are repeating. Now you start re-factoring. I can tell you; it not easy. What do you end up having? Excessive object composition or excessive class hierarchy of code. Or. you can see the same name of re-factored code with less parameters all over the place. Base class or helpers. Now look at your code after re-factoring. Lines of code you save is not much. And guess what new test case comes in and you need to re-re-factor your code.

Forth, it's not uniformly re-factored and not re-factored by responsibility or business logic. Most of test execution step code re-factoring happens due to repeating code. Not like automation framework code. Framework code is re-factored to achieve Open-Close principle or component decomposition. And it is normally uniformly re-factored because it has its own purpose. Refactoring is not simply putting repeating code in one place. Refactoring makes the code more maintainable and extensible. Existing test execution step changes in several different way, I tell you I've experienced..... It's hell.

Fifth, putting repeating code to base class causes weak cohesion and giant class (3000 lines of code kind of giant class). I'll explain this more in details later. I personally do not recommend people to use TestBase class because once you open the door, it's open for any code to go in. It stars with really essential pieces of code for bootstrap. And later add some more helpers. And later people start putting repeating test execution step code. And people start to add another base class that inherit from TestBse class. Adding if statements and switch statements. Where is the cohesion? What does this base class specifically do? Everything!


A good test automation framework continues....
 

Defining Agile Testing

Today, I like to define what exactly agile testing is. And also I like to mention drawbacks of agile testing and how to accommodate that. If you find something missing please reply on the comment section.

Following seems to be the unique parts of Agile testing. It's an interesting mind shift if you've been working as a tester in Waterfall environment.

Testing effort on defect prevention.
This is one of the fundamental differences between Agile testing and testing in Waterfall. In Agile testing, test team puts their effort on creating good quality builds before it gets to QA team (before deployment) whereas Waterfall testing puts efforts on after it gets to QA team (after deployment). This seems just a small difference but it has quite valuable implications. First, for test team to find issue before dev code gets deployed to QA environment, there should be a good communication channel between dev/pm/test and even business. Test team can help dev team not to make any assumptions about unclear spec. Test team can provide potential impact to the application due to current development(risk assessment/analysis). Test team can also provide some scripts or test data that dev team can run and use on their local environment.  Dev team can do a simple demo to the test/pm/business to verify his/her progress. All these to be happened, barrier among all disciplines has to be removed and all has to simply work as a team. I've seen a lot of issues are found during those informal discussions and demos. And those issues are quickly resolved and not logged as bugs, which can reduce the time and effort on filing a bug (repro steps, screenshots and all that stuff), triaging the bug, and dev team reproing the bug and etc. Also, enforcing writing and reviewing unit tests and checking on coverage metrics are also contributing to the defect prevention. There are a lot of things you can do to prevent defects in your software. It's a good discussion to have with your dev/pm/tests and business. You'll normally have multiple checks before code gets deployed. And you'll actually find less bugs after deployment happens. But now when you find a bug on QA environment, you can think about how you could prevent that and add more checks. It's kind of addicting. I would say..

Speed
Speed is another the key part of Agile testing. Fast feedback on developer's new check-in.  Think about the time it takes where a new mistakes goes into the check-in and those mistakes are found in testing. duration between dev puting mistakes in his/her code and finding that mistake from test. Lots of articles and papers mention about pyramid and what kind of tests take place in that pyramid. It actually has to do with speed of test execution and creating targeted and controlled tests. Each level of pyramid defines purpose of its own. Another way of looking at this is to check the  Think of find the same bug in selenium UI tests vs. fast and furious unit tests. So if your test team’s primary automation strategy is something like selenium, think again. Seriously

Consistent feedback mechanism
No doubt about this. Continuous Integration! Every check-in of the code gets tested thoroughly from unit tests to UI tests.  Gated check-in using unit test coverage percentage and promotion model of build increase the quality of the build. Test team can contribute to reviewing unit tests, writing fast component-wise testing, UI testing using mocking or spoofing.

Skilled testers.
Agile testers do not only think like users. They do think like users and engineers. They are beef-up with tools. They can pick and choose whatever tool is necessary for the execution of test of any kind. Skilled testers not only tess the code but also make good recommendations to the dev team. Testability is one of the key components of good design of the application.

Now, let’s take a look at how to accommodate drawbacks of agile testing.

Context-Driven testing
One of the tendencies of agile testers is to think a lot on mechanics of the code. Functions and components of the application. But there are places where testing strategy matters. Because each and every testing problem has to do with its context. What are the best testing strategy for calculation heavy feature? (like data driven testing) What are the best testing strategy around state based application? (like model based testing). What are the best testing strategies around infinite inputs and outputs permutation applications? (random sampling, equivalent classing, pair-wise testing) Coming up with best testing strategy around given context actually matters a lot.

Systems thinking
 Since agile testing is working on small features and make iterative progress. Testers sometimes lose the big picture of the application. Systems thinking is a great way to look at your testing. Check this video out. https://www.youtube.com/watch?v=OqEeIG8aPPk  If you trying to improve the application by taking parts separately from the system, you’ll not be successful. Watch this video, you’ll get it.

Why do you like software testing?

Do you like software testing? Why do you like software testing?

This is my first interview question. I've heard various answers on this question. Of course, the answer will not greatly affect hiring decision but I think it's been hard to come by candidates who really impressed me on this question.

You test engineers. Why do you like software testing?
It will be a same question as to ask to musician why he/she like music. Or ask teachers and professors why they like teaching.

One of the greatest value of America, at least I believe, is that the culture here encourages people do what they like to do. You've seen movies like one become successful by chasing their dreams. Sports player, musician, writer, chef, teacher, and etc.

Is there any kid who wants to be the best test engineer? Not even kid. Will there be any college student major in Computer Science wants to be the best software test engineer? Even myself did not know much about software testing. I started as junior developer and if I did not meet the software test architect who got me into software testing, I might not be in software testing field.

One of the famous test engineers, James Whitaker, (I should say used to be a test engineer), preaching about software testing as dying profession. He is proud of running his org without test engineers. (calling SDETs developers). He's got his own reasons and many of his point make sense.

Anyway, let me address my reason why I like testing.

I like software testing because it is NOT straightforward. It's not like you need to do 100 things for small applications and 10,000 things for bigger applications and done with it. And it's not like you're good if you have a certain number of unit tests, functional tests ,integration tests and system tests. It's not about the quantity; it is actually about the quality. There are so many things to consider when we're testing something. Different aspects of problems that the application solves, technology that is used, business expectations, user behavior, priorities, threats, performance, dead line date, project team, process, and etc. Finding best possible testing strategy and execution around this environment is great fun. And every feature, every story(in agile) provides different testing problems.

I like software testing because of its exploring nature. I like the thinking process of how it(new feature or project) might not work in the system and problem that comes as a consequence. I love to apply the Systems Thinking on this. Many people say going over test cases are boring. But coming up with test cases (written or not) or trying something with some suspicion about the application is great fun. "I try this because I've seen similar things before. I try this because from my understanding this new thing might cause problem on certain features that are already exist." Understanding the interactions in the system and explore the possibilities of unexpected interaction issue is really a beauty of testing. And this process start from spec reviews (sprint planning in agile) and I get to know more and more and exploring goes to many interesting directions.

I like software testing because I have many different options to execute tests and there are tons of tools out there to choose from. I would say this is more of test execution fun. This is not about writing automation code I would say. Coming up with best test execution strategy is great fun. What kind of tool is useful for data driven testing? How about model based testing? What's best test automation strategy for agile process? Experience in several different test framework and tools. Quick experiment on new tools. What if we just choose to go with manual for testing this feature? How can we represent our test results and communicate with the team? This requires endless learning. Learn to design good test framework,  learn new languages and new tools, learn how to represent test results, learn how to optimize test execution time, learn, learn and learn.

How about you? Why do you like software testing?