A good test automation framework - Part 2

I previously wrote about test automation framework with details related to coding. (I would call it part 1)

Today, I like to mention a little bit more technical explanations of a good test automation framework.

I guess you (test engineers) have worked on test automation framework that was written by someone else and also worked on designing it from scratch. Have you feel some fuzzy or itchy feeling about the test automation you're working on? Something seems to be wrong but hard to explain? Does it take hours just to add a new test case? Do you feel test execution is heavy? I have the same feeling as well. I've been seriously thinking about that and I came up with following technical details. I don't think it is just because of  different coding style nor different thinking process. Hopefully, this blog actually scratch those itchy feelings. :)

1. Goal setting: This is about initial design of test automation framework. You need to set two goals. If you can satisfy both goals from framework user (testers average 2-3 years on automation), I would say your design is successful. The first goal is the framework should be easy to understand, easy to use and easy to implement. And the second goal is you need to design test framework in a way that volume of the code increases but complexity of the code does not. This seems to general and yes it depends on the context of your application. But if you write test automation framework from scratch, please do not simply start with drawing diagrams. Think about consumer of your framework. And also how the framework will grow and not become collection of giant and complex code base.

2. Do not re-factor test execution step code. This is for the user of test framework. I mentioned from my first blog to be cautious about it. Now I change my mind. Again, Do not re-factor test executions step code.  It really has many side effects.

First, your coding is restricted by re-factored code. You framework API provide several functionality and normally those are the components that supports your test execution steps. Your test execution code should be just mix and match of those components. Once you re-factor the execution steps, you are losing your flexibility of test case implementation. It's really silly to be proud of test execution step code  to be one function and it takes 6-7 parameters. Also, you lose the readability of test execution steps. Now how can you replace your test case document with your code? It's so hard to read. DRY (Do not Repeat Yourself) does not applied to test execution step code.

Second, you are adding unnecessary complexity by re-factoring code. I have a good metaphor to back up this argument. Let's say you want to test a car and you have test system that control every part of the car. But that system provides only one button for control. With one button click you need to come up with a way to start the engine, step on gas, break, changing gear, steering wheel control, lights, radio and etc. How would you control that with one button. Yes, you need to come up with some sort of Morse Code to differentiate the operations. "bip bip wait wait bippppp bip wait bippppp" that means turn the head light on. How about you have buttons for each operations. Engine button, break button, accelerator button, light button, changing gear button and etc.

Third, you don't gain much anyways. In test automation framework, what part of the codes are most frequently and most likely added and modified? It is test executions step code. Test cases are keep added and modified as the application grows sprint by sprint. Let's say you have 8 distinct test case which exercise different aspects of the application. Let's say test case 1 and 2 have 70% of the same code. And test case 2 and 3 share 60% of code a slight different way. Test case 3 and 4 shares 65% of code but also slightly different way and so on. It looks like a lot of codes are repeating. Now you start re-factoring. I can tell you; it not easy. What do you end up having? Excessive object composition or excessive class hierarchy of code. Or. you can see the same name of re-factored code with less parameters all over the place. Base class or helpers. Now look at your code after re-factoring. Lines of code you save is not much. And guess what new test case comes in and you need to re-re-factor your code.

Forth, it's not uniformly re-factored and not re-factored by responsibility or business logic. Most of test execution step code re-factoring happens due to repeating code. Not like automation framework code. Framework code is re-factored to achieve Open-Close principle or component decomposition. And it is normally uniformly re-factored because it has its own purpose. Refactoring is not simply putting repeating code in one place. Refactoring makes the code more maintainable and extensible. Existing test execution step changes in several different way, I tell you I've experienced..... It's hell.

Fifth, putting repeating code to base class causes weak cohesion and giant class (3000 lines of code kind of giant class). I'll explain this more in details later. I personally do not recommend people to use TestBase class because once you open the door, it's open for any code to go in. It stars with really essential pieces of code for bootstrap. And later add some more helpers. And later people start putting repeating test execution step code. And people start to add another base class that inherit from TestBse class. Adding if statements and switch statements. Where is the cohesion? What does this base class specifically do? Everything!


A good test automation framework continues....