Developing sustainable Page Object Model UI automation

Today, I like to mention about Page Object Model. It's very well known E2E UI automation framework design. And we can find lots of articles on the web. What I'm trying to discuss today is more about its API and how it evolves.

Page Object Model (POM) by itself has petty good attributes as a framework design. I've been using it for my E2E UI automation for web application and mobile application. I've found some useful things to consider when we start UI automation from scratch using POM and maximize the benefit of POM.

1. Maintain the loosely coupled structure

I think it is critical to make Page Objects loosely coupled with test code. Creation of Page object should be handled in other places and the only coupling test code should have is calling its function. This loosely coupled structure make it easy for PageObject classes adopting new device type or browser type. Strategy Pattern with Factory method is very common for this. This structure ensures new device type or browser type does not impact test case code.  Also it helps to handle UI differences for different locale and market/country by using PageObject subclassing. I think this is one of the key aspects which test automation adopting business requirements

2. Do not remove Mix and Match capability
As the automation progresses, it's very common to refactor repeating code. But this refactoring work should not block the flexibility of POM which is Mix and Match. I see this as a very common issue in using POM. "Well, it's repeating many places and we should do it in one place." This actually in many cases makes harder to maintain the test code. I recommend suite level refactoring rather than test base class refactoring(refactor in similar group of test cases). Definitely similar test cases can have duplicate steps. These refactoring should happen in that suite since similar steps may required. If we put these refactored code in test base class for any tests to consume, these refactored code will normally gets more and more complex as new cases/logics are added. And it becomes a burden for maintenance. The beauty of POM is Mix and Match. Refactoring comes after Mix and Match is done when repeating code/steps are identified. Good indicator of this issue is  checking the speed of writing a new test cases. Speed of writing new test case should remain pretty much same when Page Objects are well defined. Heavily refactored code make the test case writing slower and vulnerable to unintended side effect.

3. Use IKEA furniture building strategy for Page Object APIs
I got this furniture building strategy from this video. https://www.youtube.com/watch?v=llGgO74uXMI&t=3369s. And I think it has very interesting benefit of using this strategy. This normally benefits when you're in the early phase of developing UI automation using POM. In the early phase of the UI automation with POM, PageObject functions(methods) are implemented based on test cases selected at that time. And sometimes, it's hard to tell if the function defined is specific to that test cases or os generic. And in early phase, you can see some commonalities and duplication among PageObjects. We tend to think to refactor whenever we see duplication in the code. It just comes to us as a uncomfortable feeling not to refactor when we know there is duplicate code. I would say take a deeeeeeeep breath and let it go for a while. I know this can be hard. But try to let enough test cases covered and PageObject API implemented from those cases. And when you think most of important/mostly used APIs(functions) for each PageObject are implemented. No go back to the PageObject class and try to refactor. You'll be surprise how clear it is to refactor code with very clear logic. It's hard to tell how long we should wait. it's your call, but refactoring PageObject class API after a while really make straightforward and effective.

The furniture building strategy is metaphor the speaker used in the YouTube link I added. Basically, you do not want to tighten one side all the way when you build (maybe more correctly ASSEMBLE)  the furniture. You let it wiggled a while and you tighten both ends little by little. And when it is pretty stable and clear, you tighten both ends all the way.

I like sharing three points per blog.  Have a nice day!   

No more Development testing or QA testing. Just testing

Well.. It is 20018. It's been a while that I wrote something in here. I'm still working as an SDET. There has been lots of progress in software development and software testing. Let me go over the change I've noticed over the several years since last I post (around 2014)

Even if many different companies adopting different development process or strategy, I still believe testing by itself much needed in the development process (code, test and deliver). I still see the value of the unit test, integration tests, e2e test, load/stress tests and etc. I guess Agile development is kind of norm and on top of that many teams wants to go Continuous Integration/Deployment in their development strategy. And there are many success stories about not having  QA organization as part of their software development process.

I think the important question to ask is NOT who should do which kind of testing. Rather, which testing mechanism is placed for which part of development process. This seems obvious but I have not seen many cases where development team leader goes through the entire development process with the team and what kind of tests are required where.

Here is what I think the team should do.

First, the team (engineer leadership) should decide what kind of development process or strategy they are going to use including tests. This will cover from the code getting developed to the code being deployed to the production. All the necessary testing should be identified and decided.

Second, the team needs to confirm all the test infrastructure are all available for them to write and execute. Check if there is a way for developers to test all their new features and bug fixes confidently. Check if proper regression suites are defined and executed properly.

Third, the team should follow the process and do all the things defined for development effort and testing effort. And at the end of the cycle (whichever the team decides), evaluate the make improvement if necessary.

Now, who gets to do what would be up to the team. If QA team exist, you can allocate the work that QA team can do. But the main point is that development process has to come first and then dev team and QA team can take the necessary work to complete that.

Testing is important. All the necessary testing need to be done before going into production. It is just testing not dev team testing or QA team testing.

What about finding bugs? What about regressions caused by bug fixes or new features? I'll cover that on the next blog. But development process and testing associated with the process need to be defined and applied first.



 

Thoughts around test automation framework

Today, I like to mention my thoughts around test automation.

Test automation is something that I'm passionate about. I like to continue to learn and make myself better at designing, implementing, and maintaining test automation. Throughout my career, I've seen good successful test automation, and yes, I've seen some bad ones as well. And yes, I've made lots of mistakes and bad decisions. I also have some success stories. Here are some things I consider worth noted.

Goal of test automation is value added to the project team not test automation itself. At the beginning of my career as an SDET, I was very passionate about writing a good automation framework. I read many books about coding.  And I watched lots of YouTube videos about coding practices and design. I was just crazy about being good at writing code. I was obsessed with design patterns. I felt like I knew what a good test automation should be. I challenged senior SDETs on existing automation framework and loved to have serious design discussion with test architects at the company.

As I experience more and more, I started to realized that it's not just about writing well designed, maintainable, scalable and beautiful code. It's actually about understanding the role of test automation in a project or a company and provide maximum value out of it. I started to consider various things when I design test automation framework, such as project timeline, short term and long term solution, coding skills of other SDETs, area of focus, what dev team needs, context of the application or system, testability, lab test infrastructure and etc. Sometimes I had to come up with automation in a couple weeks from scratch with coverage of priority 1 test cases. Sometimes I had to modify existing test automation framework to make it easy for inexperienced SDETs or even for manual testers to use. Those work indeed were the right choices for that given situations. It's possible to write easy to use automation framework. It's possible to build test automation starting with short term solution and transform it into long term solution without major design change. The true masters of test automation understand how to write well designed, maintainable, and scalable code. But that's just a foundation. Their adaptability and execution can bring maximum value to the company in any given situation.

Now, I'm getting some feedback on my test automation framework from other SDETs. "Jae-Jin, I believe we should never use hard-coded value in out automation" or "Jae-Jin, why you don't refactor those repeating code?" "We should use XML for all inputs" Well, I can imagine how those Sr. SDETs felt when I challenge them. My response is "well, let's discuss about that..." Fun.. Fun.. Fun

Don’t forget “Framework” part of test automation framework. So what is framework? To me, framework is an agreement. The agreement on a certain development way or convention the team will use to implement the software. Of course, this agreement is mostly introduced by architects or more experienced engineers. Then what are the benefits of having framework? Obviously, the engineers can be on the same page when it comes to implement the feature. It helps communications among engineers like in code review. And It’s hard for a new comers to make mistakes since the framework defines what code goes to where. The most important outcome of using framework to me is “as more and more feature(for dev) or test cases (for test) comes in, the volume of the code will increase, but the complexity of the code will remain the same.” This is the beauty of using framework.

A good example of development framework would be MVC framework. When a developers create a new feature, they will follow the MVC framework and put codes in the right place. Model for data representation. View for presentation layer. And Controller for orchestration and actual business logic. So when you design test automation framework, you should have some sort of agreement that everyone understands. If you don’t have “framework” nature in your test automation, you might not have test automation framework. You just have test scripts.  Take a look at your test automation. Is it a true framework?

Writing test automation framework requires discipline. Why? It's a bit different from writing production code. Just simply think about it. For production code, there are whole dedicated testers and test teams testing that code. But there is no other test team to test test automation framework code. We're writing code to test other code, which means test automation should be more correct and right. How can we achieve this without someone testing test automation code? Actually this is a big challenge for SDETs out there.

OK then what's the advantage? Generally speaking test automation code does not have as high expectations as production code in performance,  algorithmic efficiency (big O) and memory utilization. Test automation does not always required to handle exceptions or errors gracefully. And normally the test case codes are specific intent and specific expected outcome, and sometimes test execution code can ignore things that are not in scope of that particular test case. So we need to be able to utilize these facts when we're writing test automation framework.

Here are my disciplines. I discipline myself not to be fancy with my test automation code. I know I passed all those crazy interview questions to join the company. I know I am capable of writing complicated code with very efficient algorithm. But when I write test automation code, I discipline myself not to be fancy and go simple. For example, let's say I can implement n-square solution and n-log-n solution for a given problem. If n-square solution is more straightforward to implement and easy to understand, I will go for n-square solution. Yes, this is really hard for me too. But it is important that I write less error prone code.

I discipline myself to minimize the usage of conditionals (if, switch statements). Again, it is to reduce complexity of the code. I've seen crazy test automation method that takes 7-8 boolean, enum, objects as parameters. Oh.. man.. the complexity of that code had gone worse and worse. I rather have two separate method than have one method with boolean parameters. It looks fine at the beginning. But as time goes by, boolean parameters becomes enum parameters. One parameter become 2-3 additional parameters. There are some cases where it is necessary. But most of case, I rather choose redundancy over complexity. And if you try to write code without conditionals, it becomes more object oriented programming.

I discipline myself to be open-minded. I let test cases drives automation structure not my preference. I discipline myself not to be obsessed with my own design. When new test cases comes in and it does not fit in current automation design, I will not force the test cases to fit in current automation. I will change the automation structure to fit in new test cases. In other words, I will not let complexity of code grow because of new test cases. I will change the automation structure or design to keep the complexity about the same level.

Well, I'm getting sleepy.. happy testing!  

5 minute software testing tips

Today, for the first time, I create a video about software testing on Youtube!
I don't know how this will work out, but hey.. why not.

5-minutes software testing tips.. :)
Will post more and more~~

Dealing with emotion, people, and yourself in software testing

Today, I like to write about stuff around work place and somewhat related to testing.

We love testing! Don't you love doing testing all day long? I do. I really do. Well, but since it is a job, we all have crappy days and good days. Have you experience these? Developers resolve a bug by-design even if you mentioned all the details of the issue? Someone keeps bouncing bug back to you even if that's not your issue? Someone blames you on something via email and cc his manager, your manager, and some other important people?  You made a huge mistake, but your mistake gone away due to other big issue? You are just totally lost and simply take all the blame? Find a huge bug by accident? You've been keeping test automation or test case really well, but just one time when your manager sees your work, everything went wrong? Troubleshoot issue in a few second and find the root cause? Someone you believe not a very good developer/PM, but they save you from disaster? Give or take some condescending comments? You will be so happy if that guy does not work here. You'll be so happy if we don't need to do this? You get so angry about peer performance review?

I think we're so human being. Don't you think? It's not only you. It's everybody.. Right?
Here are things I think you might consider when you're dealing with people and emotion.

Do not criticize anyone publicly
Nobody likes criticism. Even if you're 100% absolutely right, do not criticize anyone in public. The other person will never appreciate your criticism. He will become very defensive. He'll try to find justification for himself and other to believe. And one day he will criticize you in public in return. I had several incidents around this. I had someone criticized me and cc'ed my manager or group alias. Oh man. It was hard to take. I came up with 100 reasons why that happens to justify the issue to myself. But I also have criticized some people with email. I thought I was cool. I email back with my awesome criticism to senior engineer's original email and yes REPLY ALL. I thought I had confidence in myself and opinions about things.. Like philosophy in testing. As a result, I've got very bad review that year. I can sense people did not like me when I raise my voice to debate. I'm not saying you need to be nice to other people to get good review. I'm saying you being confident/strongly opinionated person and you criticizing other people publicly are different thing. You can be confident and have strong opinion about everything without hurting others' feeling. When you receive dumbest email or unreasonable claim or blame from others, think how you can point out the issue without hurting others' feeling.

Leave your ego under your desk when you argue over a bug
I still get surprised by how people interpret and react to bugs. Some fight over the priority. Some doubt about severity of the bug. I've fought over bugs many many times. I have used "user perspective" cards or "terrible potential risk" cards to convince people. And yeah... I argued sometimes just for my EGO. I respectfully(?) respond to "You tester, stop breaking things!" with "I don't break things. I find things YOU broke!" But I think bugs are not our baby. It's a statement that indicate the issue which may or may not be serious impact to product/customers. It can definitely be interpreted differently to different people. Finding a bug will be our job, but how to react to the bug will be project team's responsibilities. Project management/business perspective has to be understood and development impact or effort should be understood as well. When I talk about the bug, I always try to remember what James Bach said in his talk. "Hey developer, don't think bugs as your mistake. We, testers, do not fight for bug to criticize your work. We want you to shine. It's like mother saying 'you have mustard on your mouth' when you leave the house for a date. We care about you. We want you to wipe out mustard on your mouth and shine." If you leave your ego under the desk, discuss facts around the issue and try to understand other people's perspective, it gets much smoother and easy.

Think big. Think big.
When I was a junior SDET, I was really scared of making mistakes. I get offended by people keep pointing out my mistake. I made some  automation framework changes to make it much more effective. Overall, it was great change, but it was not perfect. There is always someone point out flaws and criticize my work. I was a bit discourage to try new initiatives or change. Sometimes I got to the point "why bother." But as I gain more experience, I started to ignore those criticism. I've done many more presentations even if there are always people find flaws in my idea. I've brought so many new successful initiatives at my work even if there are people in doubts. Dr. Russell Ackoff, who is my favorite modern theorist in Systems thinking, mentioned about errors of commission and errors of omission. Error of commission is a mistake that consists of doing something wrong. Error of omission is a mistake that consists of not doing something you should have done. At work, errors of commission are visible, but errors of omission are not visible at all. But I think we should more concern about errors of omission. We should think big. Do not worry too much about small mistake you make when you doing great work. And don't worry about people's doubt on your idea if you believe it would work. We are living in this advanced and modern world, not because of people who are in doubt, but because of people who challenged status quo.

     

How to think while we're testing

Today, I like to write about something I read recently and address how it might impact our testing work.

First, I want to talk a little bit about how our brain works. Well.. Yes.. I'm no expert in neuroscience and my knowledge is limited to couple of books I read and some Googling. For the reference, what I'm about to explain came from this book, "Thinking, Fast and Slow" by Daniel Kahneman.

I wrote a blog post about assumptions and how it affects testing. link And I used James Bach's calculator testing example. I want to go a little deeper than explaining how assumptions affects testing. I want to understand why we make assumptions.

According to the book, "Thinking, Fast and Slow", humans have two different type of thinking systems. ....(BTW, I'm not saying contents of this book are all 100 % correct. It is based on credible research, and I don't think he won Nobel prize by providing BS) One is called "System 1" and the other one is "System 2". System 1 is so-called heuristic brain and System 2 is so-called rational brain. System 1 is quick, responsive, and sometimes fallible. System 2 is slow, lazy and requires energy and focus.

Here are some examples. System 1 is used when you're doing things like calculating 2+2, driving home from work (you've done many times before), brushing teeth, understand facial expression, instantiating array or string in your most comfortable programming language and etc. Basically, you've known and done so much that it does not require you to think much.

And System 2 is used when you're doing things like calculating 27*69, driving on not familiar road (without GPS), solving puzzles, writing letters, design the architecture of network system and etc. You can imagine these activities require your attention, deriving from some basic knowledge, connecting several dots of your knowledge (active interaction among neurons).       

And usage of system 1 and system 2 is very optimized. When you see 27*69 your system 1 recognize this cannot be done and ask system 2 to take over to process that. And after repetitive usage of task from System 2, System 1 can take some of the task. For example, learning chess game requires System 2 at the beginning, but once you are really good at chess game, lots of move you make can be done through system 1.

So why am I talking about this?
The calculator problem (from this blog) is basically a check for the consequence of fallible System 1. If System 1 process the problem and reaches conclusion, the problem will not move to System 2 unless you're forcing it to. And we, as test engineers, need to practice pushing our thought to System 2 while we're working. Being critical all the time is very exhausting. It requires discipline and lots of, lots of energy.

Sometimes, being on the same project for so long can prevent you from finding important bugs. Your System 1 does not give chance to System 2 to process thoughts without letting you know. You are a domain experts. You've seen lots of bugs along the way. But it can be dangerous.

So how do we force ourselves to use System 2?
I have a couple of suggestions

1. Call out your assumptions. 
It's an interesting check point. Let your brain to process flow of information about the application you're about to test first. You will think about use cases, testing strategies, test cases, test executions, report and etc. And then going over your documents or thought process again with critical mind. Try to find any assumptions you made. You're already using System 2.  

2. Focus on how rather than what.
Build your System 1 with "I need to think how to test this first, not what to test." Yes, it is important that you execute various categories of testing. Functional testing, localization testing, usability testing, load testing, performance testing, test automation, BVT/Smoke test and etc. are all important. And it seems very structured. However, it restricts you thought process to be in that category. Think about what's important, what's critical, what's necessary and how to address those with my testing. You make good use of System 2.  It will help you test is more effectively. And it will be just too fun. 

3. Be a critical thinker.
Here are three question.
"Huh?", "Really?". "So..." http://www.youtube.com/watch?feature=player_embedded&v=8TX6rzz60xQ

Problematic nature of software development

Interestingly, the nature of software development process imposes a very unique difficulty compared to other engineering process. And it is generally overlooked. That is incremental engineering effort on previous product. In other words, ‘versioning’ And I believe this particular fact actually causes lots of trouble to software development teams. Throughout years of experience, software project teams have come up with several good solution for these. I’ll explain how this trouble leads to those solutions later point. There are many books and articles about building a good software product from architectural perspective. And often times we can see that authors use metaphors like building skyscrapers and bridges to explain their points. However, building software is still significantly different from building skyscrapers or bridges even if we consider planning, designing, engineering effort, duration of work and delivery are conceptually the same. If I were to use metaphors to explain software development process, I will describe it as following with a little bit of exaggeration to point out the difference. ‘Software development is like finishing Empire State Building today and adding 50 more stories two years later. Software development is delivering Golden Gate Bridge today and adding 16 more lanes next year.’ Do you see the trouble here? Let me explain with some more specific example. Let’s say Boeing delivered ‘787 Dreamliner’ airplane a few years ago. And they are planning to build ‘797 Wright Brothers’ as the next generation of the product line. The requirements for this airplane is basically adding 250 more seats inside airplane. But Boeing engineers only allow to build 797 by physically modifying existing 787 airplane. What has to be changed? I don’t know much about plane, but I can imagine that they have to decide if 797 will be fatter or longer, which changes aerodynamics. It will also affects the size of wings and horizontal/vertical stabilizer. It will changes the weight of the airplane, so engine has to be changed. Then it might require extra gas tank. It requires more power supply and oxygen. It may require longer runway for taking off and landing. They also need to consider hitting by lightening or dealing with turbulence. The need to consider not only the airplane structure but also travelers. Extra bathrooms, emergency exits, flight attendant station have to be changed. Seating for first class and business class might be changed as well. I guess we can go on and on. And remember we’re modifying existing 787 physically. This is the nature of software development. There is no hard end product like buildings or bridges in software world. It is just the nature and we'll continue to do this. Reasonably small requirement for business can impact engineering effort significantly. Adding or modifying existing feature exposes various risks in the software product. Some feature may not be needed any more or doesn’t make sense any more, but it’s not easy to take out those dead code or features. Let’s think about how we can minimize these risk. On the next blog. :)

Testers! Earn respect, not demand

I’ve seen many cases where testers complains about how they are treated in the project team. “We are NOT a second-class citizen in this project team!” “They don’t know how difficult it is to test software!” “Testing is not an activity of try and see if it works!” Sometimes there are some conflict between QA/test team and development team. QA team present estimate for testing some feature in the meeting and get offended when development team asks, “Would it take that long? Can't you automate that?” Or QA team shows all the bugs they found and questioning quality of the code.

I believe some sort of stereotype or perception actually comes with occupation. Maybe history of software testing discipline was not well received from the beginning. The point that I’m trying to make here is that we should not blame perception about the occupation. What matters is how you perform your job. And actually you can take advantage of the perception and use it for your own benefit.

Let me give you an example. What is your perception about a gardener? You would think that a person who works as a gardener mostly is not very educated, and it’s kind of work you don’t do, not because you cannot do but because you don’t have time to do or don’t want to do. I don’t think you would respect your gardener for the occupation, but you would respect him as a human being. You still have perception about the occupation.

Let’s say one day your gardener came to you and provide you following information. He measured PH of your garden soil of front and back yard and asks you to let him apply some nitrogen to prevent clover or lawn fungus. And provide you exact dose of nitrogen he will apply based on square footage of your yard. He also provides you the information about the growth of each garden tree along with the climate of your city, and informs you that the branch of front yard tree will block certain percentage of sunlight to your living room in next summer. And he shows 3D pictures of possible garden looks with some flower beds, new trees or relocation of your existing trees and asks you if you would like to try. He always come to your house on time and do the best garden work. He cleans up every single corner of the house and always nicely fold your water hose and sweep the front side of your house. Your garden grass, flowers and trees are all healthy and well maintained, and your garden looks the best in your neighbourhood. People visiting your house always mentions how nice your garden is.

Now, would you respect his passion and professionalism as a gardener? Would you continue to have him maintain your garden? Would you recommend this guy? Would you stand when someone bad mouth about gardener being lazy and not educated?

Respect is not something you demand and get. You EARN respect from others. What is important in your testing job is doing your work with your passion and professionalism. You can differentiate yourself from others by continuously improving yourself and trying innovative approach. When you do that, perception of the QA role become meaningless. People will respect your passion and professionalism. They already have low expectation, but you exceeded far above of their expectation. Guess who gets the promotion?     

Leadership for a tester - part 3 (people)

Let's talk about the last one. People leadership.

Let me mention this first. If someone asks me to recommend one book that explains pretty well about software development/testing leadership, I would pick "Notes to a Software Team Leader" by Roy Osherove. I really appreciate every point he made in his book, and I totally agree with him. Awesome book. You should read it. He is one of my favorite guys of software development speaker. Like uncle Bob, Martin Fowler and Scott Hanselman. I've really enjoyed watching all his "Unit testing, TDD, team leader manifesto, and etc." YouTube videos. His talk is not as exciting as uncle Bob or Martin Fowler, but I respect his talk because I can feel the pains, thought process, lesson learned, experiment, and effort he went through from his own software development experience. OK. Enough for the guy. Let's start

I always wanted to mention this first when I blog about people leadership. Here goes. I'm sure you've seen lots of articles like '10 common thing that amazing leaders do', '5 habits of great leaders, 10 signs of a great leader and so on. (seems like leadership article should have some sort of list). Leadership is not some attributes added to who you are. It is actually who you become. It requires genuine effort to motivate people and to convince people with what you believe. People are not stupid. Don't believe that everyone in your organization is practical. More money and higher position may keep some practical people. But that does not keep genuine and professional people. You have to be genuine.


Don't be an assh**e because you're smarter than others. You may become a leader because you stand out. You're smart and you do your work more efficiently and fast. Being an assh**e is different from being charismatic. Charismatic leader scolds at people when the value or vision gets disturbed. Ass**e scolds at people when his/her own expectation is not met. Don't think people will respect you because you're simply smarter than them. I like the quote of Jeff Bezos (Amazon) at Princeton Univ. "Cleverness is a gift, but kindness is a choice. Would you bluff it off when you're wrong or would you apologize?"


People leader knows and understand her people individually. She understands the strength and weakness of her people. Some people needs coaching. And some people needs freedom. If she goes up, she understands each team's strength and weakness. She focuses on progress and think about how to help the progress rather than seeing the result and judge. Hitting the milestone is easy to check, but understand what kind of problem gets solved and know what kind of barrier there is are hard.

Well.. I might update this post later. I've got some more to write but I'm a little tired. :)

Leadership for a tester - part 2

Operational leadership

You can easily imagine that great strategy with poor execution cannot lead to a good result. However, great execution in software testing is somewhat hard to define.  We can do all sort of things as part of testing but we cannot clearly know all the effort actually 100% contributed to achieve this big ambiguous goal, 'excellent quality of software.'

To give you some idea, let me give you a question. Which tester do you think is more productive or effective? One who finds lots of good bugs or the one who can prevent from bugs being created? The one has in depth coding knowledge of the application or the one who deeply understands and knows users/customers usage and expectation? One who uses test process/methodology that has been working successfully or one who wants to try unproven new process/methodology that possibly benefit testing effort? One who found 20 pri2 bugs or the one who found 2 critical/pri1 bugs?

What do we do to measure the testing effectiveness? I've seen many times that test managers seriously going through irregular bug burn-down chart categorized by priority/severity or test story points fluctuation to get some sort of meaning out of it. I know data will not lie. But do those data actually represent the effectiveness of testing? I would say no.

Operational leader executes and delivers based on testing strategy. 

I like to emphasize one more time on "simultaneousness" Poor strategy ruins excellent execution. This is like building an excellent ship to go over a mountain. Poor execution makes excellent strategy a day dream. It's like a innovative and passionate person leading a team of complainers "that does not work in this company. you just want to do this to get visibility from senior management....."

Operation leader understands the strategy clearly and execute towards that. And operational leader helps to form excellent strategy.

Based on assessing the context of product/team/timeline/etc, if the testing strategy for current product is more towards CI and pyramid(ton of unit tests,  many integration tests, a few UI/E2E tests),  she will generate code coverage data from unit tests, come up with gated check-in strategy, gather feedback cycle and trends of integration tests and UI tests. She will continue to monitor the progress and tries to optimize the testing effort. Staffing will be leaning towards to the strategy as well. She will clearly deliver the importance of the strategy and ask her team to be best at those areas.  If the testing strategy is more towards exploratory execution and user-centric, she will discuss with the team to come up with session-based exploratory testing plan, ad-hoc testing plan, think outside of box exercise, crowd sourcing, dog fooding and etc. She will generate data relevant to exploratory testing area, buggy feature, or qualitative confidence. She will also categorize the user based on sex, demographic, educational level, and etc to demonstrate the exploratory nature of the testing effort.


Operational leader respects professionalism.

Operational leader put her heart on everything she is doing. As a professional test engineer, she will do her best to accomplish any task. Brainstorming on test cases. Writing bug report. Writing single line of automation code. Discussion during test plan/strategy meeting. Discussion with Dev team and PM team. Solid execution and delivery. No BS. That's an operational leader.  She is also looking to improve the process or methodology continuously. Just set herself to be the best in the industry, just a pure professionalism.

Often times, operational leader loves the work. Sometimes she dreams about work. This might sounds crazy but actually that happens. It's not like getting stress from work. Imagine, you are fall in love with video game and playing all day. And you're stuck at level 12 and you need to go home. Would you think about it? Dream about it? Do you want to be the best tester? then you should love the testing work. In any occupation, one that loves work always out perform one that works hard.  Furthermore, testing is a special occupation. If you don't value your work or don't find interest, your life is hell in the company.

So far, we talked about strategic leadership and operational leadership. I'm a little tired. Will do the people leadership on the next blog.

Leadership for a tester - part 1

Today, I like to write about leadership.

For those genius techie testers, leadership might not be very attractive topic. You already getting respect for your technical skills and delivery so you don't have to worry about being a manager.

But seriously, do you only need a leadership skill when you become a manager? I've seen lots of leaders who don't have manager or senior title. Wether you're planning to go managerial route or individual contributor route, you need leadership skill. Leaders are influential. And good companies know that and use if for measure of promotion.

There are tons of books and articles about leadership. And those are mostly written by business people (e.g. MBAs) and for the business people (e.g. MBAs). Lots of them are the same concept with a little different flavors. I found a good one yesterday. And I've been thinking about it all day to relate to my lovely work, software testing.

Here is the simple quote that explains.

"A good leader has three kinds of leadership, and he/she should be excellent at all of them simultaneously. Those are Strategic leadership, Operation leadership and People leadership."

Let's relate this to our testing world. And I want to point out one more time, good at all those three simultaneously. 
  

Strategic leadership

It's about the decision on how you're going to test your software. It requires a good understanding of what you have, what you can control and what you can do. And you make decision and execute on that. This applies to the one who is responsible for entire test organization of the company, the one who is responsible for one testing department, or the one who is responsible for a story testing. And you can guess that the decision will impact based on what you're responsible for.

I really think context-driven testing make sense in this leadership because what you have, what you can control and what you can do differ from companies to companies. Even if two companies producing similar software product, testing strategy cannot be similar. Your company/organization will have people who has different skill set. Company/development culture can be different. Process adoption and maturity are different. You might or might not be able to work with offshore. Testing strategy the company has been using might be different. And so on and on..

However, there are things we can do to make sure testing strategy works.
First, you need to communicate your testing strategy. If you're a test director or manager, you need to communicate your strategy to all of your subordinates and make them understand. Why we're doing this way and what we're expect to get out of our current strategy. If you're an individual contributor tester, you need to communicate your testing strategy with your scrum/project team. "This is how I'm going to test this story and this is why." By doing so, you are informing your plan and open  for feedback. A great strategic leader make sure others understand what're he/she is trying to do and why. And provide a way to get feedback. If you as an IC and feel that your test manager/director just care about numbers and graph, you need to challenge them to provide strategy.

Second, you need to have a mechanism that shows progress or results of your decision. This happens naturally if you open your feedback channel. For test directors and test managers, you need to check with your direct reports to understand how the strategy is working. Often times, you'll see issues, overhead or redundancy in your strategy. And for ICs, you can see how your strategy is working by going through several iteration of work(sprints/milestone). I know measuring testing progress or success is hard. (I'm planning on writing one post about this) Sometimes it's not about numbers or graph, people notice the benefits and drawbacks. You need to check your strategy progress/result and make adjustments to make perfect fit for your situation.

Third, you need to study, research and experiment a lot on strategy. Innovation does not come out of nowhere. Hear other companies stories, your subordinate's experiences, and feedback. Have a meeting with your directs to make better. Work with PMs and developers to understand their views and progress. Continuous improvement is part of strategy effort. And make sure you're doing the right thing, not doing things right.

And lastly, once you're clear on strategy, you should believe in it and execute the hell out it.    

I'll write operational and people leadership in following post.

Systems thinking and test automation

Today, I like to share some of thoughts around test automation related to Systems Thinking.

You as an SDET, it is very important to make your team understand what your test automation strategy is. When it is clear, your team can support you and align their work with your automation strategy.

OK. let's start. To be sure, I am specifically writing about functional test automation here. And, in fact, I consider test automation as regression tool not as testing activity. I like James Bach's term 'checking' to describe automation. However, I do believe that test automation as regression tool has great value.

My test automation strategy is based on 'Systems Thinking'. Dr. Russel Ackoff came up with this idea and there there has been lots of conference around this.  I posted his video on my blog, so you can watch his video.

There are two principles that I like to mention from his talk.

First, when it comes to the system, as a whole, it creates a unique behavior or property that none of its parts have.

Second, System's unique behavior or property is created by the interaction among individual components(parts) NOT by sum of individual parts behavior.

As Dr. Ackoff explained, let me take a CAR for example.
What is CAR's unique behavior/property? That is carrying you(person) one place to another. It is only unique as a system. Let's take a look at each parts of the car. Engine, transmission, door, battery and so on. None of them can carry you one place to another. But that unique property cannot be created without parts. More specifically, the interactions between parts creates that property. You can easily understand that simply having the parts without interaction is not contributing to produce a unique system's property.

The point I'm applying to my test automation strategy is the INTERACTION.

Here is one more car example. Let's say, your car stopped while you're driving. And let's assume, it is because the engine is stopped. Yes. it is obvious that broken engine caused car to stop. But if you think a slight different perspective, car actually stopped because no force or thrust was delivered to the wheel. In other words, the interaction was broken.

And now, let's say your car is stopped, but you don't know what caused it. What do modern car manufactures do to identify issue?  Of course they are using some sort of sensors and chips. And it alerts when any sensor detects something that is not normal or when certain criteria they set is not met. And then mechanics are going after actual root causes based on that information.

I think SDETs job is to put sensors all over the code using automation. Sensors that detects all the small functionalities  that comes from interaction between components and modules. Sensors that detects component/sub-component behaviors that comes from interactions of classes and methods (unit tests - this is normally written by dev).

What this thorough automation provides a simple value. Make easy/straight bugs to be found easily and hard/mysterious bugs to be found hard.  Sounds simple, but this will give tremendous trust to your work and your test automation.

Testers are like caddies in some sense

Today, I like to describe testers with some odd(?) metaphor.

I think testers are like caddies in some sense. More specifically, the guy who walks with Tiger Woods or Phil Mickelson on the field of Professional Golf Tournament. The caddies of PGA tours. 

Why? I'm going to explain why.. 

First off, Caddy is an occupation. Golf player is an occupation as well. Caddie might not get attention as much as the players, but still he/she plays very important part of professional golf tournament. It sounds like testers in software development, doesn't it?

Here is one thought on being testers. I've seen so many testers who complains a lot about the company not treating them as important as developers. Being treated like second class citizens or getting paid less. Again it's an occupation. You choose to do it. If you care about being treated like developers, then you should be a developer. If you do not enjoy testing, you should not do testing. Perception is really hard to change. If you care what others' think about you, well..I don't know. I think you should be yourself.

 However, I truly respect those testers who has been fighting for the importance of testing in software industry over the years.  

OK.. let's start.

Caddies understand the game of golf as much as players on the field. They know general information about the game like the golf course, golf rules, difficulties of each holes, distance of each hole, location of hazard, trees around the hole, bunkers and roughs. And they also know what affects each shot. They understand things that affects the shot like winds, uphill, down hill, rough, bunker, reading greens, soft or hard green, wet grass or dry grass and etc. Coming back to testers. They should also know about general information about the product/service they're testing. What kind of product it is (web, mobile, desktop and etc.), who are the users, what kind of technology the team using, what features are important for the business, what development process is used (agile, kanban or waterfall) and so on. And they also understand what affects the developments like software architecture, code review, check-in process, bug triage, regular build process, regression strategy, reporting mechanism and so on. This is really basic stuff. Good caddies and good testers take seriously about the basics of their work.

Caddies provide good information for each shot players make. You may have seen this on TV. PGA golf players always discusses with a caddy before he/she makes shot. Caddies and players both check the distance, hole location, hole surroundings,  ball location, wind, grass, slope, curves, trees and so on. That brief conversation before the shot is the result of quick analysis. Testers are similar in that sense. Test plan/strategy should show the analysis of the feature development. Whether it is formal or informal, testers should analyze the feature that is going to be developed and discuss with developers. Here are some examples. User impact, complexity of the component, impact to existing code, risk, timeline, algorithms or implementation plan, testability and so on. This requires a lot of studying and experiments. Testers should continuously learn how to do better analysis and improve.

Caddies know players more than players know themselves. Professional caddies know his/her players so well. Strength and weakness. Ups and downs of emotion. Distance of each club. Fade and hook tendencies. And they know how to communicate with those things with players. Sometimes caddies insist on certain club because they can predict what his/her player will hit. This is really important part for testers as well. Testing is not just having a list of things and execute them. Knowing the context is the key. That's why I love the context driven school guys. Testing never starts from nothing at the beginning. Product has history. It has business environment and developers. Testers should try to understand developers' coding style and thought pattern. It really requires a close attention to each developer, but it really pays off. Thought patterns are amazingly accurate to predict. Sometimes you can even guess what a developer would  implement this particular class or functions. Bug trends as well. Who makes what kind of bugs can be understood by testers.

Well. I'll stop here.. But there is actually a big difference between caddies and testers. Actual testing part. Testing performed by testers are not there in caddie's work. I'll address that later..

               

A good test automation framework - Part 2

I previously wrote about test automation framework with details related to coding. (I would call it part 1)

Today, I like to mention a little bit more technical explanations of a good test automation framework.

I guess you (test engineers) have worked on test automation framework that was written by someone else and also worked on designing it from scratch. Have you feel some fuzzy or itchy feeling about the test automation you're working on? Something seems to be wrong but hard to explain? Does it take hours just to add a new test case? Do you feel test execution is heavy? I have the same feeling as well. I've been seriously thinking about that and I came up with following technical details. I don't think it is just because of  different coding style nor different thinking process. Hopefully, this blog actually scratch those itchy feelings. :)

1. Goal setting: This is about initial design of test automation framework. You need to set two goals. If you can satisfy both goals from framework user (testers average 2-3 years on automation), I would say your design is successful. The first goal is the framework should be easy to understand, easy to use and easy to implement. And the second goal is you need to design test framework in a way that volume of the code increases but complexity of the code does not. This seems to general and yes it depends on the context of your application. But if you write test automation framework from scratch, please do not simply start with drawing diagrams. Think about consumer of your framework. And also how the framework will grow and not become collection of giant and complex code base.

2. Do not re-factor test execution step code. This is for the user of test framework. I mentioned from my first blog to be cautious about it. Now I change my mind. Again, Do not re-factor test executions step code.  It really has many side effects.

First, your coding is restricted by re-factored code. You framework API provide several functionality and normally those are the components that supports your test execution steps. Your test execution code should be just mix and match of those components. Once you re-factor the execution steps, you are losing your flexibility of test case implementation. It's really silly to be proud of test execution step code  to be one function and it takes 6-7 parameters. Also, you lose the readability of test execution steps. Now how can you replace your test case document with your code? It's so hard to read. DRY (Do not Repeat Yourself) does not applied to test execution step code.

Second, you are adding unnecessary complexity by re-factoring code. I have a good metaphor to back up this argument. Let's say you want to test a car and you have test system that control every part of the car. But that system provides only one button for control. With one button click you need to come up with a way to start the engine, step on gas, break, changing gear, steering wheel control, lights, radio and etc. How would you control that with one button. Yes, you need to come up with some sort of Morse Code to differentiate the operations. "bip bip wait wait bippppp bip wait bippppp" that means turn the head light on. How about you have buttons for each operations. Engine button, break button, accelerator button, light button, changing gear button and etc.

Third, you don't gain much anyways. In test automation framework, what part of the codes are most frequently and most likely added and modified? It is test executions step code. Test cases are keep added and modified as the application grows sprint by sprint. Let's say you have 8 distinct test case which exercise different aspects of the application. Let's say test case 1 and 2 have 70% of the same code. And test case 2 and 3 share 60% of code a slight different way. Test case 3 and 4 shares 65% of code but also slightly different way and so on. It looks like a lot of codes are repeating. Now you start re-factoring. I can tell you; it not easy. What do you end up having? Excessive object composition or excessive class hierarchy of code. Or. you can see the same name of re-factored code with less parameters all over the place. Base class or helpers. Now look at your code after re-factoring. Lines of code you save is not much. And guess what new test case comes in and you need to re-re-factor your code.

Forth, it's not uniformly re-factored and not re-factored by responsibility or business logic. Most of test execution step code re-factoring happens due to repeating code. Not like automation framework code. Framework code is re-factored to achieve Open-Close principle or component decomposition. And it is normally uniformly re-factored because it has its own purpose. Refactoring is not simply putting repeating code in one place. Refactoring makes the code more maintainable and extensible. Existing test execution step changes in several different way, I tell you I've experienced..... It's hell.

Fifth, putting repeating code to base class causes weak cohesion and giant class (3000 lines of code kind of giant class). I'll explain this more in details later. I personally do not recommend people to use TestBase class because once you open the door, it's open for any code to go in. It stars with really essential pieces of code for bootstrap. And later add some more helpers. And later people start putting repeating test execution step code. And people start to add another base class that inherit from TestBse class. Adding if statements and switch statements. Where is the cohesion? What does this base class specifically do? Everything!


A good test automation framework continues....
 

Defining Agile Testing

Today, I like to define what exactly agile testing is. And also I like to mention drawbacks of agile testing and how to accommodate that. If you find something missing please reply on the comment section.

Following seems to be the unique parts of Agile testing. It's an interesting mind shift if you've been working as a tester in Waterfall environment.

Testing effort on defect prevention.
This is one of the fundamental differences between Agile testing and testing in Waterfall. In Agile testing, test team puts their effort on creating good quality builds before it gets to QA team (before deployment) whereas Waterfall testing puts efforts on after it gets to QA team (after deployment). This seems just a small difference but it has quite valuable implications. First, for test team to find issue before dev code gets deployed to QA environment, there should be a good communication channel between dev/pm/test and even business. Test team can help dev team not to make any assumptions about unclear spec. Test team can provide potential impact to the application due to current development(risk assessment/analysis). Test team can also provide some scripts or test data that dev team can run and use on their local environment.  Dev team can do a simple demo to the test/pm/business to verify his/her progress. All these to be happened, barrier among all disciplines has to be removed and all has to simply work as a team. I've seen a lot of issues are found during those informal discussions and demos. And those issues are quickly resolved and not logged as bugs, which can reduce the time and effort on filing a bug (repro steps, screenshots and all that stuff), triaging the bug, and dev team reproing the bug and etc. Also, enforcing writing and reviewing unit tests and checking on coverage metrics are also contributing to the defect prevention. There are a lot of things you can do to prevent defects in your software. It's a good discussion to have with your dev/pm/tests and business. You'll normally have multiple checks before code gets deployed. And you'll actually find less bugs after deployment happens. But now when you find a bug on QA environment, you can think about how you could prevent that and add more checks. It's kind of addicting. I would say..

Speed
Speed is another the key part of Agile testing. Fast feedback on developer's new check-in.  Think about the time it takes where a new mistakes goes into the check-in and those mistakes are found in testing. duration between dev puting mistakes in his/her code and finding that mistake from test. Lots of articles and papers mention about pyramid and what kind of tests take place in that pyramid. It actually has to do with speed of test execution and creating targeted and controlled tests. Each level of pyramid defines purpose of its own. Another way of looking at this is to check the  Think of find the same bug in selenium UI tests vs. fast and furious unit tests. So if your test team’s primary automation strategy is something like selenium, think again. Seriously

Consistent feedback mechanism
No doubt about this. Continuous Integration! Every check-in of the code gets tested thoroughly from unit tests to UI tests.  Gated check-in using unit test coverage percentage and promotion model of build increase the quality of the build. Test team can contribute to reviewing unit tests, writing fast component-wise testing, UI testing using mocking or spoofing.

Skilled testers.
Agile testers do not only think like users. They do think like users and engineers. They are beef-up with tools. They can pick and choose whatever tool is necessary for the execution of test of any kind. Skilled testers not only tess the code but also make good recommendations to the dev team. Testability is one of the key components of good design of the application.

Now, let’s take a look at how to accommodate drawbacks of agile testing.

Context-Driven testing
One of the tendencies of agile testers is to think a lot on mechanics of the code. Functions and components of the application. But there are places where testing strategy matters. Because each and every testing problem has to do with its context. What are the best testing strategy for calculation heavy feature? (like data driven testing) What are the best testing strategy around state based application? (like model based testing). What are the best testing strategies around infinite inputs and outputs permutation applications? (random sampling, equivalent classing, pair-wise testing) Coming up with best testing strategy around given context actually matters a lot.

Systems thinking
 Since agile testing is working on small features and make iterative progress. Testers sometimes lose the big picture of the application. Systems thinking is a great way to look at your testing. Check this video out. https://www.youtube.com/watch?v=OqEeIG8aPPk  If you trying to improve the application by taking parts separately from the system, you’ll not be successful. Watch this video, you’ll get it.

Why do you like software testing?

Do you like software testing? Why do you like software testing?

This is my first interview question. I've heard various answers on this question. Of course, the answer will not greatly affect hiring decision but I think it's been hard to come by candidates who really impressed me on this question.

You test engineers. Why do you like software testing?
It will be a same question as to ask to musician why he/she like music. Or ask teachers and professors why they like teaching.

One of the greatest value of America, at least I believe, is that the culture here encourages people do what they like to do. You've seen movies like one become successful by chasing their dreams. Sports player, musician, writer, chef, teacher, and etc.

Is there any kid who wants to be the best test engineer? Not even kid. Will there be any college student major in Computer Science wants to be the best software test engineer? Even myself did not know much about software testing. I started as junior developer and if I did not meet the software test architect who got me into software testing, I might not be in software testing field.

One of the famous test engineers, James Whitaker, (I should say used to be a test engineer), preaching about software testing as dying profession. He is proud of running his org without test engineers. (calling SDETs developers). He's got his own reasons and many of his point make sense.

Anyway, let me address my reason why I like testing.

I like software testing because it is NOT straightforward. It's not like you need to do 100 things for small applications and 10,000 things for bigger applications and done with it. And it's not like you're good if you have a certain number of unit tests, functional tests ,integration tests and system tests. It's not about the quantity; it is actually about the quality. There are so many things to consider when we're testing something. Different aspects of problems that the application solves, technology that is used, business expectations, user behavior, priorities, threats, performance, dead line date, project team, process, and etc. Finding best possible testing strategy and execution around this environment is great fun. And every feature, every story(in agile) provides different testing problems.

I like software testing because of its exploring nature. I like the thinking process of how it(new feature or project) might not work in the system and problem that comes as a consequence. I love to apply the Systems Thinking on this. Many people say going over test cases are boring. But coming up with test cases (written or not) or trying something with some suspicion about the application is great fun. "I try this because I've seen similar things before. I try this because from my understanding this new thing might cause problem on certain features that are already exist." Understanding the interactions in the system and explore the possibilities of unexpected interaction issue is really a beauty of testing. And this process start from spec reviews (sprint planning in agile) and I get to know more and more and exploring goes to many interesting directions.

I like software testing because I have many different options to execute tests and there are tons of tools out there to choose from. I would say this is more of test execution fun. This is not about writing automation code I would say. Coming up with best test execution strategy is great fun. What kind of tool is useful for data driven testing? How about model based testing? What's best test automation strategy for agile process? Experience in several different test framework and tools. Quick experiment on new tools. What if we just choose to go with manual for testing this feature? How can we represent our test results and communicate with the team? This requires endless learning. Learn to design good test framework,  learn new languages and new tools, learn how to represent test results, learn how to optimize test execution time, learn, learn and learn.

How about you? Why do you like software testing?

Systems thinking and software testing

Today I like to write a little bit about Systems Thinking and how it is related to software testing.

I've got the word Systems Thinking from James Bach. I saw one of his youtube video and he mentioned about reading Introduction to Systems Thinking. So I google it and youtube it.

Systems thinking (Software testing 101)

More to come but now I want to share the video that I watched more than 20 times.

Dr. Ackoff's Systems thinking presentation at TED

A good test automation framework

Today, I like to mention a little about test automation frameworks.

If you're looking at any kind of job posting (company career site, monster.com, and etc) for a software test engineer now, you will see almost all the posting mention some level of experiences in test automation framework as a requirement. And actually I do use test framework for my testing and it is very useful tool that resolves several limitations of manual testing.

So I thought it is worthwhile to put my thought about what a good test automation framework is in my blog. First, let me scope the area of discussion since test automation framework is a pretty big topic. I'm going to set aside the discussion around pros and cons of test automation in software testing since that's not my main point of this article. And I'm talking about your work related test automation framework not the generic programming language specific test framework. For example, my test automation framework is built on top of JUnit or TestNG. JUnit and TestNG are also test automation frameworks, but those frameworks do not have the specific business logic of my testing.

OK. Let's start. What is test automation frameworks?  Wikipedia mentions "A test automation framework is a set of assumptions, concepts and tools that provide support for automated software testing." My simple definition is "a test automation framework is a program that helps testers execute and validate test cases(test scenarios). So it is a program that helps software testing. The meaning of "help" will be saving time, being more efficient in testing process, gathering useful data easily after test execution and etc compared to the manual human test execution. But it is a program, which means someone has to design, implement and maintain. This brings a very important ground rule of test automation. Benefit of using test automation should always be greater than the effort of designing, implementing and maintaining the test automation. Otherwise there is no point using test automation beside you being cool. :)

I believe having a good test automation framework is to maximize the benefits of test automation and minimize the effort of implementing and maintaining it. So let me address the characteristics of a good test automation framework.

1. Have a clear strategy and everyone should be on the same page. This is the most important and most effective way to maximize the benefit of using the test automation framework. There are many different ways to design and implement test automation frameworks. And it is REALLY hard to come up with one solution that will satisfy every test engineers, which means when a test architect or experienced SDET come up with a framework, he/she should explain the strategy of the test automation and help everyone in the team understand how it works and how to use it. Once everyone is on the same page (know how it works and how to use), implementing test cases (scenarios) is much faster and maintaining the framework is much easier. By following a certain rule or instruction, test writers are less distracted and he/she can understand others code very easily. And when it comes to a situation where the current approach does not fit, it is much easier to notice and notify test automation designer (architect or SDET) to resolve the issue with proper solution.
2. Favor object composition over class hierarchy. This is one advice from Gang of four design pattern. One of the common mistake with test framework is that some framework base class provide too many functionality and become giant class (a few thousand lines of code). This approach works fine at the beginning but it does not scale very well and becomes very complex code base as number of tests grows to handle various test conditions, error scenarios and exceptional cases. One key characteristics of good test framework is to provide flexibility to test writers. Test writers should be able to choose classes or modules they need to execute one test case. Just Mix and Match for their tests.
3. Separation between test execution and validation. This is more about validation code being independent from test execution. It also means validation code by itself does not contain any context about test cases. What this allows to the framework is that you can package one logical chunk of  validation into one class. It will takes some parameters as source of validation. You call validate() method and it will simply return true or false. No decision making process here. For example, if error expecting flag is set do this. if some other case do that. NO. JUST DO THE VALIDATION. What does this allow test writers? You gather source of validation during test execution and pick and construct validation instances you need from pool of validation classes. DO NOT let the validation code to be workflow. LET IT BE collection of appropriate validations for each test case. This promotes great re-usability of the code and less complex code.
4. Build the test framework to support the way test execution steps are defined. The ideal framework, in my opinion, is that defined (written or thought) test execution steps are transformed to test automation code with 1:1 mapping. This seems pretty easy to do but this requires thoughtful design to expose natural language like method names and encapsulate well structured implementation details. One way to think about this is that you're providing framework for test writers who are not as technical as you. This promotes great readability of the code as well. 
5. Be cautious about re-factoring codes. Re-factoring is useful work. It's a good process that turns spaghetti codes to well-structured code. But often times, test automation writers get confused by this concept where he/she does not understand the difference between test automation code and development code. If you look closely on test automation code, there are two big parts: one is test execution steps related part and the other is test automation framework part that supports test execution step code. Re-factoring should largely applied in the latter part of automation code. Here is the common problem that I've seen. At first, test cases that a tester is implementing is very similar beside some inputs, config, or environment, so he re-factor the common piece in one method. Looks good. On the next iteration or release, he adds more test cases. He notices a little variation of the execution step or input, so he add some conditionals (if/switch statement) to the common method. As more iteration or releases come by, the common method becomes more and more complex. Complexity is not the only problem. His thought process of executing test steps is restricted by existing code base and he is trying to fit his thought on that code path. Also it is very hard for other testers to read and follow the logic. Refactoring test execution steps code can be very dangerous because it restricts testers thought process, makes test automation code less flexible, and makes readability of the code very bad and it hurts other test writers to take over. I admit that it is hard process because sometime it is hard to draw a line between test step code and supporting code. The key is to have test execution steps to be flexible and to have supporting code to be well structured.
6. Pinpointing the failure is more important than handling error gracefully. Do not be scared of null pointer exception. Automation code is not production code. I don't think this needs a lot of explanations
7. Minimize bootstrap code. Setup methods like @BeforeClass in JUnit or some sort of prep code runs before the test run is useful. But it also prevent your framework from being flexible. And it also make the framework very heavy. Sometimes you just want to check a simple validation that was used in test automation for other purpose. Well modularized code does not need much bootstrap code. You should be able to run your test in main method if your automation framework is well modularized.

If you want more fundamentals around test automation check this one.  Thoughts around test automation framework
Here is  part2 with more detailed explanation

How to innovate in software testing team

1. Eagerness to be better. An old couple who run hot-dog stand in market place story.
2. Trying to find solution or ideas against ultimate goal rather than improvement of current process. Broccoli and cupcake story. 
3. Steal like an artist. Use other academic approach. Car sales man. Writers.