UI Automation Gotchas

Having recently had the opportunity to work on and refactor some of the UI automation projects, I have come across many a gotchas that creep into the automation codebase resulting in not so beautiful, a little un-manageable, random-failure throwing specs. The worst of all things to expect from an automation test suite are random  failures, because it defeats the purpose of Continuous Integration in that the team stops responding to the automation test failures saying that there always is some random test that fails. So, if there is a test that fails randomly, analyse the reason for it and fix it ASAP giving due consideration to the solution that you put in place to ensure that the spec would not fail again for wrong reasons.

  

To aid the folks writing ui automation specs, I list the common gotchas that you should be aware of so that you do not end up making the same mistakes. And here goes the list:

1. Loving to sleep

     Liberal presence of sleep() method in the automation test code is unfortunately a very common thing of the day. You must avoid sleep() method. sleep() can come into existence only upon the team consensus (including the developers) wherein I believe that the team comes to a consensus upon its usage as a last resort. Here again, I would recommend posting the situation in various public tech forums to see what the community adopts as a practice and go with that approach; it is more likely for you to get help from the community on the right approach. Bear in mind that with every sleep(), you drag the time for feedback of automation test suite to that extent. YOU MUST AVOID SLEEPING!

Here is one sample scenario where I found sleep() was used without any thought.

Scenario:

As a user

I should be able to add notes to the table

so that the information is recorded for future reference.

Code: 

Why is sleep bad in the above context? Because there is no certainty on the time it takes to successfully save a new note. Typically, it takes micro seconds for the save to be successful and you are wasting thousands of microseconds here. On occasions though, it might take more than the anticipated sleep time, in which case the test might fail because of early assertions.Alternatively, the sleep() method can be replaced with wait_until method available in capybara. What if you are using a framework other than capybara? This   post is not a recipe. So, go google or stack overflow for a solution ;) 

2. Asserting the messages in the page for grammar and exactness of expected messages

     Let's get it right automation tests are written to test functionality's correctness. Writing tests to ensure the message remain intact as expected is a stupid stretch resulting in test brittleness. And common, if the messaging were to change so be it. For the good customers sake spend your valuable time on protecting the application's functional correctness.

3. Forgetting to create your browser profile with download preferences set to automatically download specific file extensions to a custom path

     The default behavior of most browsers is to prompt for opening or saving the file when it is downloaded. With this default preference in place, it makes automation a little difficult to respond to this prompt. You may well leave the prompt unattended for the automation spec to continue with other test cases, which I have seen as a practice followed by a team out of sheer laziness and lack of interest/curiosity to resolve. Sad! Take pride in good work more than appreciations from others. Love what you do…please!

Every browser uses a profile by default. In your automation boot-strapper file, you can set the download preference for the profile the browser uses to automatically download the files with specific extensions to a custom location.

Find below a sample code for setting the Firefox preference to automatically download the files of csv or pdf type:

4. Not asserting for the presence of right elements upon performing a particular AJAX action

     Common sense is an uncommon thing. Consider a scenario below:

As a user

I should be able to add notes to the table

so that the information is recorded for future reference.

Every new note is added seamlessly by ajax call. And the Page object in the automation test for this scenario looked as below:
What do you think the problem is with the above code? The code above should be perfectly fine for the first row to be added in a table. But for the second row, it isn't right, because the has_link?("Edit note") will return true seeing the existence of it for the first row. This results in the test instability. So instead of looking for the link, the QA should check for the existence of the note text that was added. Again, take this with a pinch of salt, if the context is the new row is added in an in-place editor like table, you might have to think what would be the best fit in that situation.

5. Not employing page object pattern

     Employing Page Object Pattern is the fundamental thingy to UI Automation. If you are not doing it, you should do it for better maintainability and readability. PERIOD.


6. Littering page object with assertions

     I have repeatedly found in projects after projects where the Page objects have methods like verify_XXX containing assertions in it. It is bad in that the Page object is violating the design principle, SoC (Seperation of Concerns). Assertions is the job of the test/spec class. Page object can have methods that return the state of the page or an actions that the user performs on the page and nothing more.

     However, the common reasoning that I hear for having assertions in the Page object is that of Reusability of the assertions across tests or specs. In an attempt to go by the DRY design principle, the SoC cannot be compromised, at least in this case. Having said that, I'll explain with an example of how the trade-off between DRY and Soc can be achieved with good balance.

     Now, assume you have a code in the Page object as below:

     We can re-write the above code segments as below and we would do good to write it this way ;)


7. Narrowed down xpath

     The reasoning I hear for narrowed xpath are two things - it was spit out by recording tool and it helps speed up finding elements faster. This is a classical case where test stability and maintenance should be given more importance over speed or performance.

     So the next time you write one or come across one, make sure it is reasonably accurate.

 
8. Using xpath over css

     You should prefer css over xpath for better automation test stability.


9. Using css over id

     You should prefer id over css for better automation test performance and stability.


10. Not exploiting the automation framework and tools that are put in use

     There are occasions that you would come across wherein you feel stuck at writing an automation test case for a particular scenario. During such times, do not ignore writing the test case all together from automation. Instead, explore for ways to exploit your automation testing framework. Google, stack-everflow, etc - there are many life savers. Do spend time around looking for solutions until you get to a point where you learn that automating the scenario on hand is not possible.


11. Code littered with print statements

     If you see it delete it. If you add it, don't forget to delete it. Print statements are not functionality. You add it/those for debugging purposes. Once you are done with it, DO DELETE IT.


12. Page object littered with dead code

     By dead code, I mean code that is not used. I have seen in many occasions where there exists method definitions in the Page object that are never referred anywhere in that file or other.

     When you come across such code what should be your reaction? Simply DELETE. No joke intended!

13. Code littered with comments and commented out code

     Commented out code is well, dead code. And by now you know what you should do with dead code - mercilessly delete it. It is such a common phenomenon to find code base littered with commented out code. And I hate to see it and so instantly delete it without a second thought.

     Comments describing the method or some lines of code is a smell in itself. Whenever, you find yourself doing it or come across such things, see how you can refactor the code to avoid it.

14. Violating DRY principles in code

     This is small but irritating stuff. Be lazy, very very lazy……to repeat stuff. I have seen automation code bases littered with many instances of duplication. This is REAL PAIN TO EYES! As an example, if you are using a string "random text" in multiple lines in a spec, it is SHIT! Do not do this. Assign the string constant to a local variable and refer to it by way of that local variable in all the required places.

This is very basic stuff and it really pains to see this kind of violation in code bases.

15. Not making use of the language strengths

     You might say how does it matter if I don't use the niceties of the programming language and how can not using some niceties be an automation gotcha? Automation specs/tests is code and code should be…well, readable and maintainable - using the language niceties aid better readability and maintainability. No exception to this at all. So, be curious. Have an urge for intellectual betterment and you'll learn new stuff and will code better.

Negotiation - The road to Minimum Viable Product (MVP)

     Are you spending time in unfruitful, never-ending, repeated meetings to discuss the solution - the single silver bullet - that meets all of customer requirements? And do you care to get the shit out of that quagmire? The first thing to realize is that you are not on the road to MVP. Secondly, you should seek to hit the road to MVP. Lastly, read further and digest the lesson :)
     Imagine God from nowhere appears in front of you and asks, "My dear, what do you want?" Would you ask for ask for just an item or an endless list of items? Without exception, each one of you would ask for the latter. The situation is no different even in the business context of the software projects that we undertake. Thanks to the excellent efforts and works of the Sales or frontline folks. The customer looks at you as an all powerful Angels if not as God. Their business wants are limitless and they expect you to solve it magically in one go.
     You and your team are now in deep shit. You should be wise enough to know that all the customer wants can't be met in a single go. If you are not in a same page as I am with this statement, you better be :P Now, assuming that you are in the same page as I'm, what will you do given this circumstances? Commit that you can and end up not delivering. Bummer! You can only take one bite at a time. Don't be foolish to think that you can have everything in one gulp - that will be your road to disaster.
     So, what is the way out? NEGOTIATE.
     And to learn one of the best lessons in successful negotiations, you don't really have to hit the Harvard; you can learn it from the conversations between a mother and her kid. When a kid makes a ton of demands to its mother, the mom gives a patient hearing to what her kid has to say and subsequently repeats the wishlist of her kid to both assure the kid that she has been listening all the while and also to ensure that she understood the demands right. She would then truly collaborate with the kid to prioritise the list and finally makes the move - promise to fulfil the needs in incremental instalments as per the agreed priority. That to me is a successful win-win negotiation.
     The advantages to this incremental fulfillment are many fold. Any requirement could have been mis-understood. The requirements might change. The priorities might change. There are very many factors that influence the wishlist and the strength to fulfill it. Think about it. Didn't you in your everyday life practice this "incremental priorised delivery" to meet some of your personal needs. You merely have to apply the same principles in your business with the client. Negotiate by education to fulfill all their needs but in prioritised incremental delivery that adds business value with every single delivery.
     To rephrase, the road to MVP in typical business will look like below:

1. Get the Business Vision and Mission statement.
2. Elicit the customer wishlist.
3. Make attempts to understand them and pronounce your understanding of the wishlist. In essence, you repeat the wishlist to the customer to showcase your understanding of the same.
4. Prioritize the wishlist based on the business vision and mission statements.
5. Promise to deliver in incremental installments of business value.

     Don't trust me. Give it a shot in your current gig and feel free to share your experience by way of comments to this blog post. I'm looking forward to having a conversation with you :)