Getting test automation done is a challenge, especially within the tight deadlines imposed by Scrum. As much as the thought of continuous in-sprint test automation sounds enticing, the practicality of it may elude most Scrum teams.
In my article published here– I look at some of the main things you need to consider in order to get your test automation done within the confines of your sprint.
The first thing to focus on is a framework that is useful, is easy to understand, and helps all stakeholders participate in test automation.
This is essential because you want to make test automation a continuous activity that is a part of daily work, not a once-a-sprint (or once-a-release) work item. For this to happen, the framework must make it equally comfortable for a businessperson, developer, functional tester or automation expert to add their contribution and see the results of their efforts.
There are many business-friendly frameworks and techniques, like behavior-driven development (BDD), as well as many tools that can create tests in a domain language and then translate them to script code.
All stakeholders must be trained on using the framework, and their area of contribution must be made clear to them, with practical hand-holding. The automation tester can then focus on maintaining the framework, generating test suites and editing failing scripts, while the creation of test automation will be a continuous task assigned to everyone involved.
The next thing to focus on is collaboration between the various stakeholders. A continuous automation framework can only survive when it is being fed and tended to by everyone on the team.
The business people, like a business analyst or a product owner, can help by adding user scenarios or defining the requirements in a framework-friendly format. This may require them to be trained on the preferred format based on the framework being used
The developers can help by creating reusable methods for steps of the script. They can also create and maintain an object repository for all elements they add to the UI while testers use the pseudo names of the elements in the test scripts. This means that the scripts can be created before (and independent of) the application UI, and such scripts won’t need editing when the UI changes, as long as the object repository is kept up to date
The testers can help by adding more scenarios, specifying and creating test data, and executing the scripts periodically
How to strategize the development of test scripts is crucial to making in-sprint automation a reality. Using API-level automation whenever possible will reduce the time and effort.
Many organizations take up the transition from waterfall to agile with the best intentions in mind. Like so many other companies, you might also be seeking to replace your traditional waterfall processes with agile in a quest to shorten the time-to-market and deliver high quality applications.
The road to agile, though, can be a rocky one! That’s why, in my latest refresh post for Ranorex blog, I have put together a few lessons and tips that will help you in succeeding in moving from waterfall to agile successfully!
The beginning of coding is a good time to start writing detailed tests.
As testers think of new scenarios to validate with executable tests, they also think about potential scenarios for manual exploratory testing. Make a note of these for later pursuit.
Some quick risk analysis can help you decide what testing to do first and where to focus your efforts.
The Power of Three Rule – When unexpected problems arise, you may need to pull in more people or even the entire team. Tester, Developer and Customer (or businesspeople) can together decide on correct behavior and solutions.
As soon as testable chunks of code are available, and the automated tests that guided their coding pass, take time to explore the functionality more deeply. Try different scenarios and learn more about the code’s behavior. You should have task cards for tests that critique the product both business and technology-facing. The story is not ‘done’ until all of these test types are done.
If your exploratory tests lead the team to realise that significant functionality was not covered by the stories, write new stories for future iterations. Keep a tight reign on “Scope Creep” or your team won’t have time to deliver the value you originally planned.
Technology-facing tests that critique the product are often done best during coding. This is the time to know if the design doesn’t scale or if there are security holes.
Leaving bugs festering n the code base has a negative effect on code quality, system intuitiveness, system flexibility, team morale and velocity.
Strive for “zero tolerance” towards bug counts.
Teams have solved the problem of how to handle defects in different ways.
Some teams put all their bugs on task cards
Some teams chose to write a cared, estimate it & schedule it as a story.
Some teams suggest adding a test for every bug
The more bugs you can fix immediately, the less technical debt your application generates and the less ‘defect’ inventory you have.
Try making the estimate for each story to include (atleast) two hours or half a day for fixing associated bugs.
If a bug is really missed functionality, choose to write a card for the bug and schedule it as a story.
Code produced test-first is fairly free of bugs by the time it is checked-in.
The Daily Stand-Up helps teams maintain the close communication they need.
Use Big, visible charts such as story boards, Burndown charts and other visual cues to help keep focus and know your status.
Having story boards gives your team focus suring the stand-ups or when you are talking to someone outside the team about your progress.
Testers can help keep the iteration progressing smoothly by helping make sure everyone is communicating enough. They can help programmers and customers find a common language.
Use retrospectives to evaluate whether collaboration & communication need improving and brainstorm ways to improve.
Teams in different locations have to make a special effort to keep each other informed.
Teams take different approaches to make sure their build stays ‘green’.
The build needs to provide immediate feedback, so Keep It Short.
Tests that take too long, such as tests that update the database, functional tests above Unit level or GUI test scripts, should run in a separate build process.
Having a separate, continual ‘Full’ build with all of the regression suites is worth the investment.
During the iteration, you are automating new tests. As soon as these pass, add them to the Regression Suite.
As you start the iteration, make sure that test environments, test data, and test tools are in place to accommodate testing.
You may have brought in outside resources for the iteration to help with performance, security, usability or other forms of testing. Include them in stand-ups and discussions. Pair with them to help them understand the team’s objectives. This is an opportunity to pick up new skills!!
Consider what metrics you need during the iteration – Progress and Defect Metrics are 2 examples.
Whatever metrics you choose to measure – Go for Simplicity!
Most teams kickoff their new iteration with a planning session. – where they discuss one story at a time, writing & estimating all of the tasks needed to implement it.
Task cards need to be written along with development task cards and estimated realistically.
When writing programming task cards, make sure that coding task estimates include time for writing unit tests and for all necessary testing by programmers.
Testers should help make sure that all necessary cards are written and they have reasonable estimates.
Your job as a tester is to make sure enough time is allocated to testing and to remind the team that testing & quality are the responsibility of the whole team. When the team decides how many stories they can deliver in an iteration, the question isn’t “How much coding can we finish?” but “How much coding and testing can we complete?”
Commit Conservatively – It is always better to bring in another story later than to drop a picked story.
Working closely with customers or customer proxies is one of the most important activities as an agile tester. Good communication usually takes work.
We want “big-picture” tests to help the programmers get started in the right direction on a story. High level tests should convey the main purpose behind the story.
Don’t forget to ask the programmers what they think you might have missed. What are the high-risk areas of the code? Where do they think testing should be focused?
When Testability is an issue, make it the team’s problem to solve.
One beneficial side-effect of reviewing the tests with programmers is the cross-learning that happens.
High level test cases along with executable tests you’ll write during the iteration will form the core of the application’s documentation.
People unfamiliar with agile development often have the misconception that there’s no documentation. In fact, agile projects produce usable documentation that contains executable tests and thus, is always up to date.
Testers in agile must be proactive. Instead of waiting for work to come to them, they get up and go look for ways to contribute.
Working on stories in advance of the iteration may be useful for teams that are split across different geographic locations. By working ahead, there’s time to get information to everyone and give them a chance to give their input.
If we make our iteration planning go faster and reduce the risk of the stories we’re going to undertake, it’s worth doing some research and brainstorming before we start the iteration.
The Pre-Planning Meeting
Go Over stories for the next iteration
The Product owner explains the purpose of each story – business conditions of satisfaction.
Team brainstorms about potential risks and dependencies, asks questions and figures out the simplest path.
Pull in customers to answer questions, get a better idea.
Experiment with short Pre-Iteration discussions and Test-Writing sessions
Invest preparation time when it’s appropriate. There is a risk to ‘working ahead’.
To go Fast – We need to Slow Down First!
Teams that are distributed in multiple locations may do their iteration planning by conference call, online meeting or teleconference. ( And Cut to 2020 – Coronian Times – Every one of us is doing that!! )
One practice that Lisa’s team used was to assign each team a subset of the upcoming stories and have them write task cards in advance.
(I, too, have used this practice – only the Task Cards were in fact story Sub-tasks being created in JIRA for our user story items created by the PO)
If the customers aren’t readily available to answer questions and make decisions, other domain experts who are accessible at all times should be empowered to guide the team by determining priorities and expressing desired system behavior with examples.
(I have experienced that – our Product Owners essentially did this job for us)
Examples are an effective way to learn about and illustrate desired functionality. Using Examples, you can write high level tests to flesh out the story a bit more.
Mock-ups are essential for stories involving UI or a report. Ask your customers to draw up their ideas about how the page should look.
Before the next iteration – triage the outstanding issues with the customer. Those deemed necessary should be scheduled into the next iteration.
Use the Agile Test Quadrants to help you identify the different types of test automation tools you might need for each project, even each iteration.
Test Automation Pyramid (introduced by Mike Cohn)
Lowest Layer- Bulk of automated unit , technology facing tests. Quickest feedback, code much more quickly using xUnit family of tools
Middle layer – Automated business-facing tests that help the team. “Are we building the right thing” Tests operate at the API, behind the GUI level. Bypass the presentation layer – less expensive to write & maintain these tests. Fit & FitNesse are good examples, written in domain language
Top Tier – Should be the smallest automation effort as they have the lowest ROI. Done through GUI, operating on the presentation layer. More expensive to write, more brittle and need more maintenance.
Any tedious or repetitive task involved in developing software is a candidate for automation.
AN automated deployment process is imperative – getting automated build emails listing every change made is a big help to testers. It speeds up testing & reduces errors.
A fast running continuous integration and build process gives the greatest ROI of any automation effort.
Another useful area for automation is data creation or setup. Cleaning up test data is as important as generating it. You data creation toolkit should include ways to tear down the test data so it doesn’t affect a different test or prevent rerunning the same test.
What we shouldn’t automate
Tests that will never fail
Plan-in plenty of time for evaluating tools, setting up build processes, and experimenting with different test approaches in the initial iterations.
If management is reluctant to give the team time to implement automation, explain the trade-offs clearly. Work towards a compromise.
We will always have deadlines, and we always feel pressed for time. There is never enough time to go back and fix things. During your next planning meeting, budget time to make meaningful progress on your automation efforts.
Good test management ensures that tests can provide effective documentation of the system and of the development progress
“Toolkit for Business-Facing Tests that Support the Team”
As agile development has gained in popularity, we have more and more tools to help us capture and use them to write executable tests.
Your strategy for selecting the tools you need should be based on your team’s skill set, the technology your application uses, your team’s automation priorities, time, and budget constraints. Your strategy should NOT be based on the latest and coolest tool in the market.
In agile, simple solutions are usually best.
Some tools that can help us illustrate desired behavior with examples, brainstorm potential implementations and ripple effects and create requirements we can turn into tests are—
Software based tools
A picture is worth a thousand words, even in agile teams. Mock-ups show the customer’s desires more clearly than a narrative possibly could. They provide a good focal point to discussing the desired code behavior.
Visuals such as flow diagrams and mind maps are good ways to describe an overview of a story’s implementation, especially if it is created by a group of customers, programmers, and testers.
Tools such as Fit (Framework for Integrated Tests) and FitNesse were designed to facilitate collaboration and communication between the customer and development teams.
Finding the right electronic tools is particularly vital for distributed teams (chat, screensharing, video conferencing, calling, task boards etc.)
Selenium, Watir and WebTest are some examples of many open source tools available for GUI testing.
Home-Brewed Test Automation Tools
Bret Pettichord (2004) coined the term ‘home-brewed’ for tools agile teams create to meet their own unique testing needs. This allows even more customisation than an open source tool. They provide a way for non technical customer team members to write tests that are actually executable by the automated tool. Home-brewed tools are tailored to their needs, designed to minimize the total cost of ownership and often built on top of existing open source tools.
The best tools in the world won’t help if you don’t use them wisely. Test tools might make it very easy to specify tests, but whether you are specifying the right tests at the right time is up to you.
Writing detailed test cases that communicate desired behavior is both art and science.
Whenever a test fails in Continuous Integration (CI) and build process, the team’s highest priority should be to get the build passing again. Everyone should stop what they are doing and make sure that the build goes ‘green’ again. Determine if a bug has been introduced, or if the test simply needs to be updated to accommodate intentionally changed behavior. Fix the problem, check it in, and make sure all tests pass!
Experiment – so that you can find the right level of detail and the right test design for each story.
Keep your tests current and maintainable through refactoring.
Not all code is testable using automation but work with programmers to find alternative solutions to your problems.
Manual test scenarios can also drive programming if you share them with the programmers early. The earlier you turn them into automated tests, the faster you will realise the benefit.
Start with a simple approach, see how it works, and build on it. The important thing is to get going writing business-facing tests to support the team as you develop your product.
***Update **About face-to-face communication** during Covid-19 ***
As I am reading this book during this bizarre time of social-distancing, working remotely and entire nations on lockdown, the part about ‘face-to-face’ communication has a new meaning now. As Janet Gregory also pointed out in response to this article, our definition of face-to-face has changed over the last few weeks over the entire world! We are lucky to have technology that helps us continue effective communication within our teams, have conversations, video calls, screen shares, continue learning over webinars and continue working, feeling useful and being productive.
Hoping things change soon and we can go back to having fun, productive discussions with our team mates over coffee. Until then — Happy social distancing!
Agile transformations can be a challenging undertaking, and many organizations struggle with what is probably the hardest part of the transition: adopting an agile mindset. It is imperative that teams embrace the agile culture before they can fully embrace agile.
As I always like to say, agile is more a mindset than a process. It guides you to a better way of working and collaborating in order to deliver the most value to your users. But how you choose to implement those guidelines is up to you, and most teams coming from a traditional style of software development find this aspect the most challenging.
Teams are left to find ways to work together rather than having a process forcing them to do certain actions, follow certain processes, or organize specific meetings. There are no templates or techniques to adhere to and no rules to follow strictly.
This may come as a surprise and leave teams guessing since they are used to being told what to do and how. Agile drives them to think on their feet as they plan and replan their way through the development process. Read More–>
Being Comfortable with Visibility & Exposure
Agile gives everyone a voice and values every person’s opinion. Many teams have been used to only the manager speaking for them or having one representative in most meetings. As a result, some team members may feel flustered now that they’ll occasionally be in the spotlight. People who are not used to voicing their opinion are expected to speak in all forums. Hiding behind the team is no longer an option in agile.
This also means team members are valued as individuals and everyone’s contribution is recognized. Agile treats all team members as equals, whatever their role or designation. They are expected to estimate their own tasks, pick things to work on, collaborate with other team members, and provide value by the end of each iteration. Continue Reading–>