Use the Agile Test Quadrants to help you identify the different types of test automation tools you might need for each project, even each iteration.
Test Automation Pyramid (introduced by Mike Cohn)
Lowest Layer- Bulk of automated unit , technology facing tests. Quickest feedback, code much more quickly using xUnit family of tools
Middle layer – Automated business-facing tests that help the team. “Are we building the right thing” Tests operate at the API, behind the GUI level. Bypass the presentation layer – less expensive to write & maintain these tests. Fit & FitNesse are good examples, written in domain language
Top Tier – Should be the smallest automation effort as they have the lowest ROI. Done through GUI, operating on the presentation layer. More expensive to write, more brittle and need more maintenance.
Any tedious or repetitive task involved in developing software is a candidate for automation.
AN automated deployment process is imperative – getting automated build emails listing every change made is a big help to testers. It speeds up testing & reduces errors.
A fast running continuous integration and build process gives the greatest ROI of any automation effort.
Another useful area for automation is data creation or setup. Cleaning up test data is as important as generating it. You data creation toolkit should include ways to tear down the test data so it doesn’t affect a different test or prevent rerunning the same test.
What we shouldn’t automate
Tests that will never fail
Plan-in plenty of time for evaluating tools, setting up build processes, and experimenting with different test approaches in the initial iterations.
If management is reluctant to give the team time to implement automation, explain the trade-offs clearly. Work towards a compromise.
We will always have deadlines, and we always feel pressed for time. There is never enough time to go back and fix things. During your next planning meeting, budget time to make meaningful progress on your automation efforts.
Good test management ensures that tests can provide effective documentation of the system and of the development progress
“Why we Want to Automate Tests and What holds us back”
Test automation is a core agile practice. Agile projects depend on automation.
Manual tests take too long and are error prone.
Automation regression tests provide a safety net. They give feedback early & often.
Automated builds, deployment, version control and monitoring also go a long way toward mitigating risk and making your development process more consistent.
“Build Once, Deploy to Many” – is a tester’s dream!
Projects succeed when good people are free to do their best work. Automating tests appropriately makes that happen!
If we become complacent about our testing challenges and depend solely on automated tests to find our issues, and then just fix them enough for the test to pass – we do ourselves a disservice.
However, if we use the tests to identify problem areas and fix them the right way or refactor as needed, then we are using the safety net of automation in the right way.
When tests that illustrate examples of desired behavior are Automated, they become ‘living’ documentation of how the system actually works.
Barriers to Automate –
Programmer’s attitude – Why automate at all
The Hump of Pain – the initial learning curve
Fear – Non-programming testers fear they have nothing to contribute
Old habits, team culture
Without automated regression tests, manual regression testing will continue to grow in scope and eventually may simply be ignored.
Teams with automated tests and build processes have a more stable velocity.
Good test design practices produce simple, well-designed, continually refactored, maintainable tests.
Team culture & history may make it harder for programmers to prioritize automation of business-facing tests than coding new features. Using Agile principles & values helps the whole team overcome barriers to test automation.
Planning and developing new features at the fast pace of agile is a hard game. Knowing when you are really done and ready to deliver is even harder.
Having predetermined exit criteria helps you be able to make the decision that a feature is truly ready to ship. In my article published at TestRail Blog, I compiled a list of exit criteria you must add to your user story to make it easy to bring conformity and quality to all your features.
All Tasks Are Completed
This first one sounds obvious, but it may not be. I still see many teams struggling with getting their testing done within the sprint. Developers work on a user story and deem it done, while testers are left to play catch-up in the next sprint.
Put that practice to an end once and for all by making sure that no user story can be proclaimed done without having all tasks under it completed, including development tasks, testing tasks, design and review tasks, and any other tasks that were added to the user story at the beginning.
Ensuring all tasks are completed in a sprint also mandates that you begin thinking in depth about each user story and the tasks necessary for each activity to be completed, so that you do not miss out on anything at the end.
Tests Are Automated Whenever Possible
As our agile teams move toward continuous delivery and adopting DevOps, our testing also needs to be automated and made a part of our pipelines. Ensuring that test automation gets done within the sprint and is always up to pace with new features is essential.
By having test automation tasks be a part of a user story delivery, you can keep an eye out for opportunities to automate tests you are creating, allocate time to do that within the sprint, and have visibility of your automation percentages.
I have used the following exit criteria:
At a minimum, regression tests for the user story must be added to the automation suite
At least 50% of tests created for the user story must be automated
Automated regression must be run at least once within the sprint
Depending on what your automation goals are, decide on a meaningful standard to apply to all your user stories.
Human Testing is a craft that is more than executing a bunch of tests, performing clicks and actions. A tester has a unique understanding of the system and ways to critique it. Over time, the tester develops a deeper comprehension of the application and its intricacies, integrations, weak points, and history. This makes them the best judge to find out the failure points of the system and comment on its health.
The Product Risk Knowledge Gap is the difference between what we know about the product and what we need to know. The purpose of testing is to close or at least reduce this gap.
While automated checks can help in determining problems in what we know (and have scripted as checks), it may not help as much in the risk areas of what we do not know about the product. That requires exploration, creativity, intuition and domain knowledge. This is the human aspect of testing.
Automated scripts have some built-in steps in the form of test data that we pre-define and verifications that we add. These steps are helpful for areas of the application that we need to check, double-check or re-check a number of times, and because these types of checks can be made explicit, they can be automated. Since the same steps will be performed the same way over and over again, it is better called “checking” rather than “testing.”
I was invited by Ajay Balamurugadas to present a guest talk at his organisation Qapitol QA for their enthusiastic test team. I spoke about Layers of Test Automation to design a robust test automation framework. The topic was well received and the testers in audience showed keen interest and had some good questions! It was a good experience visiting and meeting the team and presenting a short demo of Sahi Pro too!
The testers at Qapitol QA are surely an inquisitive lot with a dynamic attitude and quest for learning. They showed a lot of interest in Sahi Pro as a tool, and also about test automation in general. I like to encourage such talent and help them in any possible way. I wish to continue the relationship and see them again in future events.
I was invited to speak at the DevOps and Agile testing Summit organised and conducted by 1.21GWs on 8th Nov 2019 at Bangalore. It was a great event which brought together many keen minds as delegates and many inspiring speakers. https://1point21gws.com/devops/bangalore/
My talk was on “The Building Blocks of a Robust Test Automation Strategy”. As we know testing teams are faced with a number of questions, decisions and challenges throughout their test automation journey. But there is no single solution for their varied problems! In this talk I outlined a number of strategies that agile teams can follow– be it their selection of what to automate and how much, what approaches to follow, whom to involve, and when to schedule these tasks so that the releases are of best quality.
I am grateful that my talk was so well received and led to great discussions later with many participants. I enjoyed the day and am always glad to be invited by the 1.21GWs team.
@Sahi Pro was also a knowledge partner at the event and delegates also got a peek into Sahi Pro via video and brochure handouts.
I am back from the trip to Canada which followed the big day that was #TQ2019. So, I finally have a chance to share my experiences. This event https://kwsqa.org/tq2019/schedule/ organised by KWSQA was special in a number of ways-
It was my first international conference talk 🙂
I was one of the few international speakers at the conference, and the one who traveled the farthest for it!
I was the only speaker presenting 2 talks!
The travel was big too – with tonnes of visa processing, a 24 hour long flight to Toronto and then a bus ride from Toronto to Cambridge (which I nearly missed 😛 owing to the infamous Toronto traffic! )
Day 1 of the event was workshops that were in progress when we reached and we got a chance to informally meet the organizers at the desk. That evening they had planned a Speaker dinner which was a great idea. I got to interact and meet with all the speakers, made some friends and so the next day seemed a little less daunting having so many known faces.
24 Sep was the big conference day. Staying at the same hotel gave me the advantage to get ready at my own pace and be on time for the breakfast. The event began with a brief intro and then split into tracks. The first talk I attended was ‘Lean Coffee Facilitators Training’ by Matt Heusser. My first time hearing him speak. His session was fun and engaging and practical. I did #sketchnotes for the talk and also participated in the activity which was fun!
After that was my own session in the next room, so I hurried to setup and get ready. The best part was that the organisers had planned a 15 minutes gap between each talk for QA/Networking which gave the speakers and the delegates some breathing room and time to get to other sessions.
I talked on ‘The What, When and How of Test Automation’ which was a 45 minutes session. The room was full and there were lots of good questions and participation from the audience. I did feel that I handled it well and the topic as well as the proposed ideas were well received! 🙂 Here are a few glimpses into my talk-
Though I was relieved having just delivered a good talk, I still had one more to go! After that was lunch hour. A few participants from my talk invited me to sit at their table and we had so many discussions about work, testing as well as my travel plans 😛
Then we got back to talks- I also attended a talk on ‘Barriers in Accessibility Testing’ by Albert Gareev which I also #sketchnoted
Post that I rushed to the lightning talks track as I had to prepare for my next talk that was a 15 minute session on ‘Gamify your Agile Workplace’. As I got there I heard Richard Strang talk about ‘Implementing an Agile QA Guild’ and his experiences that were so varied and interesting. Then I got up to speak and since I was talking about an innovation game called speed boat, I had to first draw a big speed boat on the flipchart (with my limited drawing skills:P ) with a room full of people staring! I guess I managed well as the room MC Tina Fletcher (also president of KWSQA) was impressed with my masterpiece 😛 hehe
The session went well – the best bit being our Keynote speaker Damian Synadinos attending as well volunteering for the little game we played. It was an honor and an unforgettable experience. I hope the audience took back something tangible to try out gamification in their agile teams.
With both the talks done, it was now time to relax and network. I stopped by the booths by Oracle and NPM, chatted with fellow speakers and delegates, the organizers and also got real time feedback from the attendees who chose to attend my sessions.
Post the little coffee break was the grand closing keynote by Damian and it really was an experience. He mentioned in his intro that he had some improv experience and he really uses it to the best in his speaking! The talk was funny, intriguing, had loads of content, memorable quotes as well as an activity in which I volunteered! And a big Plus — Damian mentioned me and my talk too! 🙂 🙂 All in all it was an epic performance and really inspiring as a speaker. Kudos to the effort that went behind putting this together.
The best parts were getting to know so many wonderful people like Josh, Bailey and Dani, and getting to meet @Matt Heuser who I have had the chance to work with online. A face-to-face interaction makes things seem so real and people so approachable. He is a gem of a person and so encouraging too. I also made a friend @Emna who came from Tunisia to speak at the event! We roamed the streets of Cambridge and rode buses together and by the end seemed like we have known each other for so long. I surely hope to see her again at a future conference.
The organizers at TQ2019 had really worked hard and their efforts worked out so well with such a grand event pulled off with great ease, smooth flow and right on schedule. They welcomed us with warmth and helped throughout the day. At the end of the day we all got some time to cool off with a Social event where we mingled and got a chance to express our gratitude and say good byes. I would like to personally thank Greame Harvey, Sabina, Rob, Josh Assad ,Jared and Tina Fletcher from the KWSQA committee who were all so helpful and kind.
I am thankful for getting this opportunity and look forward to staying connected with such awesome people. I am also thankful for my supporting hubby who tagged along so that we could make this into a trip – got a chance to explore Toronto, Montreal and Quebec city and of course the majestic Niagara Falls! 🙂
On top of that I get to present not one but 2 talks!! My topics are
“The What, When & How of Test Automation” 45 mins
In this I will talk about preparing robust automation strategies. Agile means pace and agile means change. With frequent time boxed releases and flexible requirements, test automation faces numerous challenges. Haven’t we all asked what to automate and how to go about the daily tasks with the automation cloud looming over our heads. Here we’ll discuss answers to some of these questions and try to outline a number of approaches that agile teams can take in their selection of what to automate, how to go about their automation and whom to involve, and when to schedule these tasks so that the releases are debt free and of best quality.
“Gamify your Agile workplace” 15 mins
In this I’ll present live some innovation games and have audience volunteers engage and play games based on known scenarios. Let’s Play and learn some useful Innovation Games that can help you gamify your agile team and workplace, making the team meetings shorter and communication more fun!
Both these topics are close to my heart and I am looking forward to sharing my thoughts with a wider audience.
I am also excited to meet all the awesome speakers at the event , as well as get to know the fantastic team of organizers behind this event!
I got a chance to present a talk at the World Test Engineering Summit organised by 1.21GWS @ Bangalore last week and it sure was a great opportunity! My talk title was
“Layers of Test Automation” wherein I presented about test automation framework creation and best practices on creating a sustainable framework. The talk was appreciated and I had great personal feedback and chats with many delegates.
It was amazing to share the stage with renowned speakers like Mrs. Renu Rajini, Mr. Mostafa Awadh from Egypt, Shivaji Raju , Sanjay Kumar and many more.
My team at Sahi Pro also decided to partner with the event and setup a demo booth, where my colleagues Pratik and Satish showcased an informative demo of Sahi Pro tool and all of its awesome capabilities. The Sahi Pro booth was a hit, appreciated by the inquisitive participants. We also held a Quiz for people visiting the booth, and the winners were awarded with fun goodies at the end of the day!
Here is a glimpse into the event-
My Talk —
Sahi Pro Booth–
Felicitation for the Quiz Winners and honoring the organizer Nitin Naveen-
So much fun and networking–
Overall this was a great experience! We would love to collaborate with 1.21GWS again in the upcoming events!