In my current role, I manage a very special team of quality engineers who are responsible for setting standards and best practices for quality assurance across the entire organisation and is also responsible for developing any necessary tooling or frameworks that might help product teams reduce toil.
This is not a new problem and hence I was looking for inspiration and lessons to learn from other organisations who have embedded quality at the heart of their software development process. I came across this presentation by two Quality Specialists who also happen to be Atlassian partners. Please go ahead and watch it if you haven’t already.
I agree with pretty much everything they said and took this opportunity to sumarise what I thought wasmost useful to remember when implementing good quality assurance practices in any organisation.
Teams to own quality
There are plenty of tools out there that can help with building quality into your software development process.
But no software will be enough if the teams do not take ownership of the things they build.
It is important to understand that software quality is a team sport. Everybody on the team needs to think quality from the moment they get a requirement from the customer to the time they push their software out to the customers. Everyone in the team has to be able to think about the quality of the product and not just the quality assurance engineers or analysts.
Lean QA
Lean QA is inspired by lean software development techniques focusing on maximising customer value by minimising waste.
Common types of waste in testing processes
- Testing everything - have you tried testing what is essential instead?
- End to end testing - often too expensive to implement in complex software systems
- Excessive docs about behaviour but nothing to automatically ensure and notify that something is not as expected
- Finding and fixing bugs - sounds easy but have you tried not creating bugs in the first place?
Better way to test
Focus your test strategy around Risk based testing
Test the high likelihood, high impact scenarios to minimise waste and maximise value!
Finding and fixing bugs
There is a cost to finding all bugs. So unless time, money and people are unlimited, finding and fixing every single bug is not a good strategy.
Similar to the Risk based testing approach discussed earlier, fix bugs that are easier to find and could cost you customers or your reputation,
If hard to find bugs take some 25 different steps to take before we encounter the bug. The likelihood of this being found by a client and affecting their use of the program is very less.
Imagine an easy to find bug, if you can find it in a few steps then it would be just as easy for the clients too. If they ran into the bug and it crashed the app they would end up uninstalling the app and you’d lose potential revenue and reputation!
Focus on fixing the easy to find bugs.
WTF is agile testing?
Is it unit testing? Is it TDD? BDD?
Agile testing: It is testing early and testing often and testing the right stuff at the right stage in the software development lifecycle.
How to get everyone do agile testing?
People first. Get everyone the right training to bring them on the same page. This helps to not call an organisation agile if it really is not agile.
Ensure that product owners or managers have the right technical support they need. Ensure everyone is aligned in terms of agile.
Identify teams who were agile vs fragile - understanding who does incremental vs iterative development - have something deployable to production from the start.
How to get better?
- Create a plan to tackle test automation
- Prioritise test automation every sprint
- Make sure everyone agrees on definition of done
Development process
- Identify a good branching strategy that enables continuous integration
- Ensure there is a way to share the same set of tools throughout the entire organisation.
Test ownership
It is important to clearly differentiate who owns what part of the testing journey.
- Developers own white box testing done at the lowest level of code to check correctness using practices like TDD.
- QA Engineers own black box tests focusing on negative cases with automated ui regression tests or the likes.
- QA analysts or engineers, also own black box tests that focus on system integration testing
- QA engineers also own black box tests that test reliability, scalability, robustness at a very high level.
- Product owners own the black box tests that are behaviour driven tests testing business flows.
When to run what tests
It is important to focus on reducing waste and not run every single test every time something happens.
Context based test runs are better than running every test everywhere every time a new commit is made.
For example:
- When developing a feature you write and execute unit tests and might even have some automated functional tests in addition to your manual tests
- When code goes into your main branch, as a pull request, you run the unit tests and automated functional tests after integration, and if it is performance sensitive, then trigger those tests too.
- Prior to deploying code, run all the safety net tests - unit, automated functional, performance/integration tests, and if required some user acceptance tests. If the last type of tests are automated that’s a big win!
- In case of a hot fix, run the automated functional tests and do manual exploratory tests and some user acceptance tests.
Based on the risk and time available to deploy, choose the most appropriate combination of tests.
You do not want a hot fix to have to wait for all the various types of test to execute to let it be deployed. It might be a priority to system get back up and running and hence skip tests that aren’t absolutely necessary to run.
Just because there are more tests does not in any way mean the system is being tested correctly. It is important to ensure your tests are testing the right thing, more often than testing the wrong thing too often and slowing you down.
Gathering and linking data for Quality
Integrating data from your knowledge base to your ticketing/project management system to your test run software is absolutely crucial.
It is also important to ensure the right people are alerted when tests fail.
For example, unit test failures must notify developers of the right team. Integration tests must notify the appropriate QA engineers who can triage and help assign the issue to the right team.
How to make this a reality
There is always going to be people who are not willing to change bad ways of working or who are cynical or just doesn’t believe we can go from bad to good.
Good questions to ask:
- Hack it and get it out? or Do it right first time?
- Re-work and fix bugs? or Work on valuable features that customers want?
- Spend precious budget on rework? or Spend precious budget delivering value
- Be on a team is ashamed of their software? or on the team that is proud of the software?
Summary
- Reduce waste in testing
- Continuous integration is essential
- Test automation is essential
- Build in quality rather than test it later
- Full traceability of a requirement to a deployed feature to an incident is brilliant and absolutely priceless if achievable