Test Flakiness: What It Is, Causes, and Solutions

Mai 24, às 17:24


5 min de leitura


1 leituras

Unreliability in tests can be a nightmare situation for testers. No matter how experienced the testing team is, test flakiness can trouble the heck out of them. It is a hell you should avoid at all costs. But...
Test Flakiness: What It Is, Causes, and Solutions

Unreliability in tests can be a nightmare situation for testers. No matter how experienced the testing team is, test flakiness can trouble the heck out of them. It is a hell you should avoid at all costs. But if you ever get sucked into the tornado of flaky tests, knowing how to come out of it is essential. If you have been in the testing field for a while, it is clear that flaky tests don’t offer any value.

They are the most common type of hurdles during mobile app test automation. Every time a testing team performs automated testing, dealing with flaky tests tends to slow down the entire Software Development Life Cycle. Therefore, it is vital to have a comprehensive picture of what test flakiness is, what causes it, and how to deal with flaky tests. Let us take a detailed look.

What are Flaky Tests?

If a test sometimes passes or fails without any valid reason during test automation, it is a flaky test. These tests might work fine for some time but abruptly begin to fail later. This phenomenon sometimes prompts testers to chalk these tests up as a system glitch that leads to ignoring the test result that has already failed.

Other times, a particular test can continue with an inconsistent performance which can encourage the tester to stop running it altogether. It drastically reduces test coverage and can lead to unnoticed failed test results.

You can miss bugs, whether it is a lower test coverage or ignoring failed tests. The changes are also easy to overlook between failing and passing test runs. But it is important to note that a change caused the test result failure.

What Causes Test Flakiness?

The first step toward dealing with test flakiness is to determine its causes. Here they are.

1. Data Management Issues

Running multiple tests with the same account can cause collisions, especially with Appium automation testing. For instance, since one test already uses the account, another might result in a log-in failure.

2. Non-Determinism

Sometimes, application behavior can show non-determinism, leading to test flakiness. It is a feature that displays erratic behaviors or results despite the same input between runs.

3. Problems In The Test Environment

Issues in the test environment are one of the leading causes of test flakiness. For example, an application loads very slowly. It can intermittently go offline or crash, run slower than usual, display network latency, or execute action before a page finishes loading.

4. Compromised Internet Connection Speed

If the traffic load increases on a line, the speed of the internet connection becomes slower and can compromise the visibility of page elements. It causes the test to fail at that particular instant even if it passes when the internet connection speed resumes.

5. Dynamic elements and application DOM

The value of the attributes of dynamic elements changes when you reload the web page. If a test doesn’t cater to dynamic elements, it can return a fail. Similarly, significant changes to the application DOM reflects UI change. It can result in test flakiness in case of an update to the DOM structure, style, or content.

How You Can Deal With Flaky Tests

Since we have already established that a complete escape from test flakiness is not possible, you should give your best to avoid or at least reduce flaky tests. If they still occur, it is crucial to deal with them before they cause further damage. Let us check out this section in detail.

1. Isolating Tests

When you are in the planning phase of your test structure, isolating test cases ensures the independent running of your tests. This improves test performance and prevents any leftover data or side effects from other tests from causing hurdles in your way.

2. Optimization of The Test Structure

Begin with small tests and keep them simple by avoiding the overloading of logic. As a result, your test structure will be more stable and will play a crucial role in preventing test flakiness.

3. No Fixed Waiting Times

If you are looking forward to avoiding or reducing flaky tests for the long haul, never rely on fixed waiting times. Adopt dynamic waiting times instead. If you end up choosing a longer waiting time, the test suit will slow down. Worse, if you don’t wait for enough, the test will fail since it won’t proceed due to the lack of readiness of the application under the test.

4. Separate Test Design and Test Data

Separating test design and test data facilitates the running of data sets via tests is a wise move. It enables you to perform parallel testing to fasten the pace of development and product delivery. Limiting dependency between different tests prevents overall failure. Therefore, you can easily figure out where the problems are arising.

5. Reducing Environmental Dependency

You can start out by limiting external system calls, parametrizing connections, and building mocking servers. Having too much dependency on the environment can backfire as they don’t 100% mimic real-world scenarios. It is not feasible to cut all dependencies. But reducing them is one of the best practices you can undertake to minimize test flakiness.


We are facing an explosion of different varieties of browsers, mobile phones, IoT devices, and so on. It has raised the demand for building easy-to-maintain and scalable automated tests and has simultaneously given rise to roadblocks on the way. Testers already realize that even the slightest changes in the code base or user interface can cause random flaking of automated tests.

Test flakiness also increases maintenance overhead duration and slows down feedback loops. As a result, the entire testing cycle gets hindered. No testing team in this world has ever encountered zero test flakiness. Most Quality Assurance teams just brush It under the rug. They keep doing it until the rug becomes bumpy, and it is impossible to ignore flaky tests anymore!

Original Source:

Continue lendo


Motorola Razr Plus é o novo dobrável rival do Galaxy Z Flip
Após duas tentativas da Motorola em emplacar — novamente — telefones dobráveis, eis que temos aqui a terceira, e aparentemente bem-vinda, tentativa. Estamos falando do Motorola Razr Plus, um smartphone...

Hoje, às 15:20


Mentoring for the LGBTQ+ Community
Once unpublished, all posts by chetanan will become hidden and only accessible to themselves. If chetanan is not suspended, they can still re-publish their posts from their dashboard. Note: Once...

Hoje, às 15:13


IA: mais um arrependido / Déficit de TI / Apple: acusação grave · NewsletterOficial
Mais um pioneiro da IA se arrepende de seu trabalho: Yoshua Bengio teria priorizado segurança em vez de utilidade se soubesse o ritmo em que a tecnologia evoluiria – ele junta-se a Geoffr...

Hoje, às 14:37

Hacker News

The Analog Thing: Analog Computing for the Future
THE ANALOG THING (THAT) THE ANALOG THING (THAT) is a high-quality, low-cost, open-source, and not-for-profit cutting-edge analog computer. THAT allows modeling dynamic systems with great speed,...

Hoje, às 14:25


[DISCUSÃO/OPINIÕES] – Outsourcing! O que, para quem, por que sim, por que não! · dougg
Quero tentar trazer nesta minha primeira publicação, uma mistura de um breve esclarecimento sobre o que são empresas de outsourcing, como elas funcionam e ganham dinheiro, mas também, ven...

Hoje, às 13:58


Duvida: JavaScript - Desenvolver uma aplicação que vai ler um arquivo *.json · RafaelMesquita
Bom dia a todos Estou estudando javascript e me deparei com uma dificuldade e preciso de ajuda *Objetivo do estudo: *desenvolver uma aplicação que vai ler um arquivo *.json Conteudo do in...

Hoje, às 13:43


Automatize suas negociações com um robô de criptomoedas
Índice Como o robô de criptomoedas Bitsgap funciona?Qual a vantagem de utilizar um robô de criptomoedas?Bitsgap é confiável? O mercado de trading tem se tornado cada vez mais popular e as possibilidades de...

Hoje, às 13:13

Hacker News

Sketch of a Post-ORM
I’ve been writing a lot of database access code as of late. It’s frustrating that in 2023, my choices are still to either write all of the boilerplate by hand, or hand all database access over to some...

Hoje, às 13:11


14 chuveiros elétricos para o banho dos seus sonhos
Índice Chuveiro ou Ducha?Tipos de chuveiro elétrico9 fatores importantes para considerar na hora de comprar chuveiros elétricosMelhores chuveiros elétricosDuo Shower LorenzettiFit HydraAcqua Storm Ultra...

Hoje, às 11:00


Learn about the difference between var, let, and const keywords in JavaScript and when to use them.
var, let, and const: What's the Difference in JavaScript? JavaScript is a dynamic and flexible language that allows you to declare variables in different ways. You can use var, let, or const keywords to...

Hoje, às 10:21