The 4 stages of flakiness (part 3/3): retrying failed tests in Jenkins

Photo by frank mckenna on Unsplash

The story so far: suffering from tests flakiness, the only way out seems to be a retry script. In part 2 a custom Jest reporter was introduced, logging all failed tests to be used in a future retry script. And here we are!

Now we have to use this information to run those failed tests again instead of failing the full CI. Because we are using Lerna and yarn workspace in Jenkins, the unit tests stage looks like this:

catchError(message: 'Tests failed', stageResult: 'FAILURE') {
   sh 'yarn lerna run test -- --since=origin/master --no-bail'
}

We also have a Jenkinsfile using script notation and the algorithm I first tried was something like:

try {
   sh 'yarn lerna run test -- --since=origin/master --no-bail'
} catch (error) {
   retryUtils.retryFailedTests(error)
}

This worked, mostly, but I had this complicated groovy file looping through failed tests, catching errors and this wasn’t great.
First, it was hard to update existing Jenkins code to use it, at least it wasn’t straightforward. Second point is that it couldn’t easily be tested locally.

Then I read a comment about having complex code in Jenkins: DON’T.

Let it handle stages and pipeline steps, that’s what Jenkins is good for. But what is the alternative?

1. The retry script

I’m no expert on shell script and my knowledge is 99% google, but as I found out… that was a great idea! Not only it was shorter, simpler to write, read and use in the existing Jenkins scripts, but it could also be tested locally anytime very easily!
There’s also a lot of information on the Internet on shell script, which definitely helped solve all my questions.

2. Try…catch in shell?

Turns out that one is super easy.
Commands can be used with || that way:

bash scriptThatCanFail.sh || bash fallback.sh

and fallback.sh will only be executed if the first script fail. So the Jenkins line can be updated to that:

sh 'yarn lerna run test -- --since=origin/master --no-bail || bash retry-tests.sh'

Now if “yarn lerna run test” returns with an error state, it will run “bash retry-tests.sh”. So in every cases where the CI was passing there would be no possible harm since the new code wouldn’t be run at all, while any failing CI would not have passed in the first place anyway.
This is a no-risk update bringing only benefits!

3. Base algorithm

The logic we want include those:

• read failed tests
• start a loop A with the maximum retries we want
• start a loop B that will run each failed tests
• update failed tests list so that at the end of each loop A we have only a list of tests that failed during this retry loop.
• rerun loop A until either we exhausted our retry limit or there’s no failed tests anymore!

4. Read a file

The syntax that we will use to read the file content will be
file_content=$(<path/to/file.txt)

In addition we want to have the file path in a variable as well as read the length of this content:

REPORT_FILE='scripts/test-failures.txt'
failed_suites=$(<${REPORT_FILE})
reportLength=${#failed_suites}
printf "\n\n$failed_suites\n\n"
printf "\n\nreportLength: $reportLength\n\n"

5. Clear file content

Writing content is easy as we just need to output an empty string and redirect it to the file:

echo "" > "${REPORT_FILE}"

6. Loop through the file content

This one start to be more complex. For starters, our “item” separator is the comma:
@workspace file/path,@workspace2 file/path2

read: from the help “Read a line from the standard input and split it into fields”. This can also be used to split the string, based on IFS special shell variable.
This is exactly what we need. By settingIFS=’,’ then call the read command with -a (make it an array), we’ll get an array of our “lines”:

IFS=',' read -ra testsToRun <<< "$failed_suites"

<<< pass the content of our string to the read command line instead of having it wait for user input, so it’s the same as running a script with:

IFS=',' read -ra testsToRun
echo "$testsToRun"

read is going to wait for user input and if we type
@workspace file/path,@workspace2 file/path2
The output should be “@workspace file/path”.

The variable testsToRun is an array and can be looped:

for testLine in "${testsToRun[@]}"
do
   echo "$testLine"
done

Another thing is that our line is actually the package name + file path separated by a space. This needs to be converted to an array again (no need for IFS this time as space is default separator):
read -ra stringarray <<< "$testLine"

The final result is:

IFS=',' read -ra testsToRun <<< "$failed_suites"
for testLine in "${testsToRun[@]}"
do
   read -ra stringarray <<< "$testLine"
   echo "running: 'yarn workspace ${stringarray[0]} test
         ${stringarray[1]}'"
done

7. While loop for max retries

This one is pretty straightforward and we just have to use the doc and updatewhile examples:

retries=0
max_retries=3
while (( $retries < $max_retries && $reportLength > 4 ))
do
   retries=$(( retries+1 ))
   printf "\nretrying tests, attempt $retries\n"
done

8. Register failures but keep going through the loop

Like before we can just run a command and if it fails fallback with ||:

yarn workspace ${stringarray[0]} test ${stringarray[1]} --silent --maxWorkers=1 $noCache || ((hasFailed=1))

9. Full script

10. Conclusion and results

We improved the CI configuration, stabilised tests, documented good practices, trained developers and finally reran failed tests. That was quite a journey and it’s still ongoing!

Though there are still improvements that can be done, having a way to rerun failed tests independently and at the file level helped regain confidence in our CI. There’s no more unexpected delays for merging. Less frustration when waiting in line and it’s becoming easier to know when a work is going to be in production. Master build is green again (most of the time), overall we have reduced or no false positives anymore, making real problems more visible.

Thanks for reading!

Mickael Meausoone — Staff Software Engineer, passionate about JavaScript and web tech in general, curious and likes solving interesting problems. Will code for chocolate.