More tolerance for flaky tests in libc++ on NetBSD
Summary:
Tests marked with the flaky attribute ("FLAKY_TEST.")
can still report false positives in local tests and on the
NetBSD buildbot.
Additionally a number of tests (probably all threaded
ones) unmarked with the flaky attribute is flaky on
NetBSD.
An ideal solution on the libcxx side would be to raise
max retries for NetBSD and mark failing tests with
the flaky flag, however this adds more maintenance
burden and constant monitoring of flaky tests.
Reduce the work and handle flaky tests as more flaky
on NetBSD and allow flakiness of other tests on
NetBSD.
Reviewers: mgorny, EricWF
Reviewed By: mgorny
Subscribers: christof, llvm-commits, libcxx-commits
Differential Revision: https://reviews.llvm.org/D56064
git-svn-id: https://llvm.org/svn/llvm-project/libcxx/trunk@350170 91177308-0d34-0410-b5e6-96231b3b80d8
diff --git a/utils/libcxx/test/format.py b/utils/libcxx/test/format.py
index 6a334ac..46b2e46 100644
--- a/utils/libcxx/test/format.py
+++ b/utils/libcxx/test/format.py
@@ -12,6 +12,7 @@
import os
import time
import random
+import platform
import lit.Test # pylint: disable=import-error
import lit.TestRunner # pylint: disable=import-error
@@ -202,6 +203,12 @@
for f in os.listdir(local_cwd) if f.endswith('.dat')]
is_flaky = self._get_parser('FLAKY_TEST.', parsers).getValue()
max_retry = 3 if is_flaky else 1
+
+ # LIBC++ tests tend to be more flaky on NetBSD, so add more retries.
+ # We don't do this on other platforms because it's slower.
+ if platform.system() in ['NetBSD']:
+ max_retry = max_retry * 3
+
for retry_count in range(max_retry):
cmd, out, err, rc = self.executor.run(exec_path, [exec_path],
local_cwd, data_files,