Hi again,
With the support of ASF Infra, we now are in a position to run arbitrary
alternative platforms in our CI builds. This is great news for the
people who have been waiting for ARM (aarch64), PowerPC/POWER (ppc64le),
mainframe (s390x), and other architecture supports.
However, we have a challenge. Emulating other architectures is
necessarily slower than running natively on provided hardware. I ran
some measurements today, and found that we're not able to pass our test
suites because tests are timing out.
For example, on native x86_64 hardware, this test finishes in under half
a second:
b64url_tests: encode_binary_test...[0.372s] ok
Running the same test on aarch64, being emulated on the same x86_64 machine:
b64url_tests: encode_binary_test...[4.493 s] ok
or 12x longer. The next test (decode_iolist_test) fails, presumably
because it hits a 5s timeout period.
I need advice from the list. Should we:
* Increase the test timeouts significantly so these tests can complete
* Restrict ourselves to running only on actual hardware (which limits us
only to aarch64, and at a stretch, ppc64le)
* Remove test timeouts entirely, rewriting the ones that wait forever to
do something different
* Something else that I haven't mentioned
Having regression testing on a variety of platforms is something that is
of benefit to the project; we've ignored it for too long.
-Joan
