On 11/27/24 00:29, Thomas Huth wrote:
On 26/11/2024 23.54, Richard Henderson wrote:
On 11/26/24 11:52, Thomas Huth wrote:
I think we want to continue to maek failing downloads as test failures, otherwise we'll
never notice when an asset is not available from the internet anymore (since SKIPs just
get ignored).
I disagree. Download failures are not rare.
That's not what I said / meant. Sure, servers can have hiccups and downloads can fail, but
that's what we have the cache for. So having a working cache is essential.
OTOH, if you simply mark tests as SKIP if the download fail, we'll likely miss if an asset
vanishes completely, since some people already have it in their cache and the remaining
people will likely just ignore skipped tests.
If the cache is populated, we will *not* miss if an asset vanishes, because we won't ever
try the URL.
If the cache is unpopulated, and the download fails, then we cannot run the test.
Indicating FAIL is *useless* because there's nothing that we can do about it, and we also
skip additional tests that CI could be running. If we skip, then we simply defer the test
to the next CI run. This isn't perfect, but it really is better.
Honestly, flaky test failures are the *worst*, because those quickly get
ignored too.
Download failures go into the "flaky" bucket.
r~
