On 27/11/2024 19.02, Richard Henderson wrote:
On 11/27/24 00:29, Thomas Huth wrote:
On 26/11/2024 23.54, Richard Henderson wrote:
On 11/26/24 11:52, Thomas Huth wrote:
I think we want to continue to maek failing downloads as test failures,
otherwise we'll never notice when an asset is not available from the
internet anymore (since SKIPs just get ignored).
I disagree. Download failures are not rare.
That's not what I said / meant. Sure, servers can have hiccups and
downloads can fail, but that's what we have the cache for. So having a
working cache is essential.
OTOH, if you simply mark tests as SKIP if the download fail, we'll likely
miss if an asset vanishes completely, since some people already have it in
their cache and the remaining people will likely just ignore skipped tests.
If the cache is populated, we will *not* miss if an asset vanishes, because
we won't ever try the URL.
Well, we'll notice it as soon as people run the tests that don't have the
asset in their cache yet.
If the cache is unpopulated, and the download fails, then we cannot run the
test. Indicating FAIL is *useless* because there's nothing that we can do
about it, and we also skip additional tests that CI could be running.
Thinking about it for a little bit longer, I think we might rather want to
distinguish the different failures that can occur during download. If we get
a 404 error, it means that the asset has completely vanished and thus the
test is broken, i.e. that's the case when we want to have a real error, I think.
But if the server is just not responding or gives a 5xx (like 503 or 504)
server error, the failure is likely just a temporary one and we should skip
the test instead. Does that sound acceptable to you? If so, I can look into
creating a patch for this.
Thomas
