On Thu, 30 Nov 2023, Rémi Denis-Courmont wrote:
You can already test it properly as things stand, and reporting is trivial,
just not to the FATE website. The question is whether this is worth adding to
FATE.
More public test coverage is better than less, isn't it?
In other words, is publishing on the FATE website worth making the tests
coverage and/or the build time worse?
By making the test coverage worse, you mean if I'd be doing the full
testing of many combinations already, and I'd stop doing that in order to
do this lesser testing instead? If I'd be doing it (I currently don't) I
guess that would be my concern, not others?
And whether I want to spend my cpu cycles on testing for a public FATE
config, even if it only tests a limited amount, in addition what tests I
(hypothetically) run privately, wasting more CPU on building ffmpeg,
that's also my concern and not others?
not to mention confusing the existing website users with weirdly
incomplete test results.
FWIW, there are a bunch of otherwise weird, limited configs as well, e.g.
http://fate.ffmpeg.org/history.cgi?slot=x86_64-archlinux-gcc-disableavcodec.
Although there, the reason for the limited number of tests is clearly
visible.
Again, for SVE, I'd rather have testing with 1 config (the default, which
is longer vectors than one usually encounters in HW) rather than none at
all. It won't catch every theoretical issue but practically would catch
many things at least.
I find that statement very misleading. This is not a question of testing 1
config vs 0. It's a question of testing 1 configuration vs all of them(*), and
reporting that one vs reporting all of them elsewhere than FATE.ffmpeg.org.
Until/unless somebody does the missing integration.
Currently I test 0 of these configurations. I would like to test 1 such
config, and publish those results on the FATE website. I don't currently
test any form of "all configs". And if I wanted to make a private setup
for testing "all configs", I really don't see how it would be mutually
exclusive with the publicly posted test results from the one config?
And in order to actually test BTI, one has to link with a sysroot that
also was built with BTI enabled - I currently use a sysroot extracted from
fedora for that. (And my tests for it use -Wl,-z,force-bti.)
I can readily believe how much of a PITA that would be to set up. I can also
believe that glibc won't allow masking the guarded page bit in mmap()/
mprotect().
That does not mean you need different builds to test each of the 4 possible
combinations (or 3 if you ignore the case of BTI without PAC, which does not
exist in real hardware). Once you have that build, you can test it with
whichever QEMU CPU settings.
I didn't mean to imply that one would have to do separate builds for all
of those. I currently don't do any testing with builds with
-mbranch-protection=standard (and with different sysroots), but I was
considering adding one such build, with the fedora sysroot - and only test
one single configuration with it (with QEMU's defaults of all features
enabled).
So, to spell out your objection in simpler terms. You are firmly against
anybody posting test results on FATE that only include checkasm but not
the rest of the tests, because you consider that this can be
misleading/confusing to people reading the test results - is that right?
Or would such a setup be acceptable to you, if someone would implement a
way of running the tests (either the full set or only a subset such as
chckasm) multiple times with different QEMU configurations, with the same
build of ffmpeg, within the same FATE run?
// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
