On Fri, 16 Dec 2016 13:19:51 -0800 Cedric BAIL <cedric.b...@free.fr> said:
> On Fri, Dec 16, 2016 at 2:43 AM, Gustavo Sverzut Barbieri
> <barbi...@gmail.com> wrote:
> > On Fri, Dec 16, 2016 at 7:54 AM, Stefan Schmidt <ste...@osg.samsung.com>
> > wrote:
> >> On 13/12/16 16:32, Gustavo Sverzut Barbieri wrote:
> >>> On Tue, Dec 13, 2016 at 11:54 AM, Stefan Schmidt <ste...@osg.samsung.com>
> >>> wrote:
> >>>> On 13/12/16 14:09, Felipe Magno de Almeida wrote:
> >>>>> On Tue, Dec 13, 2016 at 8:28 AM, Stefan Schmidt
> >>>>> <ste...@osg.samsung.com> wrote:
>
> [snip]
>
> >>>> but a test that fails on a machine that doesn't
> >>>>> handle IPv6 seems fine to me as long as it is rare enough.
> >>>>
> >>>> We assume IPv6 now, we assume a working internet connection, we assume
> >>>> udev for eeze testing, we assume a working dbus setup, in some cases we
> >>>> assume a file system which supports extended attributes, etc...
> >>>>
> >>>> If it is complicated to run make check we will have even less people
> >>>> running it. It should be the other way around. I guess I could look at
> >>>> anybody here who contributed a few patches this year and see someone who
> >>>> broke make check. If it is to complicated or fails for some reason
> >>>> people will just stop using it.
> >>>
> >>> our test suite is unusable the way it is...
> >>
> >> While I agree that the test suite needs improvements calling it unusable
> >> is over the top.
> >
> > ok, so improved... particularly for developers to run them... also
> > something that simplifies to write them, and how to do it properly.
> >
> > For instance, very few tests uses correct macros in check.h, like
> > ck_assert_int_eq(), that will highlight expected values and required
> > values, compared to failif(a == b). Some tests miss granularity, so
> > they do bunch of tests at once, missing the "unit test" aspect... and
> > while we can manually select the test suite to run (calling it
> > directly), and sometimes the component (ie: ecore_con_suite -h to list
> > them), it would be very helpful to select individual functions to
> > test, particularly under development. And some helpers to debug
> > failing tests (I always need to check for CK_FORK=no and things like
> > that).
>
> I agree with you on the missing granularity. Some tests are awful in
> that regard. Overall I like most of your point here, it is just so
> massive and our ressource are spread so thinly that it is not
> something anyone working on the core as time to work on. But this task
> are pretty simple and easy, they are quite an easy training to learn
> our API, environment and contribute their first patches. We could have
> a beginner/new comer section in our wiki for developers to know about
> this.
>
> As for the helper, a simple shell script that set CK_RUN_CASE, CK_FORK
> and start the test suite under valgrind + gdb shouldn't be to hard I
> think. I personnaly always rely on CK_RUN_CASE + valgrind
> --trace-children and do not bother with gdb, but I can see its use.
i also dislike our test suite.
1. barrier of entry. simply it takes too much footwork to add a simple 1 or 2
line test, so what everyone ends up doing is making 1 tests case that is 20 or
50 lines testing lots of things to share the footwork across multiple tests
2. the forking thing check does by default is just a pain. i keep having to set
up CK_FORK=no
3. to then run under either gdb or valgrind i hunt the correct binary down in
tests/testname/.libs dir and run it.
what i'd love to see is something like:
a directory per "dependencies" (#includes and linking info) with some master
file that lists init and shutdown code for all test in that dir. it may be as
simple as #include <Eina.h> + eina_init/shutdown or may include a dozen efl
headers init a dozen efl libs, create a canvas/window with a default background
etc.
in this directory then every file named test_something.c can just be:
test_list_append.c:
Eina_List *list = NULL;
list = eina_list_append(list, (void *)123L);
FAIL_IF(eina_list_data_find_list(list, (void *)123L) != list);
boilerplate-free. no need to edit a makefile.am. no need to declare a function
and then put it in prototypes and add it to a list of funcs for check to run...
we should have a script or something that just does this automatically. i'd
want 1 binary per test too. explicitly. if you need callbacks maybe have
test_list_append.cb.c:
static void my_cb(void *data)
{
// whatever
}
list all such funcs to that file and if it exists it'll be put above the test
that would be in main. the per-fir dependencies are wrapped around with the
includes at the top, a main function auto-generated and in that all init calls,
and then the test and then shutdown calls.
something like this would be awfully nicer than what we have.
> > what happens in most of the cases is that the suite is painful to deal
> > with so it's easier to write the "other required" bit that is the
> > example and test from there, rather than do it from the test suite...
> > which is what I did, and some guys follow the same pattern (or just
> > commit to enlightenment an usage).
> >
> >> really it lacks
> >>> granularity, it takes lots of time and most of the time it looks like
> >>> it failed since people forget to trap EINA_LOG to capture errors, then
> >>> you get tons of error messages (although the test reports success,
> >>> messages confuses users).
> >>
> >> In the normal run you see a green PASS or red FAILED. The eina_log
> >> errors are only in the log file itself.
> >
> > still, this is for build bot or random users running the tests. Not to
> > us, the developers... you're going to run the full test suite after
> > each development step... becomes costly and painful :-/
>
> I agree with Gustavo here I think. Having all our tests hidden behind
> PASS/FAILED make it a problem to deal with when running make check as
> you have to read somewhere else what is going on. Would be neat if
> they where not directed to a file actually... but then we really
> should be more careful with our printf/log !
> --
> Cedric BAIL
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
>
--
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler) ras...@rasterman.com
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
