Thanks Alex, The tests that need sudo, I've skipped. For now I don't care about these. I was able to rerun quite a few tests by skipping the ones that caused the the board to hang. I also skip all the tests that need sudo. For now I don't care about these.
I'll look at LOW_MEMORY_BOARDS, but if other SAMD21 don't need it, then my board shouldn't need it either. I also need to look at my execution environment. I don't docker (yet), so maybe I'm using incompatible gcc and/or newlib. I've seen strange problems with puts/printf, where part of the output is missing (and thus I get test timeouts). -- Kees On 31-05-2020 21:40, Alexandre Abadie wrote: > Hi Kees, > > Indeed that's a lot of failures! > > Regarding the failures during flash, the flashing over USB workflow is rather > fragile: if something went wrong (a crash) when testing an application, the > board might no recover and cannot be flashed automatically anymore after. > There's also the issue with connecting the terminal application to stdio that > can timeout sometimes. > > The main issues are with the "failures during test" list. I tried some of > them on arduino-mkr1000 and also got failures. So there are bugs. > > Note that you can run each of them individually using "make BOARD=<your > board> -C tests/<test application> flash test". This is what is called by the > "compile_and_test_board.py" script. > > Regarding "tests/bench_xtimer", you can fix the build issue by adding your > board to the LOW_MEMORY_BOARDS list [1]. For the other build failures, I > guess some Python packages are missing. Maybe add BUILD_IN_DOCKER=1 to the > test command. > > Alex > > [1] https://github.com/RIOT-OS/RIOT/blob/master/tests/bench_xtimer/Makefile#L7 > > ----- Le 31 Mai 20, à 21:06, Kees Bakker k...@ijzerbout.nl a écrit : > >> Let me explain a little bit more why I am struggling right now. >> >> First thing you should know is that I'm creating the support for >> a new SODAQ board (Sara SFF). >> >> Second, I haven't run the automated tests before, so I don't >> know what to expect. >> >> The complete run gives me the result below. No need to comment >> on it. I just wanted to let you know what I have to chew on. I will >> probably get back with more questions. >> >> ERROR:sodaq-sara-sff:Tests failed: 110 >> Failures during compilation: >> - [tests/bench_xtimer](tests/bench_xtimer/compilation.failed) >> - [tests/pkg_nanopb](tests/pkg_nanopb/compilation.failed) >> - [tests/suit_manifest](tests/suit_manifest/compilation.failed) >> >> Failures during test: >> - [examples/micropython](examples/micropython/test.failed) >> - [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed) >> - [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed) >> - [tests/bloom_bytes](tests/bloom_bytes/test.failed) >> - [tests/cond_order](tests/cond_order/test.failed) >> - >> [tests/cpp11_condition_variable](tests/cpp11_condition_variable/test.failed) >> - [tests/cpp11_mutex](tests/cpp11_mutex/test.failed) >> - [tests/cpp11_thread](tests/cpp11_thread/test.failed) >> - [tests/gnrc_ipv6_nib_dns](tests/gnrc_ipv6_nib_dns/test.failed) >> - [tests/mutex_order](tests/mutex_order/test.failed) >> - [tests/periph_gpio](tests/periph_gpio/test.failed) >> - [tests/periph_rtt](tests/periph_rtt/test.failed) >> - [tests/periph_timer](tests/periph_timer/test.failed) >> - [tests/periph_timer_periodic](tests/periph_timer_periodic/test.failed) >> - >> [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed) >> - [tests/periph_uart_nonblocking](tests/periph_uart_nonblocking/test.failed) >> - [tests/periph_wdt](tests/periph_wdt/test.failed) >> - [tests/pkg_c25519](tests/pkg_c25519/test.failed) >> - [tests/pkg_cifra](tests/pkg_cifra/test.failed) >> - [tests/pkg_cn-cbor](tests/pkg_cn-cbor/test.failed) >> - [tests/pkg_hacl](tests/pkg_hacl/test.failed) >> - [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed) >> - [tests/pkg_libb2](tests/pkg_libb2/test.failed) >> - [tests/pkg_libbase58](tests/pkg_libbase58/test.failed) >> - [tests/pkg_libcose](tests/pkg_libcose/test.failed) >> - [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed) >> - [tests/pkg_littlefs](tests/pkg_littlefs/test.failed) >> - [tests/pkg_littlefs2](tests/pkg_littlefs2/test.failed) >> - [tests/pkg_monocypher](tests/pkg_monocypher/test.failed) >> - [tests/pkg_nanocbor](tests/pkg_nanocbor/test.failed) >> - [tests/pkg_qcbor](tests/pkg_qcbor/test.failed) >> - [tests/pkg_qdsa](tests/pkg_qdsa/test.failed) >> - [tests/pkg_relic](tests/pkg_relic/test.failed) >> - [tests/pkg_spiffs](tests/pkg_spiffs/test.failed) >> - [tests/pkg_tinycbor](tests/pkg_tinycbor/test.failed) >> - [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed) >> - [tests/pkg_yxml](tests/pkg_yxml/test.failed) >> - [tests/posix_semaphore](tests/posix_semaphore/test.failed) >> - [tests/progress_bar](tests/progress_bar/test.failed) >> - [tests/pthread_tls](tests/pthread_tls/test.failed) >> - [tests/riotboot_hdr](tests/riotboot_hdr/test.failed) >> - [tests/rmutex](tests/rmutex/test.failed) >> - [tests/rmutex_cpp](tests/rmutex_cpp/test.failed) >> - [tests/shell](tests/shell/test.failed) >> - [tests/sys_arduino](tests/sys_arduino/test.failed) >> - [tests/sys_crypto](tests/sys_crypto/test.failed) >> - [tests/thread_msg_seq](tests/thread_msg_seq/test.failed) >> - [tests/usbus](tests/usbus/test.failed) >> >> Failures during test.flash: >> - [tests/cpp_ctors](tests/cpp_ctors/test.flash.failed) >> - [tests/cpp_exclude](tests/cpp_exclude/test.flash.failed) >> - [tests/cpp_ext](tests/cpp_ext/test.flash.failed) >> - [tests/driver_apds99xx](tests/driver_apds99xx/test.flash.failed) >> - [tests/driver_apds99xx_full](tests/driver_apds99xx_full/test.flash.failed) >> - [tests/driver_bme680](tests/driver_bme680/test.flash.failed) >> - [tests/driver_ds1307](tests/driver_ds1307/test.flash.failed) >> - [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.flash.failed) >> - [tests/driver_hd44780](tests/driver_hd44780/test.flash.failed) >> - [tests/driver_my9221](tests/driver_my9221/test.flash.failed) >> - [tests/embunit](tests/embunit/test.flash.failed) >> - [tests/emcute](tests/emcute/test.flash.failed) >> - [tests/float](tests/float/test.flash.failed) >> - [tests/fmt_print](tests/fmt_print/test.flash.failed) >> - [tests/gnrc_dhcpv6_client](tests/gnrc_dhcpv6_client/test.flash.failed) >> - >> [tests/gnrc_dhcpv6_client_6lbr](tests/gnrc_dhcpv6_client_6lbr/test.flash.failed) >> - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.flash.failed) >> - [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.flash.failed) >> - [tests/gnrc_ipv6_ext_opt](tests/gnrc_ipv6_ext_opt/test.flash.failed) >> - [tests/gnrc_ipv6_fwd_w_sub](tests/gnrc_ipv6_fwd_w_sub/test.flash.failed) >> - [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.flash.failed) >> - [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.flash.failed) >> - [tests/gnrc_ndp](tests/gnrc_ndp/test.flash.failed) >> - [tests/gnrc_netif](tests/gnrc_netif/test.flash.failed) >> - [tests/gnrc_rpl_p2p](tests/gnrc_rpl_p2p/test.flash.failed) >> - [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.flash.failed) >> - [tests/gnrc_sixlowpan](tests/gnrc_sixlowpan/test.flash.failed) >> - [tests/gnrc_sixlowpan_frag](tests/gnrc_sixlowpan_frag/test.flash.failed) >> - >> [tests/gnrc_sixlowpan_iphc_w_vrb](tests/gnrc_sixlowpan_iphc_w_vrb/test.flash.failed) >> - >> [tests/gnrc_sock_async_event](tests/gnrc_sock_async_event/test.flash.failed) >> - [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.flash.failed) >> - [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.flash.failed) >> - [tests/gnrc_sock_neterr](tests/gnrc_sock_neterr/test.flash.failed) >> - [tests/gnrc_sock_udp](tests/gnrc_sock_udp/test.flash.failed) >> - [tests/gnrc_tcp](tests/gnrc_tcp/test.flash.failed) >> - [tests/heap_cmd](tests/heap_cmd/test.flash.failed) >> - [tests/irq](tests/irq/test.flash.failed) >> - [tests/irq_disable_restore](tests/irq_disable_restore/test.flash.failed) >> - [tests/isr_yield_higher](tests/isr_yield_higher/test.flash.failed) >> - [tests/kconfig](tests/kconfig/test.flash.failed) >> - [tests/l2util](tests/l2util/test.flash.failed) >> - [tests/libc_newlib](tests/libc_newlib/test.flash.failed) >> - [tests/log_color](tests/log_color/test.flash.failed) >> - [tests/log_printfnoformat](tests/log_printfnoformat/test.flash.failed) >> - [tests/lwip_sock_ip](tests/lwip_sock_ip/test.flash.failed) >> - [tests/lwip_sock_tcp](tests/lwip_sock_tcp/test.flash.failed) >> - [tests/lwip_sock_udp](tests/lwip_sock_udp/test.flash.failed) >> - [tests/malloc](tests/malloc/test.flash.failed) >> - [tests/memarray](tests/memarray/test.flash.failed) >> - [tests/msg_avail](tests/msg_avail/test.flash.failed) >> - [tests/msg_send_receive](tests/msg_send_receive/test.flash.failed) >> - [tests/msg_try_receive](tests/msg_try_receive/test.flash.failed) >> - [tests/mtd_flashpage](tests/mtd_flashpage/test.flash.failed) >> - [tests/mtd_mapper](tests/mtd_mapper/test.flash.failed) >> - [tests/pkg_tensorflow-lite](tests/pkg_tensorflow-lite/test.flash.failed) >> - [tests/stdin](tests/stdin/test.flash.failed) >> - [tests/vfs_plus_stdio](tests/vfs_plus_stdio/test.flash.failed) >> - >> [tests/xtimer_mutex_lock_timeout](tests/xtimer_mutex_lock_timeout/test.flash.failed) >> - >> [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.flash.failed) >> >> >> On 30-05-2020 22:19, Kees Bakker wrote: >>> Well, not easier for me, I'm afraid. It will delay adding my board >>> even longer (which is already several months in progress). >>> And it's just a hobby for me. Otherwise it begins to look like >>> work :-) >>> >>> Nevertheless, I'll see what I can do. >>> >>> On 30-05-2020 22:14, Alexandre Abadie wrote: >>>> Hi, >>>> >>>> You can just put all of them in the same issue. It will be easier to >>>> track. That >>>> is what is done in [1]. >>>> >>>> Alex >>>> >>>> [1] https://github.com/RIOT-OS/RIOT/issues/12651 >>>> >>>> ----- Le 30 Mai 20, à 21:42, Kees Bakker k...@ijzerbout.nl a écrit : >>>> >>>>> OK, first one created. I'm afraid there are many more to come. >>>>> >>>>> On 30-05-2020 21:27, Alexandre Abadie wrote: >>>>>> Hi Kees, >>>>>> >>>>>> To my knowledge, there is no wiki page for this kind of thing. >>>>>> The simplest is probably to open an issue and list there the failures >>>>>> with their >>>>>> output. This way we can easily track the on going work to fix them. >>>>>> >>>>>> See [1] as an example. >>>>>> >>>>>> Alex >>>>>> >>>>>> [1] https://github.com/RIOT-OS/RIOT/issues/12651 >>>>>> >>>>>> ----- Le 29 Mai 20, à 23:14, Kees Bakker k...@ijzerbout.nl a écrit : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Now that I'm happily running automated tests for my SAMD21 board(s) I am >>>>>>> wondering which tests should succeed. There are several that fail, but I >>>>>>> don't know if that "normal" or not. >>>>>>> >>>>>>> Some examples of test that fail >>>>>>> >>>>>>> xtimer_periodic_wakeup: hangs at the end, last couple of printf don't >>>>>>> come out >>>>>>> tests/thread_flood: [ERROR] expected 30, created 29 >>>>>>> >>>>>>> Is there maybe a wiki with the status of the tests (per board?). >>>>>>> -- >>>>>>> Kees >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> devel@riot-os.org >>>>>>> https://lists.riot-os.org/mailman/listinfo/devel >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> devel@riot-os.org >>>>>> https://lists.riot-os.org/mailman/listinfo/devel >>>>> _______________________________________________ >>>>> devel mailing list >>>>> devel@riot-os.org >>>>> https://lists.riot-os.org/mailman/listinfo/devel >>>> _______________________________________________ >>>> devel mailing list >>>> devel@riot-os.org >>>> https://lists.riot-os.org/mailman/listinfo/devel >>> _______________________________________________ >>> devel mailing list >>> devel@riot-os.org >>> https://lists.riot-os.org/mailman/listinfo/devel >> _______________________________________________ >> devel mailing list >> devel@riot-os.org >> https://lists.riot-os.org/mailman/listinfo/devel > _______________________________________________ > devel mailing list > devel@riot-os.org > https://lists.riot-os.org/mailman/listinfo/devel _______________________________________________ devel mailing list devel@riot-os.org https://lists.riot-os.org/mailman/listinfo/devel
