Re: [riot-devel] Which tests are expected to succeed for my board
Hi Kees, Indeed that's a lot of failures! Regarding the failures during flash, the flashing over USB workflow is rather fragile: if something went wrong (a crash) when testing an application, the board might no recover and cannot be flashed automatically anymore after. There's also the issue with connecting the terminal application to stdio that can timeout sometimes. The main issues are with the "failures during test" list. I tried some of them on arduino-mkr1000 and also got failures. So there are bugs. Note that you can run each of them individually using "make BOARD= -C tests/ flash test". This is what is called by the "compile_and_test_board.py" script. Regarding "tests/bench_xtimer", you can fix the build issue by adding your board to the LOW_MEMORY_BOARDS list [1]. For the other build failures, I guess some Python packages are missing. Maybe add BUILD_IN_DOCKER=1 to the test command. Alex [1] https://github.com/RIOT-OS/RIOT/blob/master/tests/bench_xtimer/Makefile#L7 - Le 31 Mai 20, à 21:06, Kees Bakker k...@ijzerbout.nl a écrit : > Let me explain a little bit more why I am struggling right now. > > First thing you should know is that I'm creating the support for > a new SODAQ board (Sara SFF). > > Second, I haven't run the automated tests before, so I don't > know what to expect. > > The complete run gives me the result below. No need to comment > on it. I just wanted to let you know what I have to chew on. I will > probably get back with more questions. > > ERROR:sodaq-sara-sff:Tests failed: 110 > Failures during compilation: > - [tests/bench_xtimer](tests/bench_xtimer/compilation.failed) > - [tests/pkg_nanopb](tests/pkg_nanopb/compilation.failed) > - [tests/suit_manifest](tests/suit_manifest/compilation.failed) > > Failures during test: > - [examples/micropython](examples/micropython/test.failed) > - [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed) > - [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed) > - [tests/bloom_bytes](tests/bloom_bytes/test.failed) > - [tests/cond_order](tests/cond_order/test.failed) > - > [tests/cpp11_condition_variable](tests/cpp11_condition_variable/test.failed) > - [tests/cpp11_mutex](tests/cpp11_mutex/test.failed) > - [tests/cpp11_thread](tests/cpp11_thread/test.failed) > - [tests/gnrc_ipv6_nib_dns](tests/gnrc_ipv6_nib_dns/test.failed) > - [tests/mutex_order](tests/mutex_order/test.failed) > - [tests/periph_gpio](tests/periph_gpio/test.failed) > - [tests/periph_rtt](tests/periph_rtt/test.failed) > - [tests/periph_timer](tests/periph_timer/test.failed) > - [tests/periph_timer_periodic](tests/periph_timer_periodic/test.failed) > - > [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed) > - [tests/periph_uart_nonblocking](tests/periph_uart_nonblocking/test.failed) > - [tests/periph_wdt](tests/periph_wdt/test.failed) > - [tests/pkg_c25519](tests/pkg_c25519/test.failed) > - [tests/pkg_cifra](tests/pkg_cifra/test.failed) > - [tests/pkg_cn-cbor](tests/pkg_cn-cbor/test.failed) > - [tests/pkg_hacl](tests/pkg_hacl/test.failed) > - [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed) > - [tests/pkg_libb2](tests/pkg_libb2/test.failed) > - [tests/pkg_libbase58](tests/pkg_libbase58/test.failed) > - [tests/pkg_libcose](tests/pkg_libcose/test.failed) > - [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed) > - [tests/pkg_littlefs](tests/pkg_littlefs/test.failed) > - [tests/pkg_littlefs2](tests/pkg_littlefs2/test.failed) > - [tests/pkg_monocypher](tests/pkg_monocypher/test.failed) > - [tests/pkg_nanocbor](tests/pkg_nanocbor/test.failed) > - [tests/pkg_qcbor](tests/pkg_qcbor/test.failed) > - [tests/pkg_qdsa](tests/pkg_qdsa/test.failed) > - [tests/pkg_relic](tests/pkg_relic/test.failed) > - [tests/pkg_spiffs](tests/pkg_spiffs/test.failed) > - [tests/pkg_tinycbor](tests/pkg_tinycbor/test.failed) > - [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed) > - [tests/pkg_yxml](tests/pkg_yxml/test.failed) > - [tests/posix_semaphore](tests/posix_semaphore/test.failed) > - [tests/progress_bar](tests/progress_bar/test.failed) > - [tests/pthread_tls](tests/pthread_tls/test.failed) > - [tests/riotboot_hdr](tests/riotboot_hdr/test.failed) > - [tests/rmutex](tests/rmutex/test.failed) > - [tests/rmutex_cpp](tests/rmutex_cpp/test.failed) > - [tests/shell](tests/shell/test.failed) > - [tests/sys_arduino](tests/sys_arduino/test.failed) > - [tests/sys_crypto](tests/sys_crypto/test.failed) > - [tests/thread_msg_seq](tests/thread_msg_seq/test.failed) > - [tests/usbus](tests/usbus/test.failed) > > Failures during test.flash: > - [tests/cpp_ctors](tests/cpp_ctors/test.flash.failed) > - [tests/cpp_exclude](tests/cpp_exclude/test.flash.failed) > - [tests/cpp_ext](tests/cpp_ext/test.flash.failed) > - [tests/driver_apds99xx](tests/driver_apds99xx/test.flash.failed) > - [tests/driver_apds99xx_full](tests/driver_apds99xx_full/test.flash.failed) > -
Re: [riot-devel] Which tests are expected to succeed for my board
Let me explain a little bit more why I am struggling right now. First thing you should know is that I'm creating the support for a new SODAQ board (Sara SFF). Second, I haven't run the automated tests before, so I don't know what to expect. The complete run gives me the result below. No need to comment on it. I just wanted to let you know what I have to chew on. I will probably get back with more questions. ERROR:sodaq-sara-sff:Tests failed: 110 Failures during compilation: - [tests/bench_xtimer](tests/bench_xtimer/compilation.failed) - [tests/pkg_nanopb](tests/pkg_nanopb/compilation.failed) - [tests/suit_manifest](tests/suit_manifest/compilation.failed) Failures during test: - [examples/micropython](examples/micropython/test.failed) - [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed) - [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed) - [tests/bloom_bytes](tests/bloom_bytes/test.failed) - [tests/cond_order](tests/cond_order/test.failed) - [tests/cpp11_condition_variable](tests/cpp11_condition_variable/test.failed) - [tests/cpp11_mutex](tests/cpp11_mutex/test.failed) - [tests/cpp11_thread](tests/cpp11_thread/test.failed) - [tests/gnrc_ipv6_nib_dns](tests/gnrc_ipv6_nib_dns/test.failed) - [tests/mutex_order](tests/mutex_order/test.failed) - [tests/periph_gpio](tests/periph_gpio/test.failed) - [tests/periph_rtt](tests/periph_rtt/test.failed) - [tests/periph_timer](tests/periph_timer/test.failed) - [tests/periph_timer_periodic](tests/periph_timer_periodic/test.failed) - [tests/periph_timer_short_relative_set](tests/periph_timer_short_relative_set/test.failed) - [tests/periph_uart_nonblocking](tests/periph_uart_nonblocking/test.failed) - [tests/periph_wdt](tests/periph_wdt/test.failed) - [tests/pkg_c25519](tests/pkg_c25519/test.failed) - [tests/pkg_cifra](tests/pkg_cifra/test.failed) - [tests/pkg_cn-cbor](tests/pkg_cn-cbor/test.failed) - [tests/pkg_hacl](tests/pkg_hacl/test.failed) - [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed) - [tests/pkg_libb2](tests/pkg_libb2/test.failed) - [tests/pkg_libbase58](tests/pkg_libbase58/test.failed) - [tests/pkg_libcose](tests/pkg_libcose/test.failed) - [tests/pkg_libhydrogen](tests/pkg_libhydrogen/test.failed) - [tests/pkg_littlefs](tests/pkg_littlefs/test.failed) - [tests/pkg_littlefs2](tests/pkg_littlefs2/test.failed) - [tests/pkg_monocypher](tests/pkg_monocypher/test.failed) - [tests/pkg_nanocbor](tests/pkg_nanocbor/test.failed) - [tests/pkg_qcbor](tests/pkg_qcbor/test.failed) - [tests/pkg_qdsa](tests/pkg_qdsa/test.failed) - [tests/pkg_relic](tests/pkg_relic/test.failed) - [tests/pkg_spiffs](tests/pkg_spiffs/test.failed) - [tests/pkg_tinycbor](tests/pkg_tinycbor/test.failed) - [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed) - [tests/pkg_yxml](tests/pkg_yxml/test.failed) - [tests/posix_semaphore](tests/posix_semaphore/test.failed) - [tests/progress_bar](tests/progress_bar/test.failed) - [tests/pthread_tls](tests/pthread_tls/test.failed) - [tests/riotboot_hdr](tests/riotboot_hdr/test.failed) - [tests/rmutex](tests/rmutex/test.failed) - [tests/rmutex_cpp](tests/rmutex_cpp/test.failed) - [tests/shell](tests/shell/test.failed) - [tests/sys_arduino](tests/sys_arduino/test.failed) - [tests/sys_crypto](tests/sys_crypto/test.failed) - [tests/thread_msg_seq](tests/thread_msg_seq/test.failed) - [tests/usbus](tests/usbus/test.failed) Failures during test.flash: - [tests/cpp_ctors](tests/cpp_ctors/test.flash.failed) - [tests/cpp_exclude](tests/cpp_exclude/test.flash.failed) - [tests/cpp_ext](tests/cpp_ext/test.flash.failed) - [tests/driver_apds99xx](tests/driver_apds99xx/test.flash.failed) - [tests/driver_apds99xx_full](tests/driver_apds99xx_full/test.flash.failed) - [tests/driver_bme680](tests/driver_bme680/test.flash.failed) - [tests/driver_ds1307](tests/driver_ds1307/test.flash.failed) - [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.flash.failed) - [tests/driver_hd44780](tests/driver_hd44780/test.flash.failed) - [tests/driver_my9221](tests/driver_my9221/test.flash.failed) - [tests/embunit](tests/embunit/test.flash.failed) - [tests/emcute](tests/emcute/test.flash.failed) - [tests/float](tests/float/test.flash.failed) - [tests/fmt_print](tests/fmt_print/test.flash.failed) - [tests/gnrc_dhcpv6_client](tests/gnrc_dhcpv6_client/test.flash.failed) - [tests/gnrc_dhcpv6_client_6lbr](tests/gnrc_dhcpv6_client_6lbr/test.flash.failed) - [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.flash.failed) - [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.flash.failed) - [tests/gnrc_ipv6_ext_opt](tests/gnrc_ipv6_ext_opt/test.flash.failed) - [tests/gnrc_ipv6_fwd_w_sub](tests/gnrc_ipv6_fwd_w_sub/test.flash.failed) - [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.flash.failed) - [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.flash.failed) - [tests/gnrc_ndp](tests/gnrc_ndp/test.flash.failed) - [tests/gnrc_netif](tests/gnrc_netif/test.flash.failed) - [tests/gnrc_rpl_p2p](tests/gnrc_rpl_p2p/test.flash.failed) -
Re: [riot-devel] Which tests are expected to succeed for my board
On 31-05-2020 12:33, Martine Sophie Lenders wrote: > Hi, > > Am 31.05.20 um 12:25 schrieb Kaspar Schleiser: >> "... FAILED (due to $reason)", and maybe not change return code to >> something that's an error. > While I like this idea I foresee a usage problem since some tests in > that script fail due to missing root permissions (or missing > scapy_unroot). If we say 'FAILED (due to missing root permissions)' I > virtually expect any sane developer to re-run the script then with > `sudo`, leading to people wondering why they are not able to compile > applications anymore, just using `make all`, as the `bin` directory was > created with root permission. Sure this can be easily fixed, I just say > "mind the consequences" ;-). > I don't agree with the "sane" part in that sentence. ;-) -- Kees ___ devel mailing list devel@riot-os.org https://lists.riot-os.org/mailman/listinfo/devel
Re: [riot-devel] Which tests are expected to succeed for my board
Hi, Am 31.05.20 um 12:25 schrieb Kaspar Schleiser: > "... FAILED (due to $reason)", and maybe not change return code to > something that's an error. While I like this idea I foresee a usage problem since some tests in that script fail due to missing root permissions (or missing scapy_unroot). If we say 'FAILED (due to missing root permissions)' I virtually expect any sane developer to re-run the script then with `sudo`, leading to people wondering why they are not able to compile applications anymore, just using `make all`, as the `bin` directory was created with root permission. Sure this can be easily fixed, I just say "mind the consequences" ;-). Best regards, Martine > > Kaspar > ___ > devel mailing list > devel@riot-os.org > https://lists.riot-os.org/mailman/listinfo/devel ___ devel mailing list devel@riot-os.org https://lists.riot-os.org/mailman/listinfo/devel
Re: [riot-devel] Which tests are expected to succeed for my board
Hi, On 5/30/20 10:14 PM, Alexandre Abadie wrote: > You can just put all of them in the same issue. It will be easier to track. > That is what is done in [1]. I think we should add this information to compile_and_test.py. Some way to expect failure for specific tests or board/test combinations, and report accordingly. "... FAILED (due to $reason)", and maybe not change return code to something that's an error. Kaspar ___ devel mailing list devel@riot-os.org https://lists.riot-os.org/mailman/listinfo/devel
