On 05/06/2019 22:11, Sam Tobin-Hochstadt wrote:
> There are a few things here:
>
Thanks for the clarification.
> 1. The set of tests is selected mostly based on what doesn't have a
> lot of dependencies, tests things in the `racket/racket` repository,
> and completes in the Travis CI time limit.
If there are more tests I should run please let me know. :) From what
you say the lists do not seem to be complete. Even if we do not run all
at every commit we could do it overnight - as we do for emulation
testing. Also, with GitlabCI we don't have a time limit anymore.
> 2. Some tests are run under `racket` vs `raco test` for historical
> reasons, probably that could be fixed. I originally took the command
> lines from the release checklist.
OK.
> 3. The tests are all together because building racket takes most of
> the time, and therefore splitting the tests didn't make sense. For a
> system with better pipeline support, splitting things up could work.
>
I now own a large server with 40cores and bucket loads of memory. In
total I now have about 70cores at home in servers for Racket CI.
Building racket3m takes about 3mins while building racketcs takes about
8mins with gcc -O3. I have the ability now to just run several tiny jobs
simultaneously which means I can have finer grained control over what's
passing and failing and mark it accordingly.
> One important background piece of information is that because
> drdr.racket-lang.org exists, typically other CI systems have focused
> on either system coverage or configuration coverage, and mostly on the
> core tests. Obviously that isn't ideal, but even before it tested
> RacketCS, DrDr took 8+ CPU hours to run a single full test run, and so
> doing that in other CI systems wasn't possible.
>
Now it is... :) What do you mean by a single full test run? All the
tests below, or running even further tests?
> Sam
>
> On Wed, Jun 5, 2019 at 3:49 PM 'Paulo Matos' via Racket Developers
> <[email protected]> wrote:
>>
>> Hi,
>>
>> I was looking at exploding the Racket CI configurations to make them
>> more fine-grained.
>>
>> For non x86_64 architectures with default flags / racketcs etc, there
>> are many failures and I would like to make the testing more fine-grained
>> so that I can lock a configuration /to success/ as soon as possible.
>> What I mean by this is... most our jobs at the moment have:
>> 'allow_failure: true'. I want to reduce these.
>>
>> Instead of having a testing job that does:
>> - raco test -l tests/racket/test 2>&1 | tee logs/test.log
>> - racket -l tests/racket/contract/all 2>&1 | tee logs/contract-test.log
>> - raco test -l tests/json/json 2>&1 | tee logs/json-test.log
>> - raco test -l tests/file/main 2>&1 | tee logs/file-test.log
>> - raco test -l tests/net/head 2>&1 | tee logs/net-head-test.log
>> - raco test -l tests/net/uri-codec 2>&1 | tee
>> logs/net-uri-codec-test.log
>> - raco test -l tests/net/url 2>&1 | tee logs/net-url-test.log
>> - raco test -l tests/net/url-port 2>&1 | tee logs/net-url-port-test.log
>> - raco test -l tests/net/encoders 2>&1 | tee logs/net-encoders-test.log
>> - raco test -l tests/openssl/basic 2>&1 | tee
>> logs/openssl-basic-test.log
>> - raco test -l tests/openssl/https 2>&1 | tee
>> logs/openssl-https-test.log
>> - raco test -l tests/match/main 2>&1 | tee logs/match-main-test.log
>> - raco test -l tests/zo-path 2>&1 | tee logs/zo-path-test.log
>> - raco test -l tests/xml/test 2>&1 | tee logs/xml-test.log
>> - raco test -l tests/db/all-tests 2>&1 | tee logs/db-test.log
>> - raco test -c tests/stxparse 2>&1 | tee logs/stxparse-test.log
>>
>> When one of them fails, we silence the failure with 'allow_failure:
>> true'. The problem with this is that if a problem shows up somewhere
>> else, it will go unnoticed because we allow failure by default. I want
>> to make each of these a single job and allow failure only on those which
>> really fail and once they pass, lock them onto success by not allowing
>> failure anymore.
>>
>> I initially copied the above lines without the redirections from travis
>> yml. Is this list of tests up-to-date? Is this all we have and are they
>> being run correctly?
>>
>> I should point out I find it slightly strange that the tests seem to be
>> implemented in different ways since the contract tests are ran with
>> `racket -l`, and stxparse are ran with `raco test -c`. What's the reason
>> for this difference?
>>
>> Kind regards,
>> --
>> Paulo Matos
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Racket Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-dev/a010e885-4f63-44d4-e73e-5457a0159fdc%40linki.tools.
>> For more options, visit https://groups.google.com/d/optout.
>
--
Paulo Matos
--
You received this message because you are subscribed to the Google Groups
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-dev/25e833d2-b792-8fc1-d5f6-e1446f282a46%40linki.tools.
For more options, visit https://groups.google.com/d/optout.
