hi, On Mon, Aug 6, 2018 at 7:52 AM, Antoine Pitrou <anto...@python.org> wrote: > > Le 06/08/2018 à 13:42, Wes McKinney a écrit : >> hi Antoine, >> >> I completely agree. Part of why I've been so consistently pressing for >> nightly build tooling is to be able to shift more exhaustive testing >> out of per-commit runs into a daily build or an on-demand build to be >> invoked by the user either manually or by means of a bot. If you >> search in JIRA for the term "nightly" you can see a lot of issue that >> I have created for this already. > > My worry with nightly jobs is that they doesn't clearly pinpoint the > specific changeset where a regression occurred; also, since it happens > after merging, there is less incentive to clear any mess introduced by a > PR; moreover, it makes trying out a fix more difficult, since you don't > get direct feedback on a commit or PR. > > So in exchange for lowering per-commit CI times, nightly jobs require > extra human care to track regressions and make sure they get fixed.
The way that projects with much more complex testing deal with this is to have a pre-commit test run. The Apache Impala "verify merge" commit take several hours to run, for example. I don't think we are on our way to that point yet. What I'm suggesting is that we be able to write @arrow-test-bot please run exhaustive There are a lot of patches that clearly don't require exhaustive testing. In any case, we are likely to accumulate more testing than we can run on every commit / patch, but as long as we have a convenient mechanism to run the tests, then it is OK. > >> it would be useful to be >> able to validate if desired (and in an automated way) that the NMake >> build works properly > > I guess the main question is why we're testing for NMake at all. CMake > supports a range of different build tools, we can't exercise all of > them. So I'd say on each platform we should exercise at most two build > tools: > - Ninja, because it's cross-platform and the fastest (which makes it > desirable for developers *and* for CI) > - the standard platform-specific build tool, i.e. GNU make on Unix and > Visual Studio (or "msbuild") on Windows Well, I would be OK with dropping NMake in favor of supporting Ninja. > >> I think we can also improve CI build times by caching certain >> toolchain artifacts that are taking a long time to build (Thrift, I'm >> looking at you). > > The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. > Don't they include thrift? The most time consuming parts of the job are AFAIK: * The asv benchmarks * Code coverage uploads * Using valgrind * Building multiple versions of Python These can be addressed respectively by: * Nightly/opt-in testing asv (via bot) * Nightly/opt-in coverage * Nightly valgrind * No immediate solution. I would like to drop Python 2 in January 2019 - Wes > > (another thought: when do we want to drop Python 2 compatibility?) > > Regards > > Antoine.
