That's a good start. It is new enough and with few enough commits that I'd
want to do some thorough experimentation. Our build is complex enough with
a lot of ad hoc coding that we might end up maintaining whatever we
choose...
In my ideal scenario the list of "what else to test" would be manually
editable, or even strictly opt-in. Automatically testing everything that
might be affected quickly runs into scaling problems too. It could make
sense in post-commit but less so in pre-commit.
Kenn
On Wed, Jul 8, 2020 at 3:50 PM Brian Hulette <bhule...@google.com> wrote:
> > We could have one "test the things" Jenkins job if the underlying tool
> (Gradle) could resolve what needs to be run.
>
> I think this would be much better. Otherwise it seems our Jenkins
> definitions are just duplicating information that's already stored in the
> build.gradle files which seems error-prone, especially for tests validating
> combinations of artifacts. I did some quick searching and came across [1].
> It doesn't look like the project has had a lot of recent activity, but it
> claims to do what we need:
>
> > The plugin will generate new tasks on the root project for each task
> provided on the configuration with the following pattern
> ${taskName}ChangedModules.
> > These generated tasks will run the changedModules task to get the list
> of changed modules and for each one will call the given task.
>
> Of course this would only really help us with java tests as gradle doesn't
> know much about the structure of dependencies within the python (and go?)
> SDK.
>
> Brian
>
> [1] https://github.com/ismaeldivita/change-tracker-plugin
>
> On Wed, Jul 8, 2020 at 3:29 PM Kenneth Knowles <k...@apache.org> wrote:
>
>> Hi all,
>>
>> I wanted to start a discussion about getting finer grained test execution
>> more focused on particular artifacts/modules. In particular, I want to
>> gather the downsides and impossibilities. So I will make a proposal that
>> people can disagree with easily.
>>
>> Context: job_PreCommit_Java is a monolithic job that...
>>
>> - takes 40-50 minutes
>> - runs tests of maybe a bit under 100 modules
>> - executes over 10k tests
>> - runs on any change to model/, sdks/java/, runners/, examples/java/,
>> examples/kotlin/, release/ (only exception is SQL)
>> - is pretty flaky (because it conflates so many independent test flakes,
>> mostly runners and IOs)
>>
>> See a scan at
>> https://scans.gradle.com/s/dnuo4o245d2fw/timeline?sort=longest
>>
>> Proposal: Eliminate monolithic job and break into finer-grained jobs that
>> operate on two principles:
>>
>> 1. Test run should be focused on validating one artifact or a specific
>> integration of other artifacts.
>> 2. Test run should trigger only on things that could affect the validity
>> of that artifact.
>>
>> For example, a starting point is to separate:
>>
>> - core SDK
>> - runner helper libs
>> - each runner
>> - each extension
>> - each IO
>>
>> Benefits:
>>
>> - changing an IO or runner would not trigger the 20 minutes of core SDK
>> tests
>> - changing a runner would not trigger the long IO local integration tests
>> - changing the core SDK could potentially not run as many tests in
>> presubmit, but maybe it would and they would be separately reported results
>> with clear flakiness signal
>>
>> There are 72 build.gradle files under sdks/java/ and 30 under runners/.
>> They don't all require a separate job. But still there are enough that it
>> is worth automation. Does anyone know of what options we might have? It
>> does not even have to be in Jenkins. We could have one "test the things"
>> Jenkins job if the underlying tool (Gradle) could resolve what needs to be
>> run. Caching is not sufficient in my experience.
>>
>> (there are other quick fix alternatives to shrinking this time, but I
>> want to focus on bigger picture)
>>
>> Kenn
>>
>
