Good idea on having a flag to make the test run without GCP credentials. I logged this as https://issues.apache.org/jira/browse/BEAM-5771 and can look into this, unless someone else prefers.
On Tue, Oct 16, 2018 at 9:18 AM Scott Wegner <[email protected]> wrote: > There was some discussion recently about ensuring anyone can easily run > and reproduce precommit test results locally. The precommits run Dataflow > jobs, which will fail if you don't have access to an Google Cloud project. > One idea would be to add a flag to disable Google Cloud tests, i.e. > ./gradlew :javaPreCommit -PdisableGcpTests > > On Tue, Oct 16, 2018 at 8:57 AM Kenneth Knowles <[email protected]> wrote: > >> Yes, it is exactly that. The :javaPreCommit is a deliberate attempt to >> make a single task that runs all the tests that Jenkins runs, so it >> includes some lightweight smoke tests on runners, including Google Cloud >> Dataflow. >> >> With maven it was impossible to have a single mvn invocation that would >> build what was necessary but only run the ITs so it was necessary to >> conflate the two (we would use a bash Jenkins job with a few mvn >> invocations in series, losing the Jenkins Maven Plugin integration). >> >> With gradle it is easy, so we can separate them and IMO should do so. It >> does add a tiny bit of redundant build time. >> >> Kenn >> >> On Tue, Oct 16, 2018 at 8:36 AM Colm O hEigeartaigh <[email protected]> >> wrote: >> >>> Thanks Kenn, rookie mistake on my part :-) >>> >>> A further question if I may - "./gradlew :javaPreCommit" is failing for >>> me with: >>> >>> org.apache.beam.examples.WindowedWordCountIT > >>> testWindowedWordCountInBatchDynamicSharding FAILED >>> org.apache.beam.sdk.Pipeline$PipelineExecutionException at >>> WindowedWordCountIT.java:188 >>> >>> 4 tests completed, 4 failed >>> >>> > Task :beam-examples-java:directRunnerPreCommit FAILED >>> >>> Looking at the report I see: >>> >>> "Caused by: java.lang.RuntimeException: Unable to get application >>> default credentials. Please see >>> https://developers.google.com/accounts/docs/application-default-credentials >>> for details on how to specify credentials. This version of the SDK is >>> dependent on the gcloud core component version 2015.02.05 or newer to be >>> able to get credentials from the currently authorized user via gcloud auth." >>> >>> It looks like some of the examples require google credentials to run >>> properly? >>> >>> Colm. >>> >>> On Tue, Oct 16, 2018 at 4:07 PM Kenneth Knowles <[email protected]> wrote: >>> >>>> One thing to clarify is that `:javaPreCommit` is a task and `build` is >>>> another task. There's so verb-object relationship in your commandline. So >>>> as written, you've asked for a whole-project `build`, which weirdly in >>>> Gradle means "build and test". Since it is one commandline, all the >>>> necessary steps for both tasks will be in one dependency graph so they >>>> won't be executed twice. >>>> >>>> Kenn >>>> >>>> On Tue, Oct 16, 2018 at 6:11 AM Colm O hEigeartaigh < >>>> [email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Just a quick question - I was wondering why the python tests/build run >>>>> as part of the 'javaPreCommit' task? >>>>> >>>>> i.e. executing "./gradlew build :javaPreCommit" leads to python tests >>>>> being run as well, which is not something you might expect from the name >>>>> of >>>>> the task. >>>>> >>>>> Colm. >>>>> >>>>> >>>>> -- >>>>> Colm O hEigeartaigh >>>>> >>>>> Talend Community Coder >>>>> http://coders.talend.com >>>>> >>>> >>> >>> -- >>> Colm O hEigeartaigh >>> >>> Talend Community Coder >>> http://coders.talend.com >>> >> > > -- > > > > > Got feedback? tinyurl.com/swegner-feedback >
