2018-01-24 0:08 GMT+01:00 Lukasz Cwik <lc...@google.com>: > > > On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: > >> >> >> 2018-01-23 23:44 GMT+01:00 Kenneth Knowles <k...@google.com>: >> >>> Romain - really good point about docs getting out of date. >>> >>> > On 01/23/2018 09:29 AM, Romain Manni-Bucau wrote: >>>> >> Yep, >>>> >> >>>> >> a compromise can be to ensure all custom tasks have a description >>>> maybe: >>>> >>> >>> This is a great idea. >>> >>> Let's do both! I think the thing that the comments in the gradle code >>> cannot capture are the ways that you might combine them, like the way you >>> override properties, etc. >>> >>> On Tue, Jan 23, 2018 at 9:13 AM, Ismaël Mejía <ieme...@gmail.com> wrote: >>> >>>> I used to test kafka >>>> with a different version of the client quickly by doing this with >>>> maven: >>>> >>>> mvn verify -Prelease -Dkafka.clients.version=0.10.0.1 -pl >>>> 'sdks/java/io/kafka' >>>> >>>> but I don't have any idea of how to do this on Gradle. >>>> >>> >>> Did you figure this out. Luke - can you suggest something? >>> >>> > Some ideas: > * We can define a static set of properties like we did in maven in the > gradle.properties which allow users to override them. >
Isn't it only for the daemon start and therefore now that usable when you run it to validate you didn't break the compat with previous versions? > * We can just dynamically set versions of libraries by adding a function > to build_rules.gradle which checks to see if there is a property defined > for that library and use it. Many ways to map names from just the library > name, or library name + group automatically. This way we don't have to > maintain a large list of properties and each library can be overridden from > the command line. > +1, kind of maven -D becoming gradle -P so quite natural IMHO > * If we defined everything in the root build.gradle then you can edit the > single line related to the property version. > * Add a special property which is a list of overrides you want for library > dependencies/versions and have it applied in build_rules.gradle. > Can be needed as well for dependencies "linked" but more as a nice to have rather as a blocker IMHO > > >> Of course everybody has a different workflow, but I am pretty sure >>>> there are some common tasks that we can document for people (like me) >>>> that are new to gradle. >>> >>> >>> Yea, actually, it never even occurred to me that you would use the >>> command line to test against other Kafka versions :-). >>> >>> I have private gists with lots of maven invocations for doing things >>> like running ValidatesRunner or example ITs on just a particular runner. It >>> always requires multiple maven commands that are both multiple lines. I was >>> halfway for a while but I am now all gradle. I will start building the same >>> thing. >>> >>> >>>> - How to skip the Java/Python/Go build depending on your priorities. >>>> >>> >>> IMO they should NOT run by default. Everything should be >>> dependency-driven. When I ask to run Java SDK tests, or Java examples IT on >>> the Flink Runner, it is *incorrect* for any Python or Go builds to run. >>> >> > This is already the case. Only if you do './gradlew build' will everything > get built. Unless your specifically saying build only Go code which is > vague because we have some cross language dependencies where Go requires > java to build. Best to stick with gradles buildNeeded and buildDependents. > > >> >>> >>>> - How to run an individual IntegrationTest or ValidatesRunner test. >>>> >>> >>> Yes, #1 use case >>> >>> >>>> - How to skip findbugs,checkstyle,javadoc gneeration, etc. To have an >>>> ultra quick build. >>>> >>> >>> Again, I think they should be independent tasks. I should be able to run >>> *any* of them without running *any* of the others. It is incorrect for >>> anything to cause any other thing to run if it does not directly require >>> its outputs. >>> >>> There can be an aggregated "verify" command but I will actually very >>> rarely run that until I am done with a large chunk of work. >>> >>> As for what the aggregated "verify" command does, people kept arguing >>> about what to make default. As long as we have a correct build for >>> individual checks (aka not running extra things) then I am happy for the >>> default to be long and slow, but we should still build profiles for both. >>> >> >> Hmm, did you read it right Kenn? I think the idea was to skip all >> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >> doing it manually (gradle -x findbugs -x checkstyle etc...) >> > > Add to build_rules.gradle to not apply these plugins if the 'fast' profile > is enabled. > +1, that was what I had in mind Side note on that topic: shouldnt build_rules be split in versions and shared_plugins scripts? > > >> >> >>> >>> Kenn >>> >>> >>> >>>> >> >>>> >> diff --git a/examples/java/build.gradle b/examples/java/build.gradle >>>> >> index 0fc0b17df..001bd8b38 100644 >>>> >> --- a/examples/java/build.gradle >>>> >> +++ b/examples/java/build.gradle >>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>> >> ] >>>> >> for (String runner : preCommitRunners) { >>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>> >> + tasks.create(name: runner + "PreCommit", type: Test, description: >>>> "Run tests >>>> >> for runner ${runner.replace('Runner', '')}") { >>>> >> def preCommitBeamTestPipelineOptions = [ >>>> >> "--project=apache-beam-testing", >>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>> >> >>>> >> >>>> >> >>>> >> Romain Manni-Bucau >>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>> >> <http://rmannibucau.wordpress.com> | Github < >>>> https://github.com/rmannibucau> | >>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>> >> >>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net >>>> >> <mailto:j...@nanthrax.net>>: >>>> >> >>>> >> Hi Romain, >>>> >> >>>> >> I think we are pretty close: agree to add some explicit tasks & >>>> projects names. >>>> >> >>>> >> We can add additional tasks like skipAudit, for instance. >>>> >> >>>> >> As reminder, gradle tasks provides the list of tasks and gradle >>>> projects >>>> >> provides the list of projects/modules. >>>> >> >>>> >> Regards >>>> >> JB >>>> >> >>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>> >> > Hmm, I have to admit docs dont have my favor cause they are >>>> easily outdated and >>>> >> > hard to search but you hit a good point. Starting by renaming >>>> properly the tasks >>>> >> > and maybe writing what is done in build files - since it is >>>> code and even "api >>>> >> > for dev", it requires as much comments than the main api - can >>>> be better to start. >>>> >> > >>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... can >>>> be good while in >>>> >> > dev since these phases cost a looot for nothing while you >>>> validates your code in >>>> >> > runners modules for instance. >>>> >> > >>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <k...@google.com >>>> <mailto:k...@google.com> >>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a écrit : >>>> >> > >>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>> rmannibu...@gmail.com>>> wrote: >>>> >> > >>>> >> > @Kenneth: why not dropping the doc for a script with >>>> comments in the >>>> >> > project? A "RUNME.sh" ;). >>>> >> > >>>> >> > >>>> >> > That's cool, too, but also any shell one liner can be a >>>> gradle one >>>> >> liner or >>>> >> > mvn two/three liner :-). it is just trading one command >>>> that you cannot >>>> >> > guess easily for a different one that you still can't >>>> guess easily. >>>> >> > >>>> >> > For example, are the SparkRunner ValidatesRunner tests in >>>> the >>>> >> SparkRunner or >>>> >> > the core SDK or a third module that integrates the two? >>>> And why would you >>>> >> > know that the example ITs are called >>>> "sparkRunnerPreCommit"? It >>>> >> doesn't even >>>> >> > make sense really to have "precommit" or "postcommit" >>>> except as aliases to >>>> >> > make it easy to repro Jenkins' behavior - they have no >>>> other intrinsic >>>> >> meaning. >>>> >> > >>>> >> > So I was proposing a mapping from "full sentence + >>>> description" to one >>>> >> liner >>>> >> > to help people navigate the targets that we set up. Some >>>> web page or doc >>>> >> > that people can just quickly scan to find out to do common >>>> things, easier >>>> >> > than groovy or XML. >>>> >> > >>>> >> > Kenn >>>> >> > >>>> >> > >>>> >> >>>> >> -- >>>> >> Jean-Baptiste Onofré >>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>> >> http://blog.nanthrax.net >>>> >> Talend - http://www.talend.com >>>> >> >>>> >> >>>> > >>>> > -- >>>> > Jean-Baptiste Onofré >>>> > jbono...@apache.org >>>> > http://blog.nanthrax.net >>>> > Talend - http://www.talend.com >>>> >>> >>> >> >