There are a lot of ideas on how to increase usability, but I think they'll get lost in the thread. I suggest we try to capture them in Jiras.
I suggest we also find out what common use patterns are (people on this thread are probably sufficient), as different people will have different workflows. We can then make sure that all common workflows are documented. As an example, one task I often do is to run just checkstyle over a module or the entire project. Reuven On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > FYI, I did a new attempt and it works fine (pretty long). Previous try > failed. > > Regards > JB > > On 10/04/2018 19:52, Kenneth Knowles wrote: > > I've been on Idea+Gradle for ~two months, around the time I added > > https://github.com/apache/beam/pull/4583 and > > https://github.com/apache/beam/pull/4626 to make the import require > zero > > user work. I have no fear of deleting my project any time and > re-importing. > > > > I agree with not having auto-import on. It is just too slow. I can't > > remember if it was importing too often due to build outputs or if it was > > just that I was messing with the build.gradle files. Anyhow it doesn't > > really add much value. > > > > The gradle runner _is_ able to use submodules and run individual tests > > methods, and all that. > > > > Kenn > > > > > > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: > > > > Runner a test doesnt have the right classpath (idea uses out/ instead > > of build/) then when you switch on gradle runner the launching uses > > gradle which is not able to use submodules directly but reconsider > the > > whole project which is quite slow for normal dev iterations > > compare to just run the test with the right classpath and a fast > > compile step if needed. I lost literally 1h for something simple with > > that tooling, this is way too much to be acceptable on my side since > > I'm sadly not paid to work on beam (one day maybe ;)). > > > > Romain Manni-Bucau > > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > > > > > 2018-04-10 18:27 GMT+02:00 Reuven Lax <re...@google.com > > <mailto:re...@google.com>>: > > > Romain, > > > > > > Can you detail what's not working. I switched my IntelliJ over to > > Gradle > > > about two weeks ago, and haven't had any trouble. > > > > > > Reuven > > > > > > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > wrote: > > >> > > >> Ok, didn't find a way to make it working properly (only > workaround > > >> with direct commands and no good idea integration for > > debugging). I'm > > >> back with maven, if anyone knows how to properly solve it let's > > do it. > > >> If not I think JB point is to consider more than any other > criteria. > > >> > > >> Romain Manni-Bucau > > >> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > >> > > >> > > >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>: > > >> > side note: do NOT use auto-import until you are sure you can, > > it locks > > >> > regularly on beam (pby too big for idea?) and makes idea ready > > to be > > >> > killed :( > > >> > > > >> > Romain Manni-Bucau > > >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > >> > > > >> > > > >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré > > <j...@nanthrax.net <mailto:j...@nanthrax.net>>: > > >> >> It's what I did, I'm trying a complete reload now (maybe this > > step > > >> >> failed). > > >> >> > > >> >> On 10/04/2018 16:38, Lukasz Cwik wrote: > > >> >>> > > >> >>> beam-site PR/414 updates the instructions for using Intellij > > and how > > >> >>> to > > >> >>> import a module: > > >> >>> > > >> >>> 1. Create an empty IntelliJ project outside of the Beam > > source tree. > > >> >>> 2. Under Project Structure > Project, select a Project SDK. > > >> >>> 3. Under Project Structure > Modules, click the + sign to > > add a module > > >> >>> and > > >> >>> select "Import Module". > > >> >>> 1. Select the directory containing the Beam source tree. > > >> >>> 2. Tick the "Import module from external model" button > > and select > > >> >>> Gradle > > >> >>> from the list. > > >> >>> 3. Tick the following boxes. > > >> >>> * Use auto-import > > >> >>> * Create separate module per source set > > >> >>> * Store generated project files externally > > >> >>> * Use default gradle wrapper > > >> >>> 4. Delegate build actions to Gradle by going to Settings > > > Build, > > >> >>> Execution, > > >> >>> Deployment > Build Tools > Gradle and checking "Delegate > IDE > > >> >>> build/run > > >> >>> actions to gradle". > > >> >>> > > >> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > > <j...@nanthrax.net <mailto:j...@nanthrax.net> > > >> >>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote: > > >> >>> > > >> >>> That's a very important issue for contribution. Up to > > now, I used > > >> >>> Maven > > >> >>> for setup IntelliJ (and it works just fine). If we > > remove the > > >> >>> pom.xml, > > >> >>> we have to support Eclipse and IntelliJ "smoothly". > > >> >>> > > >> >>> Let me try in IntelliJ. > > >> >>> > > >> >>> Regards > > >> >>> JB > > >> >>> > > >> >>> On 10/04/2018 15:21, Romain Manni-Bucau wrote: > > >> >>> > You dont have issue due to the build setup with that > > option. I > > >> >>> get: > > >> >>> > > > >> >>> > avr. 10, 2018 3:20:10 PM > > >> >>> > > > org.apache.beam.runners.direct.DirectTransformExecutor run > > >> >>> > GRAVE: Error occurred within > > >> >>> > > > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a > > >> >>> > com.google.common.util.concurrent.ExecutionError: > > >> >>> > java.lang.NoClassDefFoundError: > > net/bytebuddy/NamingStrategy > > >> >>> > > > >> >>> > ? > > >> >>> > > > >> >>> > Romain Manni-Bucau > > >> >>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | > > Book > > >> >>> > > > >> >>> > > > >> >>> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik > > <lc...@google.com <mailto:lc...@google.com> > > >> >>> <mailto:lc...@google.com <mailto:lc...@google.com>>>: > > >> >>> >> I have found that the simplest setup is to delegate > the > > >> >>> build/test actions > > >> >>> >> to Gradle. This allows you to run unit tests very > > easily and > > >> >>> since its in > > >> >>> >> the same manner that Gradle would have, you know > > that if its > > >> >>> passing it will > > >> >>> >> pass on the command line and on Jenkins. Here is one > > site that > > >> >>> discusses how > > >> >>> >> to set this up: > > >> >>> >> > > >> >>> > > >> >>> > > >> >>> > > > http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html > > >> >>> >> > > >> >>> >> > > >> >>> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau > > >> >>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>> > > >> >>> >> wrote: > > >> >>> >>> > > >> >>> >>> What's the plan to make idea supporting gradle on > beam > > >> >>> project? > > >> >>> Do we > > >> >>> >>> import the workaround mentionned in > > >> >>> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? > > >> >>> >>> For the ones who didn't see this issue in action: > > idea will > > >> >>> compile in > > >> >>> >>> out/ instead of build/ and you will just miss all > the > > >> >>> resources > > >> >>> you > > >> >>> >>> need like some SPI registration which are used by > > all our > > >> >>> registrar => > > >> >>> >>> no way to run tests in idea without hacking the > > configuration > > >> >>> quite > > >> >>> >>> deeply :( > > >> >>> >>> > > >> >>> >>> Romain Manni-Bucau > > >> >>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > > | Book > > >> >>> >>> > > >> >>> >>> > > >> >>> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot > > >> >>> <echauc...@apache.org <mailto:echauc...@apache.org> > > <mailto:echauc...@apache.org <mailto:echauc...@apache.org>>>: > > >> >>> > > >> >>> >>>> As a gradle beginner, I could not agree more ! > > >> >>> >>>> +1 > > >> >>> >>>> Etienne > > >> >>> >>>> Le lundi 09 avril 2018 à 18:47 +0200, > > Jean-Baptiste Onofré a > > >> >>> écrit : > > >> >>> >>>> > > >> >>> >>>> Hi all, > > >> >>> >>>> > > >> >>> >>>> I did multiple gradle build since last week and I > > would like > > >> >>> to share > > >> >>> >>>> one of my concern: it's about the communities. > > >> >>> >>>> > > >> >>> >>>> If I think our users won't see any change for them > > due to > > >> >>> Gradle build > > >> >>> >>>> (I think that most of our users will still use > > Maven with > > >> >>> artifacts > > >> >>> >>>> provided by Gradle), I'm more concerned by the dev > > community > > >> >>> and the > > >> >>> >>>> contribution. > > >> >>> >>>> > > >> >>> >>>> Maven is well known and straight forward for a > > large part of > > >> >>> potential > > >> >>> >>>> contributors. I think we have to keep in mind that > > we still > > >> >>> have to grow > > >> >>> >>>> up our contributors community. > > >> >>> >>>> > > >> >>> >>>> Today, maybe I'm wrong, but I have the feeling > > that gradle > > >> >>> build is not > > >> >>> >>>> straight forward (build.gradle includes > > build_rules.gradle, > > >> >>> gathering > > >> >>> >>>> all taks all together). > > >> >>> >>>> > > >> >>> >>>> I would like to add a task in the gradle > "migration" > > >> >>> process: > > >> >>> simplify > > >> >>> >>>> the gradle structure and files, and document this. > > >> >>> >>>> > > >> >>> >>>> I know we already have a Jira about the > > documentation part, > > >> >>> but I would > > >> >>> >>>> like to "polish" and use a clean structure for the > > Gradle > > >> >>> resources. As > > >> >>> >>>> already quickly discussed, I think that having one > > gradle > > >> >>> file > > >> >>> per tasks > > >> >>> >>>> in the .gradle directory would be helpful. > > >> >>> >>>> > > >> >>> >>>> The goal is really to simplify the contribution. > > >> >>> >>>> > > >> >>> >>>> Do you agree if I add a Jira about "Gradle polish" > ? > > >> >>> >>>> Thoughts ? > > >> >>> >>>> > > >> >>> >>>> Regards > > >> >>> >>>> JB > > >> >>> >>>> > > >> >>> >>>> On 07/04/2018 04:52, Scott Wegner wrote: > > >> >>> >>>> > > >> >>> >>>> Here's an end-of-day update on migration work: > > >> >>> >>>> > > >> >>> >>>> * Snapshot unsigned dailies and signed release > > builds are > > >> >>> working (!!). > > >> >>> >>>> PR/5048 [1] merges changes from Luke's branch > > >> >>> >>>> * python precommit failing... will investigate > > python > > >> >>> precommit > > >> >>> >>>> Monday > > >> >>> >>>> * All Precommits are gradle only > > >> >>> >>>> * All Postcommits except performance tests and > > >> >>> Java_JDK_Versions_Test > > >> >>> >>>> use gradle (after PR/5047 [2] merged) > > >> >>> >>>> * Nightly snapshot release using gradle is ready; > > needs > > >> >>> PR/5048 to be > > >> >>> >>>> merged before switching > > >> >>> >>>> * ValidatesRunner_Spark failing consistently; > > investigating > > >> >>> >>>> > > >> >>> >>>> Thanks for another productive day of hacking. I'll > > pick up > > >> >>> again on > > >> >>> >>>> Monday. > > >> >>> >>>> > > >> >>> >>>> [1] https://github.com/apache/beam/pull/5048 > > >> >>> >>>> [2] https://github.com/apache/beam/pull/5047 > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau > > >> >>> >>>> <rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>> > > >> >>> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>>>> > > >> >>> wrote: > > >> >>> >>>> > > >> >>> >>>> Why building a zip per runner which its stack > > and just > > >> >>> pointing out > > >> >>> >>>> on that zip and let beam lazy load the runner: > > >> >>> >>>> > > >> >>> >>>> --runner=LazyRunner --lazyRunnerDir=... > > >> >>> --lazyRunnerOptions=... (or > > >> >>> >>>> the fromSystemProperties() if it gets merged > > a day ;)) > > >> >>> >>>> > > >> >>> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" > > <k...@google.com <mailto:k...@google.com> > > >> >>> <mailto:k...@google.com <mailto:k...@google.com>> > > >> >>> >>>> <mailto:k...@google.com > > <mailto:k...@google.com> <mailto:k...@google.com > > <mailto:k...@google.com>>>> a > > >> >>> écrit : > > >> >>> >>>> > > >> >>> >>>> I'm working on finding a solution for > > launching the > > >> >>> Nexmark > > >> >>> >>>> suite with each runner. This doesn't have > > to be > > >> >>> done > > >> >>> via Gradle, > > >> >>> >>>> but we anyhow need built artifacts that > don't > > >> >>> require > > >> >>> user > > >> >>> >>>> classpath intervention. > > >> >>> >>>> > > >> >>> >>>> It looks to me like the examples are also > > missing > > >> >>> this - they > > >> >>> >>>> have separate configuration e.g. > > >> >>> sparkRunnerPreCommit > > >> >>> but that > > >> >>> >>>> is overspecified compared to a free-form > > launching > > >> >>> of > > >> >>> a main() > > >> >>> >>>> program with a runner profile. > > >> >>> >>>> > > >> >>> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz > Cwik > > >> >>> <lc...@google.com <mailto:lc...@google.com> > > <mailto:lc...@google.com <mailto:lc...@google.com>> > > >> >>> >>>> <mailto:lc...@google.com > > <mailto:lc...@google.com> > > >> >>> <mailto:lc...@google.com <mailto:lc...@google.com>>>> > > >> >>> wrote: > > >> >>> >>>> > > >> >>> >>>> Romain, are you talking about the > > profiles that > > >> >>> exist as > > >> >>> >>>> part of the archetype examples? > > >> >>> >>>> > > >> >>> >>>> If so, then those still exist and > > haven't been > > >> >>> changed. If > > >> >>> >>>> not, can you provide a link to the > > profile in a > > >> >>> pom file to > > >> >>> >>>> be clearer? > > >> >>> >>>> > > >> >>> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain > > >> >>> Manni-Bucau > > >> >>> >>>> <rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> > > >> >>> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> > > >> >>> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>>>> > > >> >>> >>>> wrote: > > >> >>> >>>> > > >> >>> >>>> Hi Scott, > > >> >>> >>>> > > >> >>> >>>> is it right that 2 doesn't handle > the > > >> >>> hierachy anymore > > >> >>> >>>> and that it doesn't handle > > profiles for > > >> >>> runners as it is > > >> >>> >>>> currently with maven? > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> Romain Manni-Bucau > > >> >>> >>>> @rmannibucau > > >> >>> <https://twitter.com/rmannibucau> | Blog > > >> >>> >>>> > > <https://rmannibucau.metawerx.net/> | Old > > >> >>> Blog > > >> >>> >>>> > > <http://rmannibucau.wordpress.com> | Github > > >> >>> >>>> <https://github.com/rmannibucau> > > | LinkedIn > > >> >>> >>>> > > <https://www.linkedin.com/in/rmannibucau> | > > >> >>> Book > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> > > >> >>> > > >> >>> > > >> >>> > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > >> >>> >>>> > > >> >>> >>>> 2018-04-06 18:32 GMT+02:00 Scott > > Wegner > > >> >>> >>>> <sweg...@google.com > > <mailto:sweg...@google.com> > > >> >>> <mailto:sweg...@google.com <mailto:sweg...@google.com>> > > <mailto:sweg...@google.com <mailto:sweg...@google.com> > > >> >>> > > >> >>> <mailto:sweg...@google.com <mailto:sweg...@google.com > >>>>: > > >> >>> >>>> > > >> >>> >>>> I wanted to start a thread to > > summarize > > >> >>> the current > > >> >>> >>>> state of Gradle migration. > > We've made > > >> >>> lots of good > > >> >>> >>>> progress so far this week. > > Here's the > > >> >>> status from > > >> >>> >>>> what I can tell-- please add > > or correct > > >> >>> anything I > > >> >>> >>>> missed: > > >> >>> >>>> > > >> >>> >>>> * Release artifacts can be > > built and > > >> >>> published for > > >> >>> >>>> Snapshot and officlal > > releases [1] > > >> >>> >>>> * Gradle-generated releases > > have been > > >> >>> validated with > > >> >>> >>>> the the Apache Beam archetype > > >> >>> generation > > >> >>> quickstart; > > >> >>> >>>> still needs additional > > validation. > > >> >>> >>>> * Generated release pom files > > have > > >> >>> correct project > > >> >>> >>>> metadata [2] > > >> >>> >>>> * The python pre-commits are > now > > >> >>> working > > >> >>> in Gradle > > >> >>> >>>> [3] > > >> >>> >>>> * Ismaël has started a > > collaborative > > >> >>> doc > > >> >>> of Gradle > > >> >>> >>>> tips [4] as we all learn the > new > > >> >>> system-- > > >> >>> please add > > >> >>> >>>> your own. This will > > eventually feed > > >> >>> into > > >> >>> official > > >> >>> >>>> documentation on the website. > > >> >>> >>>> * Łukasz Gajowy is working on > > migrating > > >> >>> performance > > >> >>> >>>> testing framework [5] > > >> >>> >>>> * Daniel is working on > updating > > >> >>> documentation to > > >> >>> >>>> refer to Gradle instead of > maven > > >> >>> >>>> > > >> >>> >>>> If I missed anything, please > > add it to > > >> >>> this thread. > > >> >>> >>>> > > >> >>> >>>> The general roadmap we're > working > > >> >>> towards > > >> >>> is: > > >> >>> >>>> (a) Publish release artifacts > > with > > >> >>> Gradle > > >> >>> (SNAPSHOT > > >> >>> >>>> and signed releases) > > >> >>> >>>> (b) Postcommits migrated to > > Gradle > > >> >>> >>>> (c) Migrate documentation > > from maven to > > >> >>> Gradle > > >> >>> >>>> (d) Migrate perfkit suites to > use > > >> >>> Gradle > > >> >>> >>>> > > >> >>> >>>> For those of you that are > > hacking: > > >> >>> thanks > > >> >>> for your > > >> >>> >>>> help so far! Progress is > > being roughly > > >> >>> tracked on > > >> >>> >>>> the Kanban [6]; please make > > sure the > > >> >>> issues assigned > > >> >>> >>>> to you are up-to-date. Many > > of the > > >> >>> changes are > > >> >>> >>>> staged on lukecwik's local > > branch [7]; > > >> >>> we'll work on > > >> >>> >>>> merging them back soon. > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> [1] > > >> >>> >>>> https://github.com/lukecwik/incubator-beam/pull/7 > > >> >>> >>>> [2] > > >> >>> >>>> https://github.com/lukecwik/incubator-beam/pull/3 > > >> >>> >>>> [3] > > >> >>> https://github.com/apache/beam/pull/5032 > > >> >>> >>>> [4] > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> > > >> >>> > > >> >>> > > >> >>> > > > https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit > > >> >>> >>>> [5] > > >> >>> https://github.com/apache/beam/pull/5003 > > >> >>> >>>> [6] > > >> >>> >>>> > > >> >>> >>>> > > >> >>> > > >> >>> > > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 > > >> >>> >>>> > > >> >>> >>>> [7] > > >> >>> >>>> > > >> >>> >>>> > https://github.com/lukecwik/incubator-beam/tree/gradle > > >> >>> >>>> -- > > >> >>> >>>> > > >> >>> >>>> > > >> >>> >>>> Got feedback? > > >> >>> http://go/swegner-feedback > > >> >>> >>>> > > >> >>> >>>> > > >> >>> > > >> >> > > >
