Hi everyone, I was the one who initially wrote the PR with Idea instructions <https://github.com/apache/beam-site/pull/414>. I was using 2017.3 as well while writing it so all the instructions were tested on that version. I'll try testing the instructions on 2018 to see if I can reproduce the issues people are having.
On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik <lc...@google.com> wrote: > I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. > > On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Any of you using the idea 2018? the import works for me but then it is >> not as smooth as it seems for you. I'm just trying to see if it is a >> procedure thing or a version issue. >> >> Romain Manni-Bucau >> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >> >> >> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles <k...@google.com>: >> > The only reason I did "empty project then add a module" procedure was >> to get >> > all the IntelliJ files outside the source tree. IIRC directly creating >> from >> > existing sources didn't give the necessary configuration options. If you >> > don't care about being able to `git clean` then you can do the shorter >> > version. Also the particular UI for creation might have improved since I >> > tried it. I'll do it again. >> > >> > On the pom, since it is only generated for machine reading, why care if >> the >> > parent is inlined? I actually prefer to avoid coupling with things that >> you >> > have to go and look up. Using inheritance is OK for human edited poms >> > (actually IMO it is still a mistake) but it doesn't change the >> semantics of >> > a shipped pom if they are all immutable, which they should be. It is >> better >> > to have all the info right there. Is there an actually effective >> difference? >> > >> > Kenn >> > >> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot <echauc...@apache.org> >> > wrote: >> >> >> >> Hi all, >> >> I just tested gradle environment from a fresh source clone with this >> >> procedure with just a tiny change: I used "new project from existing >> >> sources" rather than create empty project and then add module. >> >> >> >> It works fine and junit runs from intellij also work. with gradle we >> pay >> >> a 2s delay (on my machine) before running the actual test for each >> run. This >> >> delay seems quite constant no matter the module: I have run all the >> tests at >> >> once in runner-core and a single test in another module with a similar >> >> delay. >> >> >> >> I also tested a debug session from intellij and it runs fine also. >> >> >> >> I'll do some more testing and keep you posted. >> >> >> >> I still have some concerns about the potential difficulty for new >> >> contributors to have to learn gradle in addition to Beam itself >> comparing to >> >> maven which is more mainstream for java developers. That could be >> >> discouraging for ex for part-time contributors >> >> >> >> Etienne >> >> >> >> Le mardi 10 avril 2018 à 14:38 +0000, Lukasz Cwik a écrit : >> >> >> >> beam-site PR/414 updates the instructions for using Intellij and how to >> >> import a module: >> >> >> >> 1. Create an empty IntelliJ project outside of the Beam source tree. >> >> 2. Under Project Structure > Project, select a Project SDK. >> >> 3. Under Project Structure > Modules, click the + sign to add a module >> and >> >> select "Import Module". >> >> 1. Select the directory containing the Beam source tree. >> >> 2. Tick the "Import module from external model" button and select >> >> Gradle >> >> from the list. >> >> 3. Tick the following boxes. >> >> * Use auto-import >> >> * Create separate module per source set >> >> * Store generated project files externally >> >> * Use default gradle wrapper >> >> 4. Delegate build actions to Gradle by going to Settings > Build, >> >> Execution, >> >> Deployment > Build Tools > Gradle and checking "Delegate IDE >> build/run >> >> actions to gradle". >> >> >> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré <j...@nanthrax.net >> > >> >> wrote: >> >> >> >> That's a very important issue for contribution. Up to now, I used Maven >> >> for setup IntelliJ (and it works just fine). If we remove the pom.xml, >> >> we have to support Eclipse and IntelliJ "smoothly". >> >> >> >> Let me try in IntelliJ. >> >> >> >> Regards >> >> JB >> >> >> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote: >> >> > You dont have issue due to the build setup with that option. I get: >> >> > >> >> > avr. 10, 2018 3:20:10 PM >> >> > org.apache.beam.runners.direct.DirectTransformExecutor run >> >> > GRAVE: Error occurred within >> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a >> >> > com.google.common.util.concurrent.ExecutionError: >> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy >> >> > >> >> > ? >> >> > >> >> > Romain Manni-Bucau >> >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >> >> > >> >> > >> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik <lc...@google.com>: >> >> >> I have found that the simplest setup is to delegate the build/test >> >> >> actions >> >> >> to Gradle. This allows you to run unit tests very easily and since >> its >> >> >> in >> >> >> the same manner that Gradle would have, you know that if its >> passing it >> >> >> will >> >> >> pass on the command line and on Jenkins. Here is one site that >> >> >> discusses how >> >> >> to set this up: >> >> >> >> >> >> >> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html >> >> >> >> >> >> >> >> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau >> >> >> <rmannibu...@gmail.com> >> >> >> wrote: >> >> >>> >> >> >>> What's the plan to make idea supporting gradle on beam project? Do >> we >> >> >>> import the workaround mentionned in >> >> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? >> >> >>> For the ones who didn't see this issue in action: idea will >> compile in >> >> >>> out/ instead of build/ and you will just miss all the resources you >> >> >>> need like some SPI registration which are used by all our >> registrar => >> >> >>> no way to run tests in idea without hacking the configuration quite >> >> >>> deeply :( >> >> >>> >> >> >>> Romain Manni-Bucau >> >> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >> >> >>> >> >> >>> >> >> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot <echauc...@apache.org >> >: >> >> >>>> As a gradle beginner, I could not agree more ! >> >> >>>> +1 >> >> >>>> Etienne >> >> >>>> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a >> écrit : >> >> >>>> >> >> >>>> Hi all, >> >> >>>> >> >> >>>> I did multiple gradle build since last week and I would like to >> share >> >> >>>> one of my concern: it's about the communities. >> >> >>>> >> >> >>>> If I think our users won't see any change for them due to Gradle >> >> >>>> build >> >> >>>> (I think that most of our users will still use Maven with >> artifacts >> >> >>>> provided by Gradle), I'm more concerned by the dev community and >> the >> >> >>>> contribution. >> >> >>>> >> >> >>>> Maven is well known and straight forward for a large part of >> >> >>>> potential >> >> >>>> contributors. I think we have to keep in mind that we still have >> to >> >> >>>> grow >> >> >>>> up our contributors community. >> >> >>>> >> >> >>>> Today, maybe I'm wrong, but I have the feeling that gradle build >> is >> >> >>>> not >> >> >>>> straight forward (build.gradle includes build_rules.gradle, >> gathering >> >> >>>> all taks all together). >> >> >>>> >> >> >>>> I would like to add a task in the gradle "migration" process: >> >> >>>> simplify >> >> >>>> the gradle structure and files, and document this. >> >> >>>> >> >> >>>> I know we already have a Jira about the documentation part, but I >> >> >>>> would >> >> >>>> like to "polish" and use a clean structure for the Gradle >> resources. >> >> >>>> As >> >> >>>> already quickly discussed, I think that having one gradle file per >> >> >>>> tasks >> >> >>>> in the .gradle directory would be helpful. >> >> >>>> >> >> >>>> The goal is really to simplify the contribution. >> >> >>>> >> >> >>>> Do you agree if I add a Jira about "Gradle polish" ? >> >> >>>> Thoughts ? >> >> >>>> >> >> >>>> Regards >> >> >>>> JB >> >> >>>> >> >> >>>> On 07/04/2018 04:52, Scott Wegner wrote: >> >> >>>> >> >> >>>> Here's an end-of-day update on migration work: >> >> >>>> >> >> >>>> * Snapshot unsigned dailies and signed release builds are working >> >> >>>> (!!). >> >> >>>> PR/5048 [1] merges changes from Luke's branch >> >> >>>> * python precommit failing... will investigate python >> precommit >> >> >>>> Monday >> >> >>>> * All Precommits are gradle only >> >> >>>> * All Postcommits except performance tests and >> Java_JDK_Versions_Test >> >> >>>> use gradle (after PR/5047 [2] merged) >> >> >>>> * Nightly snapshot release using gradle is ready; needs PR/5048 >> to be >> >> >>>> merged before switching >> >> >>>> * ValidatesRunner_Spark failing consistently; investigating >> >> >>>> >> >> >>>> Thanks for another productive day of hacking. I'll pick up again >> on >> >> >>>> Monday. >> >> >>>> >> >> >>>> [1] https://github.com/apache/beam/pull/5048 >> >> >>>> [2] https://github.com/apache/beam/pull/5047 >> >> >>>> >> >> >>>> >> >> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >> >> >>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: >> >> >>>> >> >> >>>> Why building a zip per runner which its stack and just >> pointing >> >> >>>> out >> >> >>>> on that zip and let beam lazy load the runner: >> >> >>>> >> >> >>>> --runner=LazyRunner --lazyRunnerDir=... >> --lazyRunnerOptions=... >> >> >>>> (or >> >> >>>> the fromSystemProperties() if it gets merged a day ;)) >> >> >>>> >> >> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com >> >> >>>> <mailto:k...@google.com>> a écrit : >> >> >>>> >> >> >>>> I'm working on finding a solution for launching the >> Nexmark >> >> >>>> suite with each runner. This doesn't have to be done via >> >> >>>> Gradle, >> >> >>>> but we anyhow need built artifacts that don't require >> user >> >> >>>> classpath intervention. >> >> >>>> >> >> >>>> It looks to me like the examples are also missing this - >> >> >>>> they >> >> >>>> have separate configuration e.g. sparkRunnerPreCommit but >> >> >>>> that >> >> >>>> is overspecified compared to a free-form launching of a >> >> >>>> main() >> >> >>>> program with a runner profile. >> >> >>>> >> >> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik >> >> >>>> <lc...@google.com >> >> >>>> <mailto:lc...@google.com>> wrote: >> >> >>>> >> >> >>>> Romain, are you talking about the profiles that >> exist as >> >> >>>> part of the archetype examples? >> >> >>>> >> >> >>>> If so, then those still exist and haven't been >> changed. >> >> >>>> If >> >> >>>> not, can you provide a link to the profile in a pom >> file >> >> >>>> to >> >> >>>> be clearer? >> >> >>>> >> >> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >> >> >>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com >> >> >> >> >>>> wrote: >> >> >>>> >> >> >>>> Hi Scott, >> >> >>>> >> >> >>>> is it right that 2 doesn't handle the hierachy >> >> >>>> anymore >> >> >>>> and that it doesn't handle profiles for runners >> as >> >> >>>> it is >> >> >>>> currently with maven? >> >> >>>> >> >> >>>> >> >> >>>> Romain Manni-Bucau >> >> >>>> @rmannibucau <https://twitter.com/rmannibucau> | >> >> >>>> Blog >> >> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >> >> >>>> <http://rmannibucau.wordpress.com> | Github >> >> >>>> <https://github.com/rmannibucau> | LinkedIn >> >> >>>> <https://www.linkedin.com/in/rmannibucau> | Book >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> < >> https://www.packtpub.com/application-development/java-ee-8-high-performance >> > >> >> >>>> >> >> >>>> 2018-04-06 18:32 GMT+02:00 Scott Wegner >> >> >>>> <sweg...@google.com <mailto:sweg...@google.com >> >>: >> >> >>>> >> >> >>>> I wanted to start a thread to summarize the >> >> >>>> current >> >> >>>> state of Gradle migration. We've made lots of >> >> >>>> good >> >> >>>> progress so far this week. Here's the status >> >> >>>> from >> >> >>>> what I can tell-- please add or correct >> anything >> >> >>>> I >> >> >>>> missed: >> >> >>>> >> >> >>>> * Release artifacts can be built and >> published >> >> >>>> for >> >> >>>> Snapshot and officlal releases [1] >> >> >>>> * Gradle-generated releases have been >> validated >> >> >>>> with >> >> >>>> the the Apache Beam archetype generation >> >> >>>> quickstart; >> >> >>>> still needs additional validation. >> >> >>>> * Generated release pom files have correct >> >> >>>> project >> >> >>>> metadata [2] >> >> >>>> * The python pre-commits are now working in >> >> >>>> Gradle >> >> >>>> [3] >> >> >>>> * Ismaël has started a collaborative doc of >> >> >>>> Gradle >> >> >>>> tips [4] as we all learn the new system-- >> please >> >> >>>> add >> >> >>>> your own. This will eventually feed into >> >> >>>> official >> >> >>>> documentation on the website. >> >> >>>> * Łukasz Gajowy is working on migrating >> >> >>>> performance >> >> >>>> testing framework [5] >> >> >>>> * Daniel is working on updating >> documentation to >> >> >>>> refer to Gradle instead of maven >> >> >>>> >> >> >>>> If I missed anything, please add it to this >> >> >>>> thread. >> >> >>>> >> >> >>>> The general roadmap we're working towards is: >> >> >>>> (a) Publish release artifacts with Gradle >> >> >>>> (SNAPSHOT >> >> >>>> and signed releases) >> >> >>>> (b) Postcommits migrated to Gradle >> >> >>>> (c) Migrate documentation from maven to >> Gradle >> >> >>>> (d) Migrate perfkit suites to use Gradle >> >> >>>> >> >> >>>> For those of you that are hacking: thanks for >> >> >>>> your >> >> >>>> help so far! Progress is being roughly >> tracked >> >> >>>> on >> >> >>>> the Kanban [6]; please make sure the issues >> >> >>>> assigned >> >> >>>> to you are up-to-date. Many of the changes >> are >> >> >>>> staged on lukecwik's local branch [7]; we'll >> >> >>>> work on >> >> >>>> merging them back soon. >> >> >>>> >> >> >>>> >> >> >>>> [1] >> >> >>>> https://github.com/lukecwik/incubator-beam/pull/7 >> >> >>>> [2] >> >> >>>> https://github.com/lukecwik/incubator-beam/pull/3 >> >> >>>> [3] https://github.com/apache/beam/pull/5032 >> >> >>>> [4] >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >> >> >>>> [5] https://github.com/apache/beam/pull/5003 >> >> >>>> [6] >> >> >>>> >> >> >>>> >> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >> >> >>>> >> >> >>>> [7] >> >> >>>> >> >> >>>> https://github.com/lukecwik/incubator-beam/tree/gradle >> >> >>>> -- >> >> >>>> >> >> >>>> >> >> >>>> Got feedback? http://go/swegner-feedback >> >> >>>> >> >> >>>> >> >> >> > >> >
