I got tests running rrconfiguring gradle (which was setup for another project but seems beam didnt like it) but latency is still "high" using gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few ms with default idea test runner, would be great to solve that).
I also find the integration quite fishy cause configurations are customs so idea is kind of forced to propose your modukes 3 times at least when you select the classpath (x_test being generally the working one). Also the false positive you get if you forget a cleanX is a bit annoying, maybe we should force a clean for test or at least when there is a --tests to avoid gradle to not run it cause there is no diff. So it works but dev productivity is reduced a lot and it became slow enough to think if you should do a contribution or not - at least for me. Le 11 avr. 2018 19:37, "Alexey Romanenko" <aromanenko....@gmail.com> a écrit : > I’ve managed to import a project as it’s described in documentation > (starting from empty project) using Idea 2018 and run unit tests > successfully. > For some reasons, tests, that use DirectRunner to run a pipeline, were > failed. > > WBR, > Alexey > > On 11 Apr 2018, at 19:01, Daniel Oliveira <danolive...@google.com> wrote: > > Hi everyone, I was the one who initially wrote the PR with Idea > instructions <https://github.com/apache/beam-site/pull/414>. I was using > 2017.3 as well while writing it so all the instructions were tested on that > version. I'll try testing the instructions on 2018 to see if I can > reproduce the issues people are having. > > On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik <lc...@google.com> wrote: > >> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. >> >> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> Any of you using the idea 2018? the import works for me but then it is >>> not as smooth as it seems for you. I'm just trying to see if it is a >>> procedure thing or a version issue. >>> >>> Romain Manni-Bucau >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>> >>> >>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles <k...@google.com>: >>> > The only reason I did "empty project then add a module" procedure was >>> to get >>> > all the IntelliJ files outside the source tree. IIRC directly creating >>> from >>> > existing sources didn't give the necessary configuration options. If >>> you >>> > don't care about being able to `git clean` then you can do the shorter >>> > version. Also the particular UI for creation might have improved since >>> I >>> > tried it. I'll do it again. >>> > >>> > On the pom, since it is only generated for machine reading, why care >>> if the >>> > parent is inlined? I actually prefer to avoid coupling with things >>> that you >>> > have to go and look up. Using inheritance is OK for human edited poms >>> > (actually IMO it is still a mistake) but it doesn't change the >>> semantics of >>> > a shipped pom if they are all immutable, which they should be. It is >>> better >>> > to have all the info right there. Is there an actually effective >>> difference? >>> > >>> > Kenn >>> > >>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot <echauc...@apache.org >>> > >>> > wrote: >>> >> >>> >> Hi all, >>> >> I just tested gradle environment from a fresh source clone with this >>> >> procedure with just a tiny change: I used "new project from existing >>> >> sources" rather than create empty project and then add module. >>> >> >>> >> It works fine and junit runs from intellij also work. with gradle we >>> pay >>> >> a 2s delay (on my machine) before running the actual test for each >>> run. This >>> >> delay seems quite constant no matter the module: I have run all the >>> tests at >>> >> once in runner-core and a single test in another module with a >>> similar >>> >> delay. >>> >> >>> >> I also tested a debug session from intellij and it runs fine also. >>> >> >>> >> I'll do some more testing and keep you posted. >>> >> >>> >> I still have some concerns about the potential difficulty for new >>> >> contributors to have to learn gradle in addition to Beam itself >>> comparing to >>> >> maven which is more mainstream for java developers. That could be >>> >> discouraging for ex for part-time contributors >>> >> >>> >> Etienne >>> >> >>> >> Le mardi 10 avril 2018 à 14:38 +0000, Lukasz Cwik a écrit : >>> >> >>> >> beam-site PR/414 updates the instructions for using Intellij and how >>> to >>> >> import a module: >>> >> >>> >> 1. Create an empty IntelliJ project outside of the Beam source tree. >>> >> 2. Under Project Structure > Project, select a Project SDK. >>> >> 3. Under Project Structure > Modules, click the + sign to add a >>> module and >>> >> select "Import Module". >>> >> 1. Select the directory containing the Beam source tree. >>> >> 2. Tick the "Import module from external model" button and select >>> >> Gradle >>> >> from the list. >>> >> 3. Tick the following boxes. >>> >> * Use auto-import >>> >> * Create separate module per source set >>> >> * Store generated project files externally >>> >> * Use default gradle wrapper >>> >> 4. Delegate build actions to Gradle by going to Settings > Build, >>> >> Execution, >>> >> Deployment > Build Tools > Gradle and checking "Delegate IDE >>> build/run >>> >> actions to gradle". >>> >> >>> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré < >>> j...@nanthrax.net> >>> >> wrote: >>> >> >>> >> That's a very important issue for contribution. Up to now, I used >>> Maven >>> >> for setup IntelliJ (and it works just fine). If we remove the pom.xml, >>> >> we have to support Eclipse and IntelliJ "smoothly". >>> >> >>> >> Let me try in IntelliJ. >>> >> >>> >> Regards >>> >> JB >>> >> >>> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote: >>> >> > You dont have issue due to the build setup with that option. I get: >>> >> > >>> >> > avr. 10, 2018 3:20:10 PM >>> >> > org.apache.beam.runners.direct.DirectTransformExecutor run >>> >> > GRAVE: Error occurred within >>> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a >>> >> > com.google.common.util.concurrent.ExecutionError: >>> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy >>> >> > >>> >> > ? >>> >> > >>> >> > Romain Manni-Bucau >>> >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>> >> > >>> >> > >>> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik <lc...@google.com>: >>> >> >> I have found that the simplest setup is to delegate the build/test >>> >> >> actions >>> >> >> to Gradle. This allows you to run unit tests very easily and since >>> its >>> >> >> in >>> >> >> the same manner that Gradle would have, you know that if its >>> passing it >>> >> >> will >>> >> >> pass on the command line and on Jenkins. Here is one site that >>> >> >> discusses how >>> >> >> to set this up: >>> >> >> >>> >> >> http://mrhaki.blogspot.com/2016/11/gradle-goodness- >>> delegate-build-and-run.html >>> >> >> >>> >> >> >>> >> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau >>> >> >> <rmannibu...@gmail.com> >>> >> >> wrote: >>> >> >>> >>> >> >>> What's the plan to make idea supporting gradle on beam project? >>> Do we >>> >> >>> import the workaround mentionned in >>> >> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? >>> >> >>> For the ones who didn't see this issue in action: idea will >>> compile in >>> >> >>> out/ instead of build/ and you will just miss all the resources >>> you >>> >> >>> need like some SPI registration which are used by all our >>> registrar => >>> >> >>> no way to run tests in idea without hacking the configuration >>> quite >>> >> >>> deeply :( >>> >> >>> >>> >> >>> Romain Manni-Bucau >>> >> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>> >> >>> >>> >> >>> >>> >> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot <echauc...@apache.org >>> >: >>> >> >>>> As a gradle beginner, I could not agree more ! >>> >> >>>> +1 >>> >> >>>> Etienne >>> >> >>>> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a >>> écrit : >>> >> >>>> >>> >> >>>> Hi all, >>> >> >>>> >>> >> >>>> I did multiple gradle build since last week and I would like to >>> share >>> >> >>>> one of my concern: it's about the communities. >>> >> >>>> >>> >> >>>> If I think our users won't see any change for them due to Gradle >>> >> >>>> build >>> >> >>>> (I think that most of our users will still use Maven with >>> artifacts >>> >> >>>> provided by Gradle), I'm more concerned by the dev community and >>> the >>> >> >>>> contribution. >>> >> >>>> >>> >> >>>> Maven is well known and straight forward for a large part of >>> >> >>>> potential >>> >> >>>> contributors. I think we have to keep in mind that we still have >>> to >>> >> >>>> grow >>> >> >>>> up our contributors community. >>> >> >>>> >>> >> >>>> Today, maybe I'm wrong, but I have the feeling that gradle build >>> is >>> >> >>>> not >>> >> >>>> straight forward (build.gradle includes build_rules.gradle, >>> gathering >>> >> >>>> all taks all together). >>> >> >>>> >>> >> >>>> I would like to add a task in the gradle "migration" process: >>> >> >>>> simplify >>> >> >>>> the gradle structure and files, and document this. >>> >> >>>> >>> >> >>>> I know we already have a Jira about the documentation part, but I >>> >> >>>> would >>> >> >>>> like to "polish" and use a clean structure for the Gradle >>> resources. >>> >> >>>> As >>> >> >>>> already quickly discussed, I think that having one gradle file >>> per >>> >> >>>> tasks >>> >> >>>> in the .gradle directory would be helpful. >>> >> >>>> >>> >> >>>> The goal is really to simplify the contribution. >>> >> >>>> >>> >> >>>> Do you agree if I add a Jira about "Gradle polish" ? >>> >> >>>> Thoughts ? >>> >> >>>> >>> >> >>>> Regards >>> >> >>>> JB >>> >> >>>> >>> >> >>>> On 07/04/2018 04:52, Scott Wegner wrote: >>> >> >>>> >>> >> >>>> Here's an end-of-day update on migration work: >>> >> >>>> >>> >> >>>> * Snapshot unsigned dailies and signed release builds are working >>> >> >>>> (!!). >>> >> >>>> PR/5048 [1] merges changes from Luke's branch >>> >> >>>> * python precommit failing... will investigate python >>> precommit >>> >> >>>> Monday >>> >> >>>> * All Precommits are gradle only >>> >> >>>> * All Postcommits except performance tests and >>> Java_JDK_Versions_Test >>> >> >>>> use gradle (after PR/5047 [2] merged) >>> >> >>>> * Nightly snapshot release using gradle is ready; needs PR/5048 >>> to be >>> >> >>>> merged before switching >>> >> >>>> * ValidatesRunner_Spark failing consistently; investigating >>> >> >>>> >>> >> >>>> Thanks for another productive day of hacking. I'll pick up again >>> on >>> >> >>>> Monday. >>> >> >>>> >>> >> >>>> [1] https://github.com/apache/beam/pull/5048 >>> >> >>>> [2] https://github.com/apache/beam/pull/5047 >>> >> >>>> >>> >> >>>> >>> >> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>> >> >>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: >>> >> >>>> >>> >> >>>> Why building a zip per runner which its stack and just >>> pointing >>> >> >>>> out >>> >> >>>> on that zip and let beam lazy load the runner: >>> >> >>>> >>> >> >>>> --runner=LazyRunner --lazyRunnerDir=... >>> --lazyRunnerOptions=... >>> >> >>>> (or >>> >> >>>> the fromSystemProperties() if it gets merged a day ;)) >>> >> >>>> >>> >> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com >>> >> >>>> <mailto:k...@google.com>> a écrit : >>> >> >>>> >>> >> >>>> I'm working on finding a solution for launching the >>> Nexmark >>> >> >>>> suite with each runner. This doesn't have to be done via >>> >> >>>> Gradle, >>> >> >>>> but we anyhow need built artifacts that don't require >>> user >>> >> >>>> classpath intervention. >>> >> >>>> >>> >> >>>> It looks to me like the examples are also missing this - >>> >> >>>> they >>> >> >>>> have separate configuration e.g. sparkRunnerPreCommit >>> but >>> >> >>>> that >>> >> >>>> is overspecified compared to a free-form launching of a >>> >> >>>> main() >>> >> >>>> program with a runner profile. >>> >> >>>> >>> >> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik >>> >> >>>> <lc...@google.com >>> >> >>>> <mailto:lc...@google.com>> wrote: >>> >> >>>> >>> >> >>>> Romain, are you talking about the profiles that >>> exist as >>> >> >>>> part of the archetype examples? >>> >> >>>> >>> >> >>>> If so, then those still exist and haven't been >>> changed. >>> >> >>>> If >>> >> >>>> not, can you provide a link to the profile in a pom >>> file >>> >> >>>> to >>> >> >>>> be clearer? >>> >> >>>> >>> >> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>> >> >>>> <rmannibu...@gmail.com <mailto: >>> rmannibu...@gmail.com>> >>> >> >>>> wrote: >>> >> >>>> >>> >> >>>> Hi Scott, >>> >> >>>> >>> >> >>>> is it right that 2 doesn't handle the hierachy >>> >> >>>> anymore >>> >> >>>> and that it doesn't handle profiles for runners >>> as >>> >> >>>> it is >>> >> >>>> currently with maven? >>> >> >>>> >>> >> >>>> >>> >> >>>> Romain Manni-Bucau >>> >> >>>> @rmannibucau <https://twitter.com/rmannibucau> >>> | >>> >> >>>> Blog >>> >> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>> >> >>>> <http://rmannibucau.wordpress.com> | Github >>> >> >>>> <https://github.com/rmannibucau> | LinkedIn >>> >> >>>> <https://www.linkedin.com/in/rmannibucau> | >>> Book >>> >> >>>> >>> >> >>>> >>> >> >>>> >>> >> >>>> <https://www.packtpub.com/application-development/java- >>> ee-8-high-performance> >>> >> >>>> >>> >> >>>> 2018-04-06 18:32 GMT+02:00 Scott Wegner >>> >> >>>> <sweg...@google.com <mailto:sweg...@google.com >>> >>: >>> >> >>>> >>> >> >>>> I wanted to start a thread to summarize the >>> >> >>>> current >>> >> >>>> state of Gradle migration. We've made lots >>> of >>> >> >>>> good >>> >> >>>> progress so far this week. Here's the status >>> >> >>>> from >>> >> >>>> what I can tell-- please add or correct >>> anything >>> >> >>>> I >>> >> >>>> missed: >>> >> >>>> >>> >> >>>> * Release artifacts can be built and >>> published >>> >> >>>> for >>> >> >>>> Snapshot and officlal releases [1] >>> >> >>>> * Gradle-generated releases have been >>> validated >>> >> >>>> with >>> >> >>>> the the Apache Beam archetype generation >>> >> >>>> quickstart; >>> >> >>>> still needs additional validation. >>> >> >>>> * Generated release pom files have correct >>> >> >>>> project >>> >> >>>> metadata [2] >>> >> >>>> * The python pre-commits are now working in >>> >> >>>> Gradle >>> >> >>>> [3] >>> >> >>>> * Ismaël has started a collaborative doc of >>> >> >>>> Gradle >>> >> >>>> tips [4] as we all learn the new system-- >>> please >>> >> >>>> add >>> >> >>>> your own. This will eventually feed into >>> >> >>>> official >>> >> >>>> documentation on the website. >>> >> >>>> * Łukasz Gajowy is working on migrating >>> >> >>>> performance >>> >> >>>> testing framework [5] >>> >> >>>> * Daniel is working on updating >>> documentation to >>> >> >>>> refer to Gradle instead of maven >>> >> >>>> >>> >> >>>> If I missed anything, please add it to this >>> >> >>>> thread. >>> >> >>>> >>> >> >>>> The general roadmap we're working towards >>> is: >>> >> >>>> (a) Publish release artifacts with Gradle >>> >> >>>> (SNAPSHOT >>> >> >>>> and signed releases) >>> >> >>>> (b) Postcommits migrated to Gradle >>> >> >>>> (c) Migrate documentation from maven to >>> Gradle >>> >> >>>> (d) Migrate perfkit suites to use Gradle >>> >> >>>> >>> >> >>>> For those of you that are hacking: thanks >>> for >>> >> >>>> your >>> >> >>>> help so far! Progress is being roughly >>> tracked >>> >> >>>> on >>> >> >>>> the Kanban [6]; please make sure the issues >>> >> >>>> assigned >>> >> >>>> to you are up-to-date. Many of the changes >>> are >>> >> >>>> staged on lukecwik's local branch [7]; we'll >>> >> >>>> work on >>> >> >>>> merging them back soon. >>> >> >>>> >>> >> >>>> >>> >> >>>> [1] >>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/7 >>> >> >>>> [2] >>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/3 >>> >> >>>> [3] https://github.com/apache/ >>> beam/pull/5032 >>> >> >>>> [4] >>> >> >>>> >>> >> >>>> >>> >> >>>> >>> >> >>>> https://docs.google.com/document/d/ >>> 1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>> >> >>>> [5] https://github.com/apache/ >>> beam/pull/5003 >>> >> >>>> [6] >>> >> >>>> >>> >> >>>> https://issues.apache.org/jira/secure/RapidBoard.jspa? >>> rapidView=242 >>> >> >>>> >>> >> >>>> [7] >>> >> >>>> >>> >> >>>> https://github.com/lukecwik/incubator-beam/tree/gradle >>> >> >>>> -- >>> >> >>>> >>> >> >>>> >>> >> >>>> Got feedback? http://go/swegner-feedback >>> >> >>>> >>> >> >>>> >>> >> >>> > >>> >> >
