Running Dataflow validates runner tests requires you to have permission to launch jobs on the 'apache-beam-testing' project. You'll need to override dataflowProject and dataflowTempRoot with a GCP project and GCS bucket you have access to.
On Wed, Apr 11, 2018 at 3:17 PM Daniel Oliveira <danolive...@google.com> wrote: > Alexey, are you referring to tests run with "./gradlew > :beam-runners-direct-java:needsRunnerTests"? That command works fine for me > in both versions of IDEA, but I believe the same tests fail if you run them > directly through "./gradlew test". > > However, I am having issues with a bunch of validatesRunner tests, mostly > be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not > sure if it's because of a code change or a gradle config, I'll keep looking > into it. > > On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> I got tests running rrconfiguring gradle (which was setup for another >> project but seems beam didnt like it) but latency is still "high" using >> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few >> ms with default idea test runner, would be great to solve that). >> >> I also find the integration quite fishy cause configurations are customs >> so idea is kind of forced to propose your modukes 3 times at least when you >> select the classpath (x_test being generally the working one). >> >> Also the false positive you get if you forget a cleanX is a bit annoying, >> maybe we should force a clean for test or at least when there is a --tests >> to avoid gradle to not run it cause there is no diff. >> >> So it works but dev productivity is reduced a lot and it became slow >> enough to think if you should do a contribution or not - at least for me. >> >> Le 11 avr. 2018 19:37, "Alexey Romanenko" <aromanenko....@gmail.com> a >> écrit : >> >>> I’ve managed to import a project as it’s described in documentation >>> (starting from empty project) using Idea 2018 and run unit tests >>> successfully. >>> For some reasons, tests, that use DirectRunner to run a pipeline, were >>> failed. >>> >>> WBR, >>> Alexey >>> >>> On 11 Apr 2018, at 19:01, Daniel Oliveira <danolive...@google.com> >>> wrote: >>> >>> Hi everyone, I was the one who initially wrote the PR with Idea >>> instructions <https://github.com/apache/beam-site/pull/414>. I was >>> using 2017.3 as well while writing it so all the instructions were tested >>> on that version. I'll try testing the instructions on 2018 to see if I can >>> reproduce the issues people are having. >>> >>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik <lc...@google.com> wrote: >>> >>>> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet. >>>> >>>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> Any of you using the idea 2018? the import works for me but then it is >>>>> not as smooth as it seems for you. I'm just trying to see if it is a >>>>> procedure thing or a version issue. >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>> >>>>> >>>>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles <k...@google.com>: >>>>> > The only reason I did "empty project then add a module" procedure >>>>> was to get >>>>> > all the IntelliJ files outside the source tree. IIRC directly >>>>> creating from >>>>> > existing sources didn't give the necessary configuration options. If >>>>> you >>>>> > don't care about being able to `git clean` then you can do the >>>>> shorter >>>>> > version. Also the particular UI for creation might have improved >>>>> since I >>>>> > tried it. I'll do it again. >>>>> > >>>>> > On the pom, since it is only generated for machine reading, why care >>>>> if the >>>>> > parent is inlined? I actually prefer to avoid coupling with things >>>>> that you >>>>> > have to go and look up. Using inheritance is OK for human edited poms >>>>> > (actually IMO it is still a mistake) but it doesn't change the >>>>> semantics of >>>>> > a shipped pom if they are all immutable, which they should be. It is >>>>> better >>>>> > to have all the info right there. Is there an actually effective >>>>> difference? >>>>> > >>>>> > Kenn >>>>> > >>>>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot < >>>>> echauc...@apache.org> >>>>> > wrote: >>>>> >> >>>>> >> Hi all, >>>>> >> I just tested gradle environment from a fresh source clone with this >>>>> >> procedure with just a tiny change: I used "new project from existing >>>>> >> sources" rather than create empty project and then add module. >>>>> >> >>>>> >> It works fine and junit runs from intellij also work. with gradle >>>>> we pay >>>>> >> a 2s delay (on my machine) before running the actual test for each >>>>> run. This >>>>> >> delay seems quite constant no matter the module: I have run all the >>>>> tests at >>>>> >> once in runner-core and a single test in another module with a >>>>> similar >>>>> >> delay. >>>>> >> >>>>> >> I also tested a debug session from intellij and it runs fine also. >>>>> >> >>>>> >> I'll do some more testing and keep you posted. >>>>> >> >>>>> >> I still have some concerns about the potential difficulty for new >>>>> >> contributors to have to learn gradle in addition to Beam itself >>>>> comparing to >>>>> >> maven which is more mainstream for java developers. That could be >>>>> >> discouraging for ex for part-time contributors >>>>> >> >>>>> >> Etienne >>>>> >> >>>>> >> Le mardi 10 avril 2018 à 14:38 +0000, Lukasz Cwik a écrit : >>>>> >> >>>>> >> beam-site PR/414 updates the instructions for using Intellij and >>>>> how to >>>>> >> import a module: >>>>> >> >>>>> >> 1. Create an empty IntelliJ project outside of the Beam source tree. >>>>> >> 2. Under Project Structure > Project, select a Project SDK. >>>>> >> 3. Under Project Structure > Modules, click the + sign to add a >>>>> module and >>>>> >> select "Import Module". >>>>> >> 1. Select the directory containing the Beam source tree. >>>>> >> 2. Tick the "Import module from external model" button and >>>>> select >>>>> >> Gradle >>>>> >> from the list. >>>>> >> 3. Tick the following boxes. >>>>> >> * Use auto-import >>>>> >> * Create separate module per source set >>>>> >> * Store generated project files externally >>>>> >> * Use default gradle wrapper >>>>> >> 4. Delegate build actions to Gradle by going to Settings > Build, >>>>> >> Execution, >>>>> >> Deployment > Build Tools > Gradle and checking "Delegate IDE >>>>> build/run >>>>> >> actions to gradle". >>>>> >> >>>>> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré < >>>>> j...@nanthrax.net> >>>>> >> wrote: >>>>> >> >>>>> >> That's a very important issue for contribution. Up to now, I used >>>>> Maven >>>>> >> for setup IntelliJ (and it works just fine). If we remove the >>>>> pom.xml, >>>>> >> we have to support Eclipse and IntelliJ "smoothly". >>>>> >> >>>>> >> Let me try in IntelliJ. >>>>> >> >>>>> >> Regards >>>>> >> JB >>>>> >> >>>>> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote: >>>>> >> > You dont have issue due to the build setup with that option. I >>>>> get: >>>>> >> > >>>>> >> > avr. 10, 2018 3:20:10 PM >>>>> >> > org.apache.beam.runners.direct.DirectTransformExecutor run >>>>> >> > GRAVE: Error occurred within >>>>> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a >>>>> >> > com.google.common.util.concurrent.ExecutionError: >>>>> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy >>>>> >> > >>>>> >> > ? >>>>> >> > >>>>> >> > Romain Manni-Bucau >>>>> >> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>> >> > >>>>> >> > >>>>> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik <lc...@google.com>: >>>>> >> >> I have found that the simplest setup is to delegate the >>>>> build/test >>>>> >> >> actions >>>>> >> >> to Gradle. This allows you to run unit tests very easily and >>>>> since its >>>>> >> >> in >>>>> >> >> the same manner that Gradle would have, you know that if its >>>>> passing it >>>>> >> >> will >>>>> >> >> pass on the command line and on Jenkins. Here is one site that >>>>> >> >> discusses how >>>>> >> >> to set this up: >>>>> >> >> >>>>> >> >> >>>>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html >>>>> >> >> >>>>> >> >> >>>>> >> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau >>>>> >> >> <rmannibu...@gmail.com> >>>>> >> >> wrote: >>>>> >> >>> >>>>> >> >>> What's the plan to make idea supporting gradle on beam project? >>>>> Do we >>>>> >> >>> import the workaround mentionned in >>>>> >> >>> https://youtrack.jetbrains.com/issue/IDEA-175172? >>>>> >> >>> For the ones who didn't see this issue in action: idea will >>>>> compile in >>>>> >> >>> out/ instead of build/ and you will just miss all the resources >>>>> you >>>>> >> >>> need like some SPI registration which are used by all our >>>>> registrar => >>>>> >> >>> no way to run tests in idea without hacking the configuration >>>>> quite >>>>> >> >>> deeply :( >>>>> >> >>> >>>>> >> >>> Romain Manni-Bucau >>>>> >> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot < >>>>> echauc...@apache.org>: >>>>> >> >>>> As a gradle beginner, I could not agree more ! >>>>> >> >>>> +1 >>>>> >> >>>> Etienne >>>>> >> >>>> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a >>>>> écrit : >>>>> >> >>>> >>>>> >> >>>> Hi all, >>>>> >> >>>> >>>>> >> >>>> I did multiple gradle build since last week and I would like >>>>> to share >>>>> >> >>>> one of my concern: it's about the communities. >>>>> >> >>>> >>>>> >> >>>> If I think our users won't see any change for them due to >>>>> Gradle >>>>> >> >>>> build >>>>> >> >>>> (I think that most of our users will still use Maven with >>>>> artifacts >>>>> >> >>>> provided by Gradle), I'm more concerned by the dev community >>>>> and the >>>>> >> >>>> contribution. >>>>> >> >>>> >>>>> >> >>>> Maven is well known and straight forward for a large part of >>>>> >> >>>> potential >>>>> >> >>>> contributors. I think we have to keep in mind that we still >>>>> have to >>>>> >> >>>> grow >>>>> >> >>>> up our contributors community. >>>>> >> >>>> >>>>> >> >>>> Today, maybe I'm wrong, but I have the feeling that gradle >>>>> build is >>>>> >> >>>> not >>>>> >> >>>> straight forward (build.gradle includes build_rules.gradle, >>>>> gathering >>>>> >> >>>> all taks all together). >>>>> >> >>>> >>>>> >> >>>> I would like to add a task in the gradle "migration" process: >>>>> >> >>>> simplify >>>>> >> >>>> the gradle structure and files, and document this. >>>>> >> >>>> >>>>> >> >>>> I know we already have a Jira about the documentation part, >>>>> but I >>>>> >> >>>> would >>>>> >> >>>> like to "polish" and use a clean structure for the Gradle >>>>> resources. >>>>> >> >>>> As >>>>> >> >>>> already quickly discussed, I think that having one gradle file >>>>> per >>>>> >> >>>> tasks >>>>> >> >>>> in the .gradle directory would be helpful. >>>>> >> >>>> >>>>> >> >>>> The goal is really to simplify the contribution. >>>>> >> >>>> >>>>> >> >>>> Do you agree if I add a Jira about "Gradle polish" ? >>>>> >> >>>> Thoughts ? >>>>> >> >>>> >>>>> >> >>>> Regards >>>>> >> >>>> JB >>>>> >> >>>> >>>>> >> >>>> On 07/04/2018 04:52, Scott Wegner wrote: >>>>> >> >>>> >>>>> >> >>>> Here's an end-of-day update on migration work: >>>>> >> >>>> >>>>> >> >>>> * Snapshot unsigned dailies and signed release builds are >>>>> working >>>>> >> >>>> (!!). >>>>> >> >>>> PR/5048 [1] merges changes from Luke's branch >>>>> >> >>>> * python precommit failing... will investigate python >>>>> precommit >>>>> >> >>>> Monday >>>>> >> >>>> * All Precommits are gradle only >>>>> >> >>>> * All Postcommits except performance tests and >>>>> Java_JDK_Versions_Test >>>>> >> >>>> use gradle (after PR/5047 [2] merged) >>>>> >> >>>> * Nightly snapshot release using gradle is ready; needs >>>>> PR/5048 to be >>>>> >> >>>> merged before switching >>>>> >> >>>> * ValidatesRunner_Spark failing consistently; investigating >>>>> >> >>>> >>>>> >> >>>> Thanks for another productive day of hacking. I'll pick up >>>>> again on >>>>> >> >>>> Monday. >>>>> >> >>>> >>>>> >> >>>> [1] https://github.com/apache/beam/pull/5048 >>>>> >> >>>> [2] https://github.com/apache/beam/pull/5047 >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>>>> >> >>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: >>>>> >> >>>> >>>>> >> >>>> Why building a zip per runner which its stack and just >>>>> pointing >>>>> >> >>>> out >>>>> >> >>>> on that zip and let beam lazy load the runner: >>>>> >> >>>> >>>>> >> >>>> --runner=LazyRunner --lazyRunnerDir=... >>>>> --lazyRunnerOptions=... >>>>> >> >>>> (or >>>>> >> >>>> the fromSystemProperties() if it gets merged a day ;)) >>>>> >> >>>> >>>>> >> >>>> Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com >>>>> >> >>>> <mailto:k...@google.com>> a écrit : >>>>> >> >>>> >>>>> >> >>>> I'm working on finding a solution for launching the >>>>> Nexmark >>>>> >> >>>> suite with each runner. This doesn't have to be done >>>>> via >>>>> >> >>>> Gradle, >>>>> >> >>>> but we anyhow need built artifacts that don't require >>>>> user >>>>> >> >>>> classpath intervention. >>>>> >> >>>> >>>>> >> >>>> It looks to me like the examples are also missing >>>>> this - >>>>> >> >>>> they >>>>> >> >>>> have separate configuration e.g. sparkRunnerPreCommit >>>>> but >>>>> >> >>>> that >>>>> >> >>>> is overspecified compared to a free-form launching of >>>>> a >>>>> >> >>>> main() >>>>> >> >>>> program with a runner profile. >>>>> >> >>>> >>>>> >> >>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik >>>>> >> >>>> <lc...@google.com >>>>> >> >>>> <mailto:lc...@google.com>> wrote: >>>>> >> >>>> >>>>> >> >>>> Romain, are you talking about the profiles that >>>>> exist as >>>>> >> >>>> part of the archetype examples? >>>>> >> >>>> >>>>> >> >>>> If so, then those still exist and haven't been >>>>> changed. >>>>> >> >>>> If >>>>> >> >>>> not, can you provide a link to the profile in a >>>>> pom file >>>>> >> >>>> to >>>>> >> >>>> be clearer? >>>>> >> >>>> >>>>> >> >>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>>>> >> >>>> <rmannibu...@gmail.com <mailto: >>>>> rmannibu...@gmail.com>> >>>>> >> >>>> wrote: >>>>> >> >>>> >>>>> >> >>>> Hi Scott, >>>>> >> >>>> >>>>> >> >>>> is it right that 2 doesn't handle the hierachy >>>>> >> >>>> anymore >>>>> >> >>>> and that it doesn't handle profiles for >>>>> runners as >>>>> >> >>>> it is >>>>> >> >>>> currently with maven? >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> Romain Manni-Bucau >>>>> >> >>>> @rmannibucau <https://twitter.com/rmannibucau> >>>>> | >>>>> >> >>>> Blog >>>>> >> >>>> <https://rmannibucau.metawerx.net/> | Old >>>>> Blog >>>>> >> >>>> <http://rmannibucau.wordpress.com> | Github >>>>> >> >>>> <https://github.com/rmannibucau> | LinkedIn >>>>> >> >>>> <https://www.linkedin.com/in/rmannibucau> | >>>>> Book >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> < >>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>>>> > >>>>> >> >>>> >>>>> >> >>>> 2018-04-06 18:32 GMT+02:00 Scott Wegner >>>>> >> >>>> <sweg...@google.com <mailto: >>>>> sweg...@google.com>>: >>>>> >> >>>> >>>>> >> >>>> I wanted to start a thread to summarize >>>>> the >>>>> >> >>>> current >>>>> >> >>>> state of Gradle migration. We've made >>>>> lots of >>>>> >> >>>> good >>>>> >> >>>> progress so far this week. Here's the >>>>> status >>>>> >> >>>> from >>>>> >> >>>> what I can tell-- please add or correct >>>>> anything >>>>> >> >>>> I >>>>> >> >>>> missed: >>>>> >> >>>> >>>>> >> >>>> * Release artifacts can be built and >>>>> published >>>>> >> >>>> for >>>>> >> >>>> Snapshot and officlal releases [1] >>>>> >> >>>> * Gradle-generated releases have been >>>>> validated >>>>> >> >>>> with >>>>> >> >>>> the the Apache Beam archetype generation >>>>> >> >>>> quickstart; >>>>> >> >>>> still needs additional validation. >>>>> >> >>>> * Generated release pom files have correct >>>>> >> >>>> project >>>>> >> >>>> metadata [2] >>>>> >> >>>> * The python pre-commits are now working >>>>> in >>>>> >> >>>> Gradle >>>>> >> >>>> [3] >>>>> >> >>>> * Ismaël has started a collaborative doc >>>>> of >>>>> >> >>>> Gradle >>>>> >> >>>> tips [4] as we all learn the new system-- >>>>> please >>>>> >> >>>> add >>>>> >> >>>> your own. This will eventually feed into >>>>> >> >>>> official >>>>> >> >>>> documentation on the website. >>>>> >> >>>> * Łukasz Gajowy is working on migrating >>>>> >> >>>> performance >>>>> >> >>>> testing framework [5] >>>>> >> >>>> * Daniel is working on updating >>>>> documentation to >>>>> >> >>>> refer to Gradle instead of maven >>>>> >> >>>> >>>>> >> >>>> If I missed anything, please add it to >>>>> this >>>>> >> >>>> thread. >>>>> >> >>>> >>>>> >> >>>> The general roadmap we're working towards >>>>> is: >>>>> >> >>>> (a) Publish release artifacts with Gradle >>>>> >> >>>> (SNAPSHOT >>>>> >> >>>> and signed releases) >>>>> >> >>>> (b) Postcommits migrated to Gradle >>>>> >> >>>> (c) Migrate documentation from maven to >>>>> Gradle >>>>> >> >>>> (d) Migrate perfkit suites to use Gradle >>>>> >> >>>> >>>>> >> >>>> For those of you that are hacking: thanks >>>>> for >>>>> >> >>>> your >>>>> >> >>>> help so far! Progress is being roughly >>>>> tracked >>>>> >> >>>> on >>>>> >> >>>> the Kanban [6]; please make sure the >>>>> issues >>>>> >> >>>> assigned >>>>> >> >>>> to you are up-to-date. Many of the >>>>> changes are >>>>> >> >>>> staged on lukecwik's local branch [7]; >>>>> we'll >>>>> >> >>>> work on >>>>> >> >>>> merging them back soon. >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> [1] >>>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/7 >>>>> >> >>>> [2] >>>>> >> >>>> https://github.com/lukecwik/incubator-beam/pull/3 >>>>> >> >>>> [3] >>>>> https://github.com/apache/beam/pull/5032 >>>>> >> >>>> [4] >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>>>> >> >>>> [5] >>>>> https://github.com/apache/beam/pull/5003 >>>>> >> >>>> [6] >>>>> >> >>>> >>>>> >> >>>> >>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>>>> >> >>>> >>>>> >> >>>> [7] >>>>> >> >>>> >>>>> >> >>>> https://github.com/lukecwik/incubator-beam/tree/gradle >>>>> >> >>>> -- >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> Got feedback? http://go/swegner-feedback >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>>> > >>>>> >>>> >>>
