Hi Michael, no problem for the thread, that's the goal of the mailing list ;)
And yes, you got my idea about a "meta" module: easy way of building the "whole" Java SDK. The purpose is not to create a uber jar, but more to simplify the build for Java SDK developers. Do you want me to complete your PR with what I did ? Regards JB On 02/04/2019 16:49, Michael Luckey wrote: > Going to fork the BEAM-4046 discussion. And, JB, I apologise for > hijacking your thread. > > As for the original question, I understood a request for a meta project > which will enable easier handling of java projects. E.g. instead of > requiring the user to call > > ./gradlew module1:build module2:build ... moduleN.build > > a meta project with build task defined something about > > build.dependsOn module1:build > build.dependsOn module2:build > ... > build.dependsOn moduleN:build > > And other task as found usable. > > Not a project which in itself creates some uberjar, which I also believe > would rather difficult to implement. > > On Tue, Apr 2, 2019 at 5:13 AM Kenneth Knowles <[email protected] > <mailto:[email protected]>> wrote: > > Oh, yikes. It seems > like https://github.com/gradle/gradle/issues/847 indicates that the > feature to use the default names in Gradle is practically > nonfunctional. If that bug is as severe as it looks, I have to > retract my position. Like we could never have sdks/java/core and > sdks/py/core, right? > > Kenn > > On Mon, Apr 1, 2019 at 6:27 PM Michael Luckey <[email protected] > <mailto:[email protected]>> wrote: > > FWIW, hacked something as showcase for BEAM-4046 [1] > > This is miserably broken, but a > > ./gradlew projects > > or > > ./gradlew -p sdks/java build > > should work. Anything else is likely to cause issues. If u hit > stack overflow exception, it's likely caused > by https://github.com/gradle/gradle/issues/847 > > To continue here, lots of cleanup has to be done. We might also > need to rename folders etc, do better reflect semantic intentions. > > [1] https://github.com/apache/beam/pull/8194 > > On Mon, Apr 1, 2019 at 11:56 PM Kenneth Knowles <[email protected] > <mailto:[email protected]>> wrote: > > > > On Mon, Apr 1, 2019 at 2:20 PM Lukasz Cwik <[email protected] > <mailto:[email protected]>> wrote: > > > > On Mon, Apr 1, 2019 at 2:00 PM Kenneth Knowles > <[email protected] <mailto:[email protected]>> wrote: > > > As to building an aggregated "Java" project, I think > the blocker will be supporting conflicting deps. For > IOs like ElasticSearch and runners like Flink the > conflict is essential and deliberate, to support > multiple versions of other services. And that is not > even talking about transitive dep conflicts. I think > Python and Go don't have this issue simply because > they haven't tackled those problems. > > Are you talking about just a shortcut for building > (super easy to just add since we are using Gradle) > or a new artifact that you want to distribute? > > On Mon, Apr 1, 2019 at 10:01 AM Lukasz Cwik > <[email protected] <mailto:[email protected]>> wrote: > > During the gradle migration, we used to have > something like: > > include(":sdks:java:core") > include(":sdks:java:extensions:sql") > include(":sdks:python") > > Just to be super clear, this is Gradle default and > is equivalent to just leaving it blank. > > > but we discovered the Maven module names that > were used during publishing were "core" / "sql" > / ... (effectively the directory name) instead > of "beam-sdks-java-core". > > > Isn't this managed by the publication > plugin? > https://docs.gradle.org/current/userguide/publishing_maven.html#sec:identity_values_in_the_generated_pom > "overriding > the default identity values is easy: simply specify > the groupId, artifactId or version attributes when > configuring the MavenPublication." > > > During the gradle migration this wasn't that easy. The > new maven publish plugin improved a lot since then. > > > Using the default at the time also broke the > artifact names for intra project dependencies > that we generate[1]. Finally, we also ran into > an issue because we had more then one Gradle > project with the same directory name even though > they were under a different parent folder (I > think it was "core") and that was leading to > some strange build time behavior. > > > Weird. But I think the Jira should still stand as a > move towards simplifying our build and making it > more discoverable for new contributors. > > > Agree on the JIRA makes sense, just calling out that > there were other issues that this naming had caused in > the past which should be checked before we call this done. > > > Totally agree. It will be quite a large task with a lot of > boilerplate that might not be separable from technical > blockers that come up as you go through the boilerplate. > > Kenn > > > > Kenn > > > We didn't migrate to a flat project structure > where each project is a folder underneath the > root project because of the existing Maven build > rules that were being maintained in parallel and > I'm not sure if people would want to have a flat > project structure either. > > 1: > https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1055 > > On Mon, Apr 1, 2019 at 9:49 AM Michael Luckey > <[email protected] > <mailto:[email protected]>> wrote: > > Hi, > > although I did not yet manage to get deeper > involved into actual development, I think > this ability would be a useful addition. > > But I would also like to point out, that > this is kind of implicit, as soon we > get > https://issues.apache.org/jira/browse/BEAM-4046 > included. > > For instance, we would change the current > setup from > > include "beam-sdks-java-core" > project(":beam-sdks-java-core").dir = > file("sdks/java/core") > > to something like > > include(":sdks:java:core") > include(":sdks:java:extensions:sql") > include(":sdks:python") > > > With this in place a plain > > $ ./gradlew -p sdks/java build > > > would exactly do what you want. And, of > course, this will also work for > 'sdks/java/io', 'runners/' etc. Hope, you > get the point. > > Currently, we deviate from gradle default > convention and therefore have to implement > some quirks to restore default behaviour. > And I somehow dislike the structure > introduced by parent/child folders, which > will be destroyed by our current project > definitions. > > But, to be honest, although I have some > clear understanding on how to proceed here - > especially regarding the requirement to keep > the change backwards compatible - we might > decide not to switch. Because deeper > investigation might reveal issues, which I > am currently not aware of. > > Best, > > michel > > On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste > Onofré <[email protected] > <mailto:[email protected]>> wrote: > > Hi guys, > > I would like to introduce a Gradle > "meta" project for the build: > beam-sdks-java. > > The idea is to simply build all Java SDK > related resources (core, IO, ...). > > The purpose is also to be aligned with > the other SDKs which provide > beam-sdks-go and beam-sdks-python. > > Thoughts ? > > Regards > JB > -- > Jean-Baptiste Onofré > [email protected] > <mailto:[email protected]> > http://blog.nanthrax.net > Talend - http://www.talend.com > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
