Going to fork the BEAM-4046 discussion. And, JB, I apologise for hijacking
your thread.
As for the original question, I understood a request for a meta project
which will enable easier handling of java projects. E.g. instead of
requiring the user to call
./gradlew module1:build module2:build ... moduleN.build
a meta project with build task defined something about
build.dependsOn module1:build
build.dependsOn module2:build
...
build.dependsOn moduleN:build
And other task as found usable.
Not a project which in itself creates some uberjar, which I also believe
would rather difficult to implement.
On Tue, Apr 2, 2019 at 5:13 AM Kenneth Knowles <[email protected]> wrote:
> Oh, yikes. It seems like https://github.com/gradle/gradle/issues/847 indicates
> that the feature to use the default names in Gradle is practically
> nonfunctional. If that bug is as severe as it looks, I have to retract my
> position. Like we could never have sdks/java/core and sdks/py/core, right?
>
> Kenn
>
> On Mon, Apr 1, 2019 at 6:27 PM Michael Luckey <[email protected]> wrote:
>
>> FWIW, hacked something as showcase for BEAM-4046 [1]
>>
>> This is miserably broken, but a
>>
>> ./gradlew projects
>>
>> or
>>
>> ./gradlew -p sdks/java build
>>
>> should work. Anything else is likely to cause issues. If u hit stack
>> overflow exception, it's likely caused by
>> https://github.com/gradle/gradle/issues/847
>>
>> To continue here, lots of cleanup has to be done. We might also need to
>> rename folders etc, do better reflect semantic intentions.
>>
>> [1] https://github.com/apache/beam/pull/8194
>>
>> On Mon, Apr 1, 2019 at 11:56 PM Kenneth Knowles <[email protected]> wrote:
>>
>>>
>>>
>>> On Mon, Apr 1, 2019 at 2:20 PM Lukasz Cwik <[email protected]> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Apr 1, 2019 at 2:00 PM Kenneth Knowles <[email protected]> wrote:
>>>>
>>>>>
>>>>> As to building an aggregated "Java" project, I think the blocker will
>>>>> be supporting conflicting deps. For IOs like ElasticSearch and runners
>>>>> like
>>>>> Flink the conflict is essential and deliberate, to support multiple
>>>>> versions of other services. And that is not even talking about transitive
>>>>> dep conflicts. I think Python and Go don't have this issue simply because
>>>>> they haven't tackled those problems.
>>>>>
>>>>> Are you talking about just a shortcut for building (super easy to just
>>>>> add since we are using Gradle) or a new artifact that you want to
>>>>> distribute?
>>>>>
>>>>> On Mon, Apr 1, 2019 at 10:01 AM Lukasz Cwik <[email protected]> wrote:
>>>>>
>>>>>> During the gradle migration, we used to have something like:
>>>>>>
>>>>>> include(":sdks:java:core")
>>>>>> include(":sdks:java:extensions:sql")
>>>>>> include(":sdks:python")
>>>>>>
>>>>>> Just to be super clear, this is Gradle default and is equivalent to
>>>>> just leaving it blank.
>>>>>
>>>>>
>>>>>> but we discovered the Maven module names that were used during
>>>>>> publishing were "core" / "sql" / ... (effectively the directory name)
>>>>>> instead of "beam-sdks-java-core".
>>>>>>
>>>>>
>>>>> Isn't this managed by the publication plugin?
>>>>> https://docs.gradle.org/current/userguide/publishing_maven.html#sec:identity_values_in_the_generated_pom
>>>>> "overriding
>>>>> the default identity values is easy: simply specify the groupId,
>>>>> artifactId
>>>>> or version attributes when configuring the MavenPublication."
>>>>>
>>>>
>>>> During the gradle migration this wasn't that easy. The new maven
>>>> publish plugin improved a lot since then.
>>>>
>>>>
>>>>> Using the default at the time also broke the artifact names for intra
>>>>>> project dependencies that we generate[1]. Finally, we also ran into an
>>>>>> issue because we had more then one Gradle project with the same directory
>>>>>> name even though they were under a different parent folder (I think it
>>>>>> was
>>>>>> "core") and that was leading to some strange build time behavior.
>>>>>>
>>>>>
>>>>> Weird. But I think the Jira should still stand as a move towards
>>>>> simplifying our build and making it more discoverable for new
>>>>> contributors.
>>>>>
>>>>
>>>> Agree on the JIRA makes sense, just calling out that there were other
>>>> issues that this naming had caused in the past which should be checked
>>>> before we call this done.
>>>>
>>>
>>> Totally agree. It will be quite a large task with a lot of boilerplate
>>> that might not be separable from technical blockers that come up as you go
>>> through the boilerplate.
>>>
>>> Kenn
>>>
>>>
>>>
>>> Kenn
>>>>>
>>>>>
>>>>>> We didn't migrate to a flat project structure where each project is a
>>>>>> folder underneath the root project because of the existing Maven build
>>>>>> rules that were being maintained in parallel and I'm not sure if people
>>>>>> would want to have a flat project structure either.
>>>>>>
>>>>>> 1:
>>>>>> https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1055
>>>>>>
>>>>>> On Mon, Apr 1, 2019 at 9:49 AM Michael Luckey <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> although I did not yet manage to get deeper involved into actual
>>>>>>> development, I think this ability would be a useful addition.
>>>>>>>
>>>>>>> But I would also like to point out, that this is kind of implicit,
>>>>>>> as soon we get https://issues.apache.org/jira/browse/BEAM-4046
>>>>>>> included.
>>>>>>>
>>>>>>> For instance, we would change the current setup from
>>>>>>>
>>>>>>> include "beam-sdks-java-core"
>>>>>>> project(":beam-sdks-java-core").dir = file("sdks/java/core")
>>>>>>>
>>>>>>> to something like
>>>>>>>
>>>>>>> include(":sdks:java:core")
>>>>>>> include(":sdks:java:extensions:sql")
>>>>>>> include(":sdks:python")
>>>>>>>
>>>>>>>
>>>>>>> With this in place a plain
>>>>>>>
>>>>>>> $ ./gradlew -p sdks/java build
>>>>>>>
>>>>>>> would exactly do what you want. And, of course, this will also work
>>>>>>> for 'sdks/java/io', 'runners/' etc. Hope, you get the point.
>>>>>>>
>>>>>>> Currently, we deviate from gradle default convention and therefore
>>>>>>> have to implement some quirks to restore default behaviour. And I
>>>>>>> somehow
>>>>>>> dislike the structure introduced by parent/child folders, which will be
>>>>>>> destroyed by our current project definitions.
>>>>>>>
>>>>>>> But, to be honest, although I have some clear understanding on how
>>>>>>> to proceed here - especially regarding the requirement to keep the
>>>>>>> change
>>>>>>> backwards compatible - we might decide not to switch. Because deeper
>>>>>>> investigation might reveal issues, which I am currently not aware of.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> michel
>>>>>>>
>>>>>>> On Mon, Apr 1, 2019 at 5:52 PM Jean-Baptiste Onofré <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>> I would like to introduce a Gradle "meta" project for the build:
>>>>>>>> beam-sdks-java.
>>>>>>>>
>>>>>>>> The idea is to simply build all Java SDK related resources (core,
>>>>>>>> IO, ...).
>>>>>>>>
>>>>>>>> The purpose is also to be aligned with the other SDKs which provide
>>>>>>>> beam-sdks-go and beam-sdks-python.
>>>>>>>>
>>>>>>>> Thoughts ?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>> --
>>>>>>>> Jean-Baptiste Onofré
>>>>>>>> [email protected]
>>>>>>>> http://blog.nanthrax.net
>>>>>>>> Talend - http://www.talend.com
>>>>>>>>
>>>>>>>
