Hi Luke,
let me take a concrete example.
I'm developing a Beam extension. Most of the time, I will ask myself:
1. How do I do the build.gradle and include my extension in Beam project
2. How do my extension is compile (the equivalent of
maven-compiler-plugin) and how to deal with dependency and scope
3. How my extension is packaged
4. How my test are executed
Of course, we can document this when we are on the happy path.
Now, imagine, dependency is not correctly resolved, rat is failing,
findbugs is complaining, etc. I have to check the "core" Beam gradle rules.
Today, I think it's not easy at all to find the corresponding tasks and
configuration.
I saw several projects storing the tasks in .gradle (like Apache Geode
for instance).
My point is that I think it's a right timing to polish/cleanup our
build. Else, I'm afraid we will never do that in the future.
Regards
JB
On 09/04/2018 19:50, Lukasz Cwik wrote:
JB, learning the build system in a project is hopefully avoided by most
users if the contribution guide can clearly explain what users need to
do. But for everyone else who wants to change a dependency version or
add a dependency it should be as simple as copy/paste (which I believe
it already is). Note that people who want to do anything more
complicated like add new jenkins job configurations for new integration
test targets needs to learn how the build system works, how maven
profiles work, how to plumb the required set of attributes from Jenkins
into the test target via the build system.
Kenn, I have to disagree with a lot in 1 and 2. For users who are
unfamiliar with Maven, we didn't setup the Maven build in such a way
that anyone unfamiliar with Maven knew what was going on or try to use
concepts unnatural to Maven in an attempt to make it seem easier. I
believe we should stick with Gradle/Groovy conventions. Users who are
not familiar with how Gradle/Groovy works will either ask the community
or look at StackOverflow and doing things like passing the project
around into functions is extremely uncommon compared to using the
current context and applying closures to it. For users who are familiar
with Gradle, the builds should be natural Gradle.
*- If you are going to refer to an identifier, it should have an
explicit point of origin (not strings smashed together in a loop)*
I assume your referring to how the examples java precommit is done. It
is the only case of it to my knowledge and extra hands to migrate it to
be like how the validates runner tests run would be useful. Filed
https://issues.apache.org/jira/browse/BEAM-4033
On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles <k...@google.com
<mailto:k...@google.com>> wrote:
Huge +1 to the idea of investing in simplification and polish of the
Gradle files before considering the migration complete.
1. build.gradle files should be as close to straight-line
configuration as possible:
- You should be able to understand a module's build easily,
locally, without knowing Groovy
- As little Groovy programming as possible, focused on providing
well-defined utilities
2. Explicit > implicit, especially for config files
- Ditto about being able to understand a module's build locally
- Minimize globally inherited config, in favor of explicitly
calling utilities
- Passing current project to a utility is better than trying to
make something the "closure"
- If you are going to refer to an identifier, it should have an
explicit point of origin (not strings smashed together in a loop)
When in doubt, a little redundancy to improve readability is worth
it in this context.
Kenn
On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré <j...@nanthrax.net
<mailto:j...@nanthrax.net>> wrote:
Hi all,
I did multiple gradle build since last week and I would like to
share
one of my concern: it's about the communities.
If I think our users won't see any change for them due to Gradle
build
(I think that most of our users will still use Maven with artifacts
provided by Gradle), I'm more concerned by the dev community and the
contribution.
Maven is well known and straight forward for a large part of
potential
contributors. I think we have to keep in mind that we still have
to grow
up our contributors community.
Today, maybe I'm wrong, but I have the feeling that gradle build
is not
straight forward (build.gradle includes build_rules.gradle,
gathering
all taks all together).
I would like to add a task in the gradle "migration" process:
simplify
the gradle structure and files, and document this.
I know we already have a Jira about the documentation part, but
I would
like to "polish" and use a clean structure for the Gradle
resources. As
already quickly discussed, I think that having one gradle file
per tasks
in the .gradle directory would be helpful.
The goal is really to simplify the contribution.
Do you agree if I add a Jira about "Gradle polish" ?
Thoughts ?
Regards
JB
On 07/04/2018 04:52, Scott Wegner wrote:
> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are
working (!!).
> PR/5048 [1] merges changes from Luke's branch
> * python precommit failing... will investigate python
precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and
Java_JDK_Versions_Test
> use gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs
PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up
again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>
<mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>
wrote:
>
> Why building a zip per runner which its stack and just
pointing out
> on that zip and let beam lazy load the runner:
>
> --runner=LazyRunner --lazyRunnerDir=...
--lazyRunnerOptions=... (or
> the fromSystemProperties() if it gets merged a day ;))
>
> Le 6 avr. 2018 20:21, "Kenneth Knowles" <k...@google.com
<mailto:k...@google.com>
> <mailto:k...@google.com <mailto:k...@google.com>>> a écrit :
>
> I'm working on finding a solution for launching the
Nexmark
> suite with each runner. This doesn't have to be done
via Gradle,
> but we anyhow need built artifacts that don't require
user
> classpath intervention.
>
> It looks to me like the examples are also missing
this - they
> have separate configuration e.g. sparkRunnerPreCommit
but that
> is overspecified compared to a free-form launching of
a main()
> program with a runner profile.
>
> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik
<lc...@google.com <mailto:lc...@google.com>
> <mailto:lc...@google.com <mailto:lc...@google.com>>>
wrote:
>
> Romain, are you talking about the profiles that
exist as
> part of the archetype examples?
>
> If so, then those still exist and haven't been
changed. If
> not, can you provide a link to the profile in a
pom file to
> be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> <rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com>>> wrote:
>
> Hi Scott,
>
> is it right that 2 doesn't handle the
hierachy anymore
> and that it doesn't handle profiles for
runners as it is
> currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau
<https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
>
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner
> <sweg...@google.com
<mailto:sweg...@google.com> <mailto:sweg...@google.com
<mailto:sweg...@google.com>>>:
>
> I wanted to start a thread to summarize
the current
> state of Gradle migration. We've made
lots of good
> progress so far this week. Here's the
status from
> what I can tell-- please add or correct
anything I
> missed:
>
> * Release artifacts can be built and
published for
> Snapshot and officlal releases [1]
> * Gradle-generated releases have been
validated with
> the the Apache Beam archetype generation
quickstart;
> still needs additional validation.
> * Generated release pom files have
correct project
> metadata [2]
> * The python pre-commits are now working
in Gradle [3]
> * Ismaël has started a collaborative doc
of Gradle
> tips [4] as we all learn the new system--
please add
> your own. This will eventually feed into
official
> documentation on the website.
> * Łukasz Gajowy is working on migrating
performance
> testing framework [5]
> * Daniel is working on updating
documentation to
> refer to Gradle instead of maven
>
> If I missed anything, please add it to
this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle
(SNAPSHOT
> and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to
Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks
for your
> help so far! Progress is being roughly
tracked on
> the Kanban [6]; please make sure the
issues assigned
> to you are up-to-date. Many of the
changes are
> staged on lukecwik's local branch [7];
we'll work on
> merging them back soon.
>
>
> [1]
https://github.com/lukecwik/incubator-beam/pull/7
> [2]
https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4]
>
https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6]
>
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>
> [7]
> https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>
>
> --
>
>
> Got feedback? http://go/swegner-feedback
