I want to clarify one thing: I am not certain the requirement of ASL2 applies to example code snippets. I am also not sure if it makes a material difference to users. I _am_ sure we would need to deal with some process to use something other than ASL2, so I'd rather not.
Kenn On Tue, Jan 18, 2022 at 6:17 AM Kenneth Knowles <k...@apache.org> wrote: > Agree with Luke here. "Just git clone and go" is a big part of it. > > But also the answer to "I simply don't know what one would put in a Python > repo than, other than a bare setup.py that lists a dependency on > apache_beam" is answered by David's initial email and his repo, namely: > > - GitHub Actions configuration > - README.md > - example that already runs > - LICENSE (notably you've got it as MIT but to be part of Apache software > it needs to be ASL2) > > Kenn > > On Fri, Jan 14, 2022 at 2:34 PM Luke Cwik <lc...@google.com> wrote: > >> I think for consistency it makes sense to users to be told to checkout >> this git repo for the language of your choice and run. Some repos will have >> more/less than others when it comes to setup necessary. >> >> On Fri, Jan 14, 2022 at 2:26 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> +1 for doing this for Java, as setting up a project there is quite >>> complicated. I simply don't know what one would put in a Python repo >>> than, other than a bare setup.py that lists a dependency on >>> apache_beam. We don't have recommendations on file layout, etc. more >>> than that (though there's plenty of generic advice to be found out >>> there on the topic). I have a hunch go is similar, and javascript >>> would be as well (npm install apache-beam and your package.json file >>> gets updated). >>> >>> On Fri, Jan 14, 2022 at 2:17 PM Luke Cwik <lc...@google.com> wrote: >>> > >>> > There are several examples already within the Beam repo found in: >>> > https://github.com/apache/beam/tree/master/examples >>> > https://github.com/apache/beam/tree/master/sdks/go/examples >>> > >>> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples >>> > >>> > >>> > On Fri, Jan 14, 2022 at 11:07 AM Sachin Agarwal <sachi...@google.com> >>> wrote: >>> >> >>> >> I'd love to do something other than Wordcount just for >>> novelty/freshness but agreed with the suggestion that having an example in >>> each quickstart would be ideal. >>> >> >>> >> On Fri, Jan 14, 2022 at 11:06 AM David Huntsperger < >>> dhuntsper...@google.com> wrote: >>> >>> >>> >>> + 1 to a separate repo for each language. >>> >>> >>> >>> Would it make sense to include the Wordcount example in each repo? I >>> know that makes the repos less minimal, but we could rewrite the >>> quickstarts around these repos instead of the current Wordcount examples. >>> Or maybe we don't need to use the Wordcount example in the quickstarts... >>> >>> >>> >>> On Wed, Jan 12, 2022 at 1:54 PM David Cavazos <dcava...@google.com> >>> wrote: >>> >>>> >>> >>>> I agree with dropping the archetypes. Less maintenance is >>> preferable, and the github repos are more flexible and maintainable. >>> >>>> >>> >>>> How about we create: >>> >>>> >>> >>>> apache/beam-starter-java >>> >>>> apache/beam-starter-python >>> >>>> apache/beam-starter-go >>> >>>> >>> >>>> During our OKR planning, +Keith Malvetti would prefer having repos >>> for all languages. It makes sense for consistency as well. >>> >>>> >>> >>>> On Mon, Jan 10, 2022 at 5:14 PM Luke Cwik <lc...@google.com> wrote: >>> >>>>> >>> >>>>> As long as we have tags so that people can pull out a specific >>> version of the examples that coincides with a specific SDK version then we >>> could drop the archetypes. >>> >>>>> >>> >>>>> On Mon, Jan 10, 2022 at 4:09 PM Brian Hulette <bhule...@google.com> >>> wrote: >>> >>>>>> >>> >>>>>> > Being such minimal examples, I don't expect them to break >>> commonly, but I think it would be good to make sure tests aren't failing >>> when a release is published. >>> >>>>>> >>> >>>>>> Yeah it would be very unfortunate if we discovered a breakage >>> after the release. Agree we should verify RCs (document as part of the >>> release process), or even better, add automation to verify the repo against >>> snapshots. The automation could be nice to have anyway since it provides an >>> example for users to follow if they want to test against snapshots and >>> report issues to us sooner. >>> >>>>>> >>> >>>>>> >>> >>>>>> If we move forward with this can we drop the archetype? >>> >>>>>> >>> >>>>>> On Fri, Jan 7, 2022 at 3:54 PM Luke Cwik <lc...@google.com> >>> wrote: >>> >>>>>>> >>> >>>>>>> Sounds reasonable. >>> >>>>>>> >>> >>>>>>> On Wed, Jan 5, 2022 at 12:47 PM David Cavazos < >>> dcava...@google.com> wrote: >>> >>>>>>>> >>> >>>>>>>> I personally like the idea of a separate repo since we can see >>> how a true minimal project looks like. Having it in the main repo would >>> inherit build file configurations and other settings that would be >>> different from a clean project, so it could be non-trivial to adapt. Also >>> as its own repo, it's easier to clone and modify, or create an instance of >>> the template. >>> >>>>>>>> >>> >>>>>>>> Dependabot can take care of updating the Beam version and other >>> dependencies automatically. Testing is already set up via GitHub actions >>> for every pull request, so it would automatically be tested as soon as >>> there is a new dependency version available. >>> >>>>>>>> >>> >>>>>>>> Being such minimal examples, I don't expect them to break >>> commonly, but I think it would be good to make sure tests aren't failing >>> when a release is published. >>> >>>>>>>> >>> >>>>>>>> I'm okay with having one repo per language, and having all the >>> build systems we want to support for them. As long as we document which >>> files are for which build system. That way there are less repos to maintain. >>> >>>>>>>> >>> >>>>>>>> On Mon, Dec 13, 2021 at 9:25 AM Luke Cwik <lc...@google.com> >>> wrote: >>> >>>>>>>>> >>> >>>>>>>>> The github repo is definitely more flexible then the >>> archetypes but the archetypes have a few conveniences since they are >>> integrated with apache/beam repo. For example, updates/testing are done at >>> the same time a corresponding change to the main repo is done (like library >>> version updates), they are released when the SDK is released. >>> >>>>>>>>> >>> >>>>>>>>> Should these be part of the main repo, or a single starter >>> repo containing all the starters or one per language or one per build >>> system? >>> >>>>>>>>> >>> >>>>>>>>> When should updates to the starter happen? >>> >>>>>>>>> How as a community do we get them to happen (e.g. release >>> manager owns it)? >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Sun, Dec 12, 2021 at 4:06 PM David Cavazos < >>> dcava...@google.com> wrote: >>> >>>>>>>>>> >>> >>>>>>>>>> We could do the Maven archetype, but that wouldn't work very >>> well for Gradle and SBT users. I think a GitHub template might be the more >>> flexible option, and we could have something similar for other languages as >>> well. Having said that, we could still create a Maven archetype. If someone >>> is familiar with that process, please let me know since I'm not too >>> familiar with Maven and its ecosystem. >>> >>>>>>>>>> >>> >>>>>>>>>> @Ahmet Altay I think right now we only need to pin down the >>> name of the repo, create it, and move the code there. I was thinking either >>> `apache/beam-java-template` or `apache/beam-java-starter`. What do you >>> think? >>> >>>>>>>>>> >>> >>>>>>>>>> What would be the next steps on creating the repo? >>> >>>>>>>>>> >>> >>>>>>>>>> On Thu, Dec 9, 2021 at 11:09 AM Ahmet Altay <al...@google.com> >>> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>> This is great David. Was there any progress on this? Do you >>> need help? >>> >>>>>>>>>>> >>> >>>>>>>>>>> On Wed, Dec 1, 2021 at 3:54 PM Brian Hulette < >>> bhule...@google.com> wrote: >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> This is cool, thanks! >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> We do have a template in apache/beam already, built with >>> Maven Archetype [1]. It's what powers the Java quickstart [2]. Could we >>> de-dupe these (e.g. reference the GitHub template in the quickstart, or >>> co-locate the archetype with the GitHub template)? >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> As far as creating an Apache repo, would we put this >>> somewhere like apache/beam-java-template? I think apache repositories like >>> beam-* are allowed. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Brian >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> [1] https://maven.apache.org/archetype/index.html >>> >>>>>>>>>>>> [2] >>> https://beam.apache.org/get-started/quickstart-java/#get-the-example-code >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> On Wed, Dec 1, 2021 at 11:30 AM David Cavazos < >>> dcava...@google.com> wrote: >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> +Ahmet Altay >>> >>>>>>>>>>>>> +Valentyn Tymofieiev >>> >>>>>>>>>>>>> +Kenneth Knowles >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> Please feel free to include anyone else! >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> On Mon, Oct 25, 2021 at 11:31 AM David Cavazos < >>> dcava...@google.com> wrote: >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Hi Beam community! >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> To make it easier to create a new Beam Java project, I've >>> been working on a GitHub template containing a minimal Beam Java pipeline >>> for people to start with. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Link to the GitHub template: >>> https://github.com/davidcavazos/beam-java >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> So far, here's what the template contains: >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Minimal "Hello World" Beam pipeline >>> >>>>>>>>>>>>>> Minimal test file >>> >>>>>>>>>>>>>> Build files for Gradle, sbt, and Maven (Direct runner) >>> >>>>>>>>>>>>>> Continuous integration via GitHub actions (around 1-2 >>> minutes to run) >>> >>>>>>>>>>>>>> README with instructions on how to build, run, test, and >>> add other runners >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> It's easy to create a new GitHub repo from a template. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Next steps >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Some reviewers to make sure everyone is happy with it 🙂 >>> >>>>>>>>>>>>>> Right now it lives in my personal GitHub account, so we >>> need to create an Apache repo to host it >>> >>>>>>>>>>>>>> Update/create docs with instructions on how to create a >>> new Beam Java pipeline >>> >>
