I like the goal: for things where the build has extra setup, have an example that is fully functional on its own. There is of course the problem of "where does it end?" since this is infinity things.
The other piece is that a user wanting to know some of these bits may be past the "clone and go" stage of their project. They probably already have a project and now they need a working example to read and learn from. So it could be just one additional repo `beam-working-examples` where each subdirectory is an independent working setup. I do like having it a separate repo to avoid the temptation to leverage anything from the Beam build. And each subdirectory should be entirely independent and we also have to avoid the temptation to share configuration across them, or it would defeat the purpose. Kenn On Tue, Feb 15, 2022 at 9:28 PM Reza Ardeshir Rokni <raro...@gmail.com> wrote: > Hi, > > This is great! > > What do folks think about also having a less minimal set of starters? For > Java I am thinking about protobuf / autovalue. For Python maybe an > opinionated setup with tox etc... Again this would just contain 'hello' > world samples to get folks going. > > Regards > Reza > > On Wed, 9 Feb 2022 at 13:56, Robert Burke <r...@google.com> wrote: > >> SGTM. >> >> On Wed, Feb 9, 2022 at 1:09 PM Kenneth Knowles <k...@apache.org> wrote: >> >>> Based on discussion on https://issues.apache.org/jira/browse/LEGAL-601 >>> I think it will be simplest to license it under ASL2 and include a NOTICE >>> file. The user will be free to "clone and go". >>> >>> I would bring these points back to the dev list: >>> >>> - ASL2 is what people expect from an ASF project, so it is "least >>> surprise" >>> - Dual-licensing is possible (but I think not worthwhile due to its >>> impact on contributor license agreements) >>> - ASL2 says "You must cause any modified files to carry prominent >>> notices stating that You changed the files" which won't apply to the user's >>> code and I would guess they simply won't bother with for files in the >>> template. Or maybe there is a clever way to phrase the header so it is >>> already good to go. >>> - ASL2 says if the work includes a NOTICE file, you have to includes >>> the attributions from it. The NOTICE file is required by ASF policy. We can >>> easily set it up to be a noop for the user. >>> >>> So my overall take is that we should go ahead with ASL2 and a simple >>> NOTICE file. Check the Jira for details. >>> >>> Kenn >>> >>> On Mon, Feb 7, 2022 at 10:47 AM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> And I've created the repos just now. >>>> >>>> Kenn >>>> >>>> On Mon, Feb 7, 2022 at 10:39 AM Kenneth Knowles <k...@apache.org> >>>> wrote: >>>> >>>>> Legal question asked at >>>>> https://issues.apache.org/jira/browse/LEGAL-601 >>>>> >>>>> Kenn >>>>> >>>>> On Fri, Feb 4, 2022 at 7:58 AM Danny McCormick < >>>>> dannymccorm...@google.com> wrote: >>>>> >>>>>> Sure - I'm happy to help out with the Actions setup (and/or with the >>>>>> Go template). I will say though, the Actions config should be pretty darn >>>>>> simple for these examples - >>>>>> https://github.com/davidcavazos/beam-java/blob/main/.github/workflows/test.yaml >>>>>> seems right, for each language configuration we're targeting we basically >>>>>> just want a job with: >>>>>> >>>>>> - checkout >>>>>> - setup-<language> >>>>>> - inlined script to run tests >>>>>> >>>>>> Always happy to help with or consult on any actions issues 🙂 >>>>>> >>>>>> Thanks, >>>>>> Danny >>>>>> >>>>>> On Fri, Feb 4, 2022 at 10:21 AM Kerry Donny-Clark <kerr...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Danny has extensive experience with GitHub actions, and may be able >>>>>>> to help out. >>>>>>> Kerry >>>>>>> >>>>>>> On Thu, Feb 3, 2022, 11:47 PM Kenneth Knowles <k...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm convinced on all points. My main motivation was to keep it >>>>>>>> simple. But of course we should keep it simple for users, not us :-) >>>>>>>> >>>>>>>> I can take on the task of asking about MIT license and requesting >>>>>>>> the repos be created. Not sure if it needs my level of privileges but >>>>>>>> I'm >>>>>>>> happy to do it anyhow. >>>>>>>> >>>>>>>> Kenn >>>>>>>> >>>>>>>> On Wed, Feb 2, 2022 at 10:30 AM Robert Bradshaw < >>>>>>>> rober...@google.com> wrote: >>>>>>>> >>>>>>>>> On Wed, Feb 2, 2022 at 10:12 AM David Cavazos <dcava...@google.com> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > MIT is much more permissive, but I also don't have any problems >>>>>>>>> changing it to Apache license. In any case, how about we create the >>>>>>>>> following repos? >>>>>>>>> >>>>>>>>> For these starter projects, we don't want to encumber any users of >>>>>>>>> these templates with any particular licensing requirements (right?) >>>>>>>>> and we don't even care about attribution. We want these to be >>>>>>>>> pretty >>>>>>>>> much as close to public domain as possible. That's not what the >>>>>>>>> Apache >>>>>>>>> licence does. (If it's even relevant, a good argument could likely >>>>>>>>> be >>>>>>>>> made for de minis or fair use, but I think it's best to be explicit >>>>>>>>> about this. Perhaps this'd be a good question for apache legal? >>>>>>>>> >>>>>>>>> > apache/beam-starter-java >>>>>>>>> > apache/beam-starter-python >>>>>>>>> > apache/beam-starter-go >>>>>>>>> > apache/beam-starter-kotlin >>>>>>>>> > apache/beam-starter-scala >>>>>>>>> > >>>>>>>>> > We'll start by populating the Java one which is the most >>>>>>>>> pressing one and the one that is ready, but the rest should be >>>>>>>>> simpler. >>>>>>>>> > >>>>>>>>> > +David Huntsperger, tldr; these are minimal starter projects for >>>>>>>>> every language. Once we have Java, Python and Go, it might be a good >>>>>>>>> idea >>>>>>>>> to change the quickstarts to use these instead of the word count. >>>>>>>>> There is >>>>>>>>> already a dedicated word count walkthrough so I think that is already >>>>>>>>> covered. >>>>>>>>> > >>>>>>>>> > If we all agree on the repo names, who can help us create them? >>>>>>>>> > >>>>>>>>> > On Thu, Jan 27, 2022 at 12:58 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >> >>>>>>>>> >> On Tue, Jan 18, 2022 at 6:17 AM Kenneth Knowles < >>>>>>>>> k...@apache.org> wrote: >>>>>>>>> >> > >>>>>>>>> >> > Agree with Luke here. "Just git clone and go" is a big part >>>>>>>>> of it. >>>>>>>>> >> > >>>>>>>>> >> > But also the answer to "I simply don't know what one would >>>>>>>>> put in a Python repo than, other than a bare setup.py that lists a >>>>>>>>> dependency on apache_beam" is answered by David's initial email and >>>>>>>>> his >>>>>>>>> repo, namely: >>>>>>>>> >> > >>>>>>>>> >> > - GitHub Actions configuration >>>>>>>>> >> > - README.md >>>>>>>>> >> > - example that already runs >>>>>>>>> >> >>>>>>>>> >> OK, fair enough. >>>>>>>>> >> >>>>>>>>> >> > - LICENSE (notably you've got it as MIT but to be part of >>>>>>>>> Apache software it needs to be ASL2) >>>>>>>>> >> >>>>>>>>> >> On the topic of licence, it's a bit tricky because one doesn't >>>>>>>>> want to >>>>>>>>> >> bind the users of such a template as being a derivative work of >>>>>>>>> a >>>>>>>>> >> too-restrictive licence. The licence of the template itself >>>>>>>>> should >>>>>>>>> >> generally be very permissive. >>>>>>>>> >> >>>>>>>>> >> > On Fri, Jan 14, 2022 at 2:34 PM Luke Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >> >> >>>>>>>>> >> >> I think for consistency it makes sense to users to be told >>>>>>>>> to checkout this git repo for the language of your choice and run. >>>>>>>>> Some >>>>>>>>> repos will have more/less than others when it comes to setup >>>>>>>>> necessary. >>>>>>>>> >> >> >>>>>>>>> >> >> On Fri, Jan 14, 2022 at 2:26 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>> >> >>> +1 for doing this for Java, as setting up a project there >>>>>>>>> is quite >>>>>>>>> >> >>> complicated. I simply don't know what one would put in a >>>>>>>>> Python repo >>>>>>>>> >> >>> than, other than a bare setup.py that lists a dependency on >>>>>>>>> >> >>> apache_beam. We don't have recommendations on file layout, >>>>>>>>> etc. more >>>>>>>>> >> >>> than that (though there's plenty of generic advice to be >>>>>>>>> found out >>>>>>>>> >> >>> there on the topic). I have a hunch go is similar, and >>>>>>>>> javascript >>>>>>>>> >> >>> would be as well (npm install apache-beam and your >>>>>>>>> package.json file >>>>>>>>> >> >>> gets updated). >>>>>>>>> >> >>> >>>>>>>>> >> >>> On Fri, Jan 14, 2022 at 2:17 PM Luke Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > There are several examples already within the Beam repo >>>>>>>>> found in: >>>>>>>>> >> >>> > https://github.com/apache/beam/tree/master/examples >>>>>>>>> >> >>> > >>>>>>>>> https://github.com/apache/beam/tree/master/sdks/go/examples >>>>>>>>> >> >>> > >>>>>>>>> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > On Fri, Jan 14, 2022 at 11:07 AM Sachin Agarwal < >>>>>>>>> sachi...@google.com> wrote: >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> I'd love to do something other than Wordcount just for >>>>>>>>> novelty/freshness but agreed with the suggestion that having an >>>>>>>>> example in >>>>>>>>> each quickstart would be ideal. >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> On Fri, Jan 14, 2022 at 11:06 AM David Huntsperger < >>>>>>>>> dhuntsper...@google.com> wrote: >>>>>>>>> >> >>> >>> >>>>>>>>> >> >>> >>> + 1 to a separate repo for each language. >>>>>>>>> >> >>> >>> >>>>>>>>> >> >>> >>> Would it make sense to include the Wordcount example in >>>>>>>>> each repo? I know that makes the repos less minimal, but we could >>>>>>>>> rewrite >>>>>>>>> the quickstarts around these repos instead of the current Wordcount >>>>>>>>> examples. Or maybe we don't need to use the Wordcount example in the >>>>>>>>> quickstarts... >>>>>>>>> >> >>> >>> >>>>>>>>> >> >>> >>> On Wed, Jan 12, 2022 at 1:54 PM David Cavazos < >>>>>>>>> dcava...@google.com> wrote: >>>>>>>>> >> >>> >>>> >>>>>>>>> >> >>> >>>> I agree with dropping the archetypes. Less maintenance >>>>>>>>> is preferable, and the github repos are more flexible and >>>>>>>>> maintainable. >>>>>>>>> >> >>> >>>> >>>>>>>>> >> >>> >>>> How about we create: >>>>>>>>> >> >>> >>>> >>>>>>>>> >> >>> >>>> apache/beam-starter-java >>>>>>>>> >> >>> >>>> apache/beam-starter-python >>>>>>>>> >> >>> >>>> apache/beam-starter-go >>>>>>>>> >> >>> >>>> >>>>>>>>> >> >>> >>>> During our OKR planning, +Keith Malvetti would prefer >>>>>>>>> having repos for all languages. It makes sense for consistency as >>>>>>>>> well. >>>>>>>>> >> >>> >>>> >>>>>>>>> >> >>> >>>> On Mon, Jan 10, 2022 at 5:14 PM Luke Cwik < >>>>>>>>> lc...@google.com> wrote: >>>>>>>>> >> >>> >>>>> >>>>>>>>> >> >>> >>>>> As long as we have tags so that people can pull out a >>>>>>>>> specific version of the examples that coincides with a specific SDK >>>>>>>>> version >>>>>>>>> then we could drop the archetypes. >>>>>>>>> >> >>> >>>>> >>>>>>>>> >> >>> >>>>> On Mon, Jan 10, 2022 at 4:09 PM Brian Hulette < >>>>>>>>> bhule...@google.com> wrote: >>>>>>>>> >> >>> >>>>>> >>>>>>>>> >> >>> >>>>>> > Being such minimal examples, I don't expect them >>>>>>>>> to break commonly, but I think it would be good to make sure tests >>>>>>>>> aren't >>>>>>>>> failing when a release is published. >>>>>>>>> >> >>> >>>>>> >>>>>>>>> >> >>> >>>>>> Yeah it would be very unfortunate if we discovered a >>>>>>>>> breakage after the release. Agree we should verify RCs (document as >>>>>>>>> part of >>>>>>>>> the release process), or even better, add automation to verify the >>>>>>>>> repo >>>>>>>>> against snapshots. The automation could be nice to have anyway since >>>>>>>>> it >>>>>>>>> provides an example for users to follow if they want to test against >>>>>>>>> snapshots and report issues to us sooner. >>>>>>>>> >> >>> >>>>>> >>>>>>>>> >> >>> >>>>>> >>>>>>>>> >> >>> >>>>>> If we move forward with this can we drop the >>>>>>>>> archetype? >>>>>>>>> >> >>> >>>>>> >>>>>>>>> >> >>> >>>>>> On Fri, Jan 7, 2022 at 3:54 PM Luke Cwik < >>>>>>>>> lc...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>> >>>>>>>>> >> >>> >>>>>>> Sounds reasonable. >>>>>>>>> >> >>> >>>>>>> >>>>>>>>> >> >>> >>>>>>> On Wed, Jan 5, 2022 at 12:47 PM David Cavazos < >>>>>>>>> dcava...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>> >>>>>>>>> >> >>> >>>>>>>> I personally like the idea of a separate repo >>>>>>>>> since we can see how a true minimal project looks like. Having it in >>>>>>>>> the >>>>>>>>> main repo would inherit build file configurations and other settings >>>>>>>>> that >>>>>>>>> would be different from a clean project, so it could be non-trivial to >>>>>>>>> adapt. Also as its own repo, it's easier to clone and modify, or >>>>>>>>> create an >>>>>>>>> instance of the template. >>>>>>>>> >> >>> >>>>>>>> >>>>>>>>> >> >>> >>>>>>>> Dependabot can take care of updating the Beam >>>>>>>>> version and other dependencies automatically. Testing is already set >>>>>>>>> up via >>>>>>>>> GitHub actions for every pull request, so it would automatically be >>>>>>>>> tested >>>>>>>>> as soon as there is a new dependency version available. >>>>>>>>> >> >>> >>>>>>>> >>>>>>>>> >> >>> >>>>>>>> Being such minimal examples, I don't expect them >>>>>>>>> to break commonly, but I think it would be good to make sure tests >>>>>>>>> aren't >>>>>>>>> failing when a release is published. >>>>>>>>> >> >>> >>>>>>>> >>>>>>>>> >> >>> >>>>>>>> I'm okay with having one repo per language, and >>>>>>>>> having all the build systems we want to support for them. As long as >>>>>>>>> we >>>>>>>>> document which files are for which build system. That way there are >>>>>>>>> less >>>>>>>>> repos to maintain. >>>>>>>>> >> >>> >>>>>>>> >>>>>>>>> >> >>> >>>>>>>> On Mon, Dec 13, 2021 at 9:25 AM Luke Cwik < >>>>>>>>> lc...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>> The github repo is definitely more flexible then >>>>>>>>> the archetypes but the archetypes have a few conveniences since they >>>>>>>>> are >>>>>>>>> integrated with apache/beam repo. For example, updates/testing are >>>>>>>>> done at >>>>>>>>> the same time a corresponding change to the main repo is done (like >>>>>>>>> library >>>>>>>>> version updates), they are released when the SDK is released. >>>>>>>>> >> >>> >>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>> Should these be part of the main repo, or a >>>>>>>>> single starter repo containing all the starters or one per language >>>>>>>>> or one >>>>>>>>> per build system? >>>>>>>>> >> >>> >>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>> When should updates to the starter happen? >>>>>>>>> >> >>> >>>>>>>>> How as a community do we get them to happen (e.g. >>>>>>>>> release manager owns it)? >>>>>>>>> >> >>> >>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>> On Sun, Dec 12, 2021 at 4:06 PM David Cavazos < >>>>>>>>> dcava...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>> We could do the Maven archetype, but that >>>>>>>>> wouldn't work very well for Gradle and SBT users. I think a GitHub >>>>>>>>> template >>>>>>>>> might be the more flexible option, and we could have something >>>>>>>>> similar for >>>>>>>>> other languages as well. Having said that, we could still create a >>>>>>>>> Maven >>>>>>>>> archetype. If someone is familiar with that process, please let me >>>>>>>>> know >>>>>>>>> since I'm not too familiar with Maven and its ecosystem. >>>>>>>>> >> >>> >>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>> @Ahmet Altay I think right now we only need to >>>>>>>>> pin down the name of the repo, create it, and move the code there. I >>>>>>>>> was >>>>>>>>> thinking either `apache/beam-java-template` or >>>>>>>>> `apache/beam-java-starter`. >>>>>>>>> What do you think? >>>>>>>>> >> >>> >>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>> What would be the next steps on creating the >>>>>>>>> repo? >>>>>>>>> >> >>> >>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>> On Thu, Dec 9, 2021 at 11:09 AM Ahmet Altay < >>>>>>>>> al...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>> This is great David. Was there any progress on >>>>>>>>> this? Do you need help? >>>>>>>>> >> >>> >>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>> On Wed, Dec 1, 2021 at 3:54 PM Brian Hulette < >>>>>>>>> bhule...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> This is cool, thanks! >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> We do have a template in apache/beam already, >>>>>>>>> built with Maven Archetype [1]. It's what powers the Java quickstart >>>>>>>>> [2]. >>>>>>>>> Could we de-dupe these (e.g. reference the GitHub template in the >>>>>>>>> quickstart, or co-locate the archetype with the GitHub template)? >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> As far as creating an Apache repo, would we >>>>>>>>> put this somewhere like apache/beam-java-template? I think apache >>>>>>>>> repositories like beam-* are allowed. >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> Brian >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> [1] >>>>>>>>> https://maven.apache.org/archetype/index.html >>>>>>>>> >> >>> >>>>>>>>>>>> [2] >>>>>>>>> https://beam.apache.org/get-started/quickstart-java/#get-the-example-code >>>>>>>>> >> >>> >>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>> On Wed, Dec 1, 2021 at 11:30 AM David Cavazos < >>>>>>>>> dcava...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>> +Ahmet Altay >>>>>>>>> >> >>> >>>>>>>>>>>>> +Valentyn Tymofieiev >>>>>>>>> >> >>> >>>>>>>>>>>>> +Kenneth Knowles >>>>>>>>> >> >>> >>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>> Please feel free to include anyone else! >>>>>>>>> >> >>> >>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>> On Mon, Oct 25, 2021 at 11:31 AM David >>>>>>>>> Cavazos <dcava...@google.com> wrote: >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> Hi Beam community! >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> To make it easier to create a new Beam Java >>>>>>>>> project, I've been working on a GitHub template containing a minimal >>>>>>>>> Beam >>>>>>>>> Java pipeline for people to start with. >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> Link to the GitHub template: >>>>>>>>> https://github.com/davidcavazos/beam-java >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> So far, here's what the template contains: >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> Minimal "Hello World" Beam pipeline >>>>>>>>> >> >>> >>>>>>>>>>>>>> Minimal test file >>>>>>>>> >> >>> >>>>>>>>>>>>>> Build files for Gradle, sbt, and Maven >>>>>>>>> (Direct runner) >>>>>>>>> >> >>> >>>>>>>>>>>>>> Continuous integration via GitHub actions >>>>>>>>> (around 1-2 minutes to run) >>>>>>>>> >> >>> >>>>>>>>>>>>>> README with instructions on how to build, >>>>>>>>> run, test, and add other runners >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> It's easy to create a new GitHub repo from a >>>>>>>>> template. >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> Next steps >>>>>>>>> >> >>> >>>>>>>>>>>>>> >>>>>>>>> >> >>> >>>>>>>>>>>>>> Some reviewers to make sure everyone is >>>>>>>>> happy with it 🙂 >>>>>>>>> >> >>> >>>>>>>>>>>>>> Right now it lives in my personal GitHub >>>>>>>>> account, so we need to create an Apache repo to host it >>>>>>>>> >> >>> >>>>>>>>>>>>>> Update/create docs with instructions on how >>>>>>>>> to create a new Beam Java pipeline >>>>>>>>> >>>>>>>>
