3 for all reasons above. Keeping the examples in the Beam repository is a straightforward way to let the examples being visible and maintainable. And as Lukasz Cwik mentioned, the Quickstart makes a clear and easy approach for users to generate the archetype and run examples. I did not see enough benefits to make tons of efforts splitting an example repository. It make sense to keep consistent.
- Yifan On Mon, Aug 13, 2018 at 10:21 AM Andrew Pilloud <apill...@google.com> wrote: > +1 for 2, it would be nice to have a example project in it's own GitHub > repo. Might not need to be an "official" repo thought. Could we provide > links to community supplied examples? > > On Thu, Aug 9, 2018, 2:30 PM Robert Bradshaw <rober...@google.com> wrote: > >> (3) >> >> In particular, I see a lot of value for (quoting the proposal) >> >> """ >> Since then, there have been >> numerous updates, increased Python parity, and new features that do >> not have accompanying examples employing best practices and >> demonstrating an end-to-end experience for new users. We would like to >> leverage the existing examples by raising their visibility and >> auditing them. >> """ >> >> and I think the situation would become *worse* on all these fronts >> with a separate repo (as well as the other issues mentioned, >> especially complexity). We should consider lowering the bar to liking >> to user-maintained examples that don't merit being in the main repo, >> as well as guidelines for adding examples in the main repo itself. >> On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía <ieme...@gmail.com> wrote: >> > >> > 3 for all the reasons discussed above. I think there are better ways to >> improve the status quo without the extra maintenance of having a new repo >> for this. >> > >> > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <al...@google.com> wrote: >> >> >> >> If we go forward with (3), could we actually update our documentation >> on how we will support casual example contributions? I think we will need >> to have information on how to add links to the new examples people want to >> add to the set, what examples would be good additions to the Beam repo and >> what examples would be better maintained somewhere else by their owners, >> and what could they expect from our community when they work on such >> examples. >> >> >> >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <mig...@google.com> >> wrote: >> >>> >> >>> 3 (if contributors are up for voting) - We want to have beam >> maintained examples in main repo. This will give good man to users and >> allow us to test those easily with minimal maintenance. >> >>> >> >>> We can add links to opensource user repositories to our >> documentation/wiki. This will be flexible enough to provide external >> examples on one hand, and avoid responsibility of maintaining user code on >> the other hand. >> >>> >> >>> --Mikhail >> >>> >> >>> Have feedback? >> >>> >> >>> >> >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <rfern...@google.com> >> wrote: >> >>>> >> >>>> Here is the Rose', David's, and Gris' proposal in text form, I hope >> >>>> the copy/paste helps: >> >>>> >> >>>> >> >>>> Apache Beam Examples Repository >> >>>> >> >>>> Authors: Rose Nguyen (rtngu...@google.com), David Cavazos >> >>>> (dcava...@google.com), Gris Cuevas (g...@apache.org) >> >>>> >> >>>> Status: Proposal >> >>>> Created: 2018-07-30 >> >>>> Updated: 2018-07-30 >> >>>> >> >>>> Summary >> >>>> >> >>>> The Apache Beam Community creates and contributes examples to the >> core >> >>>> Apache Beam Github repository. We want to make the process easier and >> >>>> less dependent in the core repository by creating a separate repo, >> >>>> dedicated solely to Community examples, contribution guidelines and >> >>>> add the examples to the website. >> >>>> >> >>>> Background >> >>>> >> >>>> The original batch of examples on the Apache Beam GitHub repository >> >>>> was donated by Cloud Dataflow at the time of Java SDK 1.x to >> >>>> demonstrate the capability of this programming model. These initial >> >>>> examples were intended to demonstrate how a user can put together >> >>>> their code components and try out Beam. Since then, there have been >> >>>> numerous updates, increased Python parity, and new features that do >> >>>> not have accompanying examples employing best practices and >> >>>> demonstrating an end-to-end experience for new users. We would like >> to >> >>>> leverage the existing examples by raising their visibility and >> >>>> auditing them. This is also an opportunity to establish >> >>>> contribution/maintenance guidelines for community contributions and >> to >> >>>> start hosting the examples on the Beam site in an official >> repository. >> >>>> Attracting and retaining new users necessitates updated, concrete >> >>>> examples that exhibit the range of capabilities of Beam. >> >>>> >> >>>> Proposed Tasks >> >>>> >> >>>> We would like to create a new GitHub Repository under the Apache >> >>>> Software Foundation Org page for Apache Beam Community Examples. This >> >>>> repo would be similar to apache/beam-site. The name we’d like to have >> >>>> is apache/beam-examples. We will also move all current examples to >> >>>> this repo, perform an audit to outline best practices and guidelines >> >>>> and then publish them in the Apache Beam website. >> >>>> >> >>>> >> >>>> Here is an outlined list of tasks we propose: >> >>>> >> >>>> Send Apache Beam Example Repository proposal to the mailing list >> >>>> (David) - July 31 >> >>>> >> >>>> Create the GitHub Repo (PMC would need to do this) - Request help >> >>>> after proposal is refined/accepted >> >>>> >> >>>> Move current examples to new repo (David) -- 2 weeks after item 2 is >> completed >> >>>> >> >>>> Add a note to let people we need to audit for best practices >> >>>> >> >>>> Audit current examples and define best practices (David, Rose, Gris) >> >>>> -- Target date: week of 8/20 >> >>>> >> >>>> Write guidelines on adding new examples and maintaining them (Gris, >> >>>> Rose) -- Week after audit is completed >> >>>> >> >>>> Add examples to website (Rose) -- 1 week after guidelines are written >> >>>> >> >>>> Publish guidelines in website (Rose) -- 1 week after guidelines are >> written >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy < >> lukasz.gaj...@gmail.com> wrote: >> >>>> > >> >>>> > I'd also vote for 3: I don't see much added value in separating >> the repos and I see much additional effort to be done in maintaining extra >> repo(s) (updating examples when new version of beam sdk comes out) and >> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn >> archetypes and how easy the examples can be to get starter examples from a >> common repo only strengthens my opinion. >> >>>> > >> >>>> > Regarding 2: I think it's not good to have some official examples >> here and some there - IMO it can make a false impression (user experience) >> that some examples are less important than the others. Maybe a good idea is >> to encourage users to share their (independent, non official) examples and >> create a list of such on the beam site instead of 2? >> >>>> > >> >>>> > Łukasz >> >>>> > >> >>>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko < >> aromanenko....@gmail.com> napisał(a): >> >>>> >> >> >>>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying >> why we need to have examples and main code in the same repository (+ >> website code base will move there soon). I don’t see any huge benefits to >> have examples aside and, at the same time, it will bring additional >> complexity and burden for project support. >> >>>> >> >> >>>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>>> >> >> >>>> >> Hi guys, >> >>>> >> >> >>>> >> For this kind of discussion, I would prefer to avoid Google Doc >> and >> >>>> >> directly put the point/proposal on the mailing list. >> >>>> >> >> >>>> >> It's easier for the community to follow. >> >>>> >> >> >>>> >> The statement is more for 3 because it's more convenient for >> users to >> >>>> >> easily find the examples and include in the distribution. >> >>>> >> >> >>>> >> Regards >> >>>> >> JB >> >>>> >> >> >>>> >> On 08/08/2018 23:25, Charles Chen wrote: >> >>>> >> >> >>>> >> It looks like the main claim is that 1 and 2 have the benefit of >> >>>> >> increasing visibility for examples on the Beam site. I agree with >> >>>> >> Robert's comments on the doc which claim that this is orthogonal >> to >> >>>> >> whether a separate repository is created (the comments are >> unresolved: >> >>>> >> >> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY >> ). >> >>>> >> >> >>>> >> I would add that the maintenance and testing burden has not been >> >>>> >> adequately addressed in the proposal (i.e. are we creating new >> Jenkins >> >>>> >> jobs?; will postcommits on the main Beam repo run examples >> tests?; are >> >>>> >> we releasing artifacts--if so, is this together with the main >> package or >> >>>> >> separately in new packages?). If we go with the half-way >> solution in >> >>>> >> (2), there is also the issue of where the threshold is--for >> example, if >> >>>> >> a user-contributed example is particularly useful, do we move it >> to the >> >>>> >> main repo? >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <g...@google.com >> >>>> >> <mailto:g...@google.com>> wrote: >> >>>> >> >> >>>> >> I'd vote for 2. >> >>>> >> >> >>>> >> Giving independence to an example repository and creating the >> right >> >>>> >> infrastructure to maintain them will give visibility to the >> efforts >> >>>> >> our users are creating to solve their uses cases with Beam. I >> also >> >>>> >> want to make the process of sharing common work more easily. >> >>>> >> >> >>>> >> Re:The examples that will remain in core, I agree that it's >> crucial >> >>>> >> to keep some examples for testing. >> >>>> >> >> >>>> >> >> >>>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <lc...@google.com >> >>>> >> <mailto:lc...@google.com>> wrote: >> >>>> >> >> >>>> >> I would vote for 3. >> >>>> >> >> >>>> >> My reasoning is that Java has a good mechanism to get a >> >>>> >> starter/example project going by using the the maven >> archetypes >> >>>> >> already. Our quickstart guide for Apache Beam for the Java >> SDK >> >>>> >> already covers generating the examples archetype. >> >>>> >> We could point users to the starter project at the end of >> the >> >>>> >> java quickstart. >> >>>> >> >> >>>> >> If python/go have a similar mechanism that is commonly >> used, I >> >>>> >> would go with those over creating a separate repo for >> examples >> >>>> >> and adding the maintenance burden involved. >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang < >> ruw...@google.com >> >>>> >> <mailto:ruw...@google.com>> wrote: >> >>>> >> >> >>>> >> 2 - examples that rely on experimental API can still >> stay in >> >>>> >> where they are because such examples could be changed. >> >>>> >> >> >>>> >> -Rui >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen < >> c...@google.com >> >>>> >> <mailto:c...@google.com>> wrote: >> >>>> >> >> >>>> >> 3 - We benefit from increased test coverage by >> having >> >>>> >> examples together with the rest of the code. As >> Robert >> >>>> >> mentions in the doc, hosting the Beam examples in >> the >> >>>> >> main repository is the best way to keep the >> examples >> >>>> >> visible, tested and maintained. Given that we >> recently >> >>>> >> moved to a single repository for the website since >> that >> >>>> >> previously caused a lot of pain, it makes sense to >> be >> >>>> >> consistent here. >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay >> >>>> >> <al...@google.com <mailto:al...@google.com>> >> wrote: >> >>>> >> >> >>>> >> 2 - Similar to Huygaa, I see value in keeping >> a core >> >>>> >> set of examples tested and maintained against >> head. >> >>>> >> At the same time I understand the value of a >> growing >> >>>> >> set of community grown examples that are >> targeted >> >>>> >> against a pre-defined versions of Beam and not >> >>>> >> necessarily updated at every release. >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa >> Batsaikhan >> >>>> >> <bat...@google.com <mailto:bat...@google.com>> >> wrote: >> >>>> >> >> >>>> >> 2 - I like the idea of having a separate >> repo >> >>>> >> where we can have more freedom to check in >> >>>> >> examples. However, we benefit from having >> >>>> >> immediate core examples in Beam for testing >> >>>> >> purposes. >> >>>> >> >> >>>> >> On Wed, Aug 8, 2018 at 9:38 AM David >> Cavazos >> >>>> >> <dcava...@google.com >> >>>> >> <mailto:dcava...@google.com>> wrote: >> >>>> >> >> >>>> >> Hi everyone! >> >>>> >> >> >>>> >> We discussed several options as well >> as some >> >>>> >> of the implications of each option. >> Please >> >>>> >> vote for your favorite option, feel >> free to >> >>>> >> back it up with any reasons that make >> you >> >>>> >> feel that way. >> >>>> >> >> >>>> >> 1) Move *all* samples to a *new >> >>>> >> *examples*repository* >> >>>> >> 2) Move *some* samples to a *new >> >>>> >> *examples*repository* >> >>>> >> 3) Leave samples where they are >> >>>> >> >> >>>> >> Some implications to creating a new >> repository: >> >>>> >> - Every example would be independent >> from >> >>>> >> every other example, so tests can be >> run in >> >>>> >> parallel >> >>>> >> - Examples would now show how to use >> Beam >> >>>> >> /externally/ >> >>>> >> - The examples repository would need a >> >>>> >> testing infrastructure >> >>>> >> - Decoupling makes examples easier to >> test >> >>>> >> on different versions >> >>>> >> - Easier to copy-paste an existing >> example >> >>>> >> and start from there, almost like a >> template >> >>>> >> - Smaller size for the core Beam >> library >> >>>> >> - Two different repositories to >> maintain >> >>>> >> - Versioning could mirror Beam's >> current version >> >>>> >> >> >>>> >> Link to proposal >> >>>> >> < >> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing >> > >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> -- >> >>>> >> Jean-Baptiste Onofré >> >>>> >> jbono...@apache.org >> >>>> >> http://blog.nanthrax.net >> >>>> >> Talend - http://www.talend.com >> >>>> >> >> >>>> >> >> >> >> >> >> >
