3 for all the reasons discussed above. I think there are better ways to improve the status quo without the extra maintenance of having a new repo for this.
On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay <[email protected]> wrote: > If we go forward with (3), could we actually update our documentation on > how we will support casual example contributions? I think we will need to > have information on how to add links to the new examples people want to add > to the set, what examples would be good additions to the Beam repo and what > examples would be better maintained somewhere else by their owners, and > what could they expect from our community when they work on such examples. > > On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin <[email protected]> > wrote: > >> 3 (if contributors are up for voting) - We want to have beam maintained >> examples in main repo. This will give good man to users and allow us to >> test those easily with minimal maintenance. >> >> We can add links to opensource user repositories to our >> documentation/wiki. This will be flexible enough to provide external >> examples on one hand, and avoid responsibility of maintaining user code on >> the other hand. >> >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> >> >> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez <[email protected]> >> wrote: >> >>> Here is the Rose', David's, and Gris' proposal in text form, I hope >>> the copy/paste helps: >>> >>> >>> Apache Beam Examples Repository >>> >>> Authors: Rose Nguyen ([email protected]), David Cavazos >>> ([email protected]), Gris Cuevas ([email protected]) >>> >>> Status: Proposal >>> Created: 2018-07-30 >>> Updated: 2018-07-30 >>> >>> Summary >>> >>> The Apache Beam Community creates and contributes examples to the core >>> Apache Beam Github repository. We want to make the process easier and >>> less dependent in the core repository by creating a separate repo, >>> dedicated solely to Community examples, contribution guidelines and >>> add the examples to the website. >>> >>> Background >>> >>> The original batch of examples on the Apache Beam GitHub repository >>> was donated by Cloud Dataflow at the time of Java SDK 1.x to >>> demonstrate the capability of this programming model. These initial >>> examples were intended to demonstrate how a user can put together >>> their code components and try out Beam. Since then, there have been >>> numerous updates, increased Python parity, and new features that do >>> not have accompanying examples employing best practices and >>> demonstrating an end-to-end experience for new users. We would like to >>> leverage the existing examples by raising their visibility and >>> auditing them. This is also an opportunity to establish >>> contribution/maintenance guidelines for community contributions and to >>> start hosting the examples on the Beam site in an official repository. >>> Attracting and retaining new users necessitates updated, concrete >>> examples that exhibit the range of capabilities of Beam. >>> >>> Proposed Tasks >>> >>> We would like to create a new GitHub Repository under the Apache >>> Software Foundation Org page for Apache Beam Community Examples. This >>> repo would be similar to apache/beam-site. The name we’d like to have >>> is apache/beam-examples. We will also move all current examples to >>> this repo, perform an audit to outline best practices and guidelines >>> and then publish them in the Apache Beam website. >>> >>> >>> Here is an outlined list of tasks we propose: >>> >>> Send Apache Beam Example Repository proposal to the mailing list >>> (David) - July 31 >>> >>> Create the GitHub Repo (PMC would need to do this) - Request help >>> after proposal is refined/accepted >>> >>> Move current examples to new repo (David) -- 2 weeks after item 2 is >>> completed >>> >>> Add a note to let people we need to audit for best practices >>> >>> Audit current examples and define best practices (David, Rose, Gris) >>> -- Target date: week of 8/20 >>> >>> Write guidelines on adding new examples and maintaining them (Gris, >>> Rose) -- Week after audit is completed >>> >>> Add examples to website (Rose) -- 1 week after guidelines are written >>> >>> Publish guidelines in website (Rose) -- 1 week after guidelines are >>> written >>> >>> >>> >>> On Thu, Aug 9, 2018 at 6:22 AM Łukasz Gajowy <[email protected]> >>> wrote: >>> > >>> > I'd also vote for 3: I don't see much added value in separating the >>> repos and I see much additional effort to be done in maintaining extra >>> repo(s) (updating examples when new version of beam sdk comes out) and >>> their infrastructure (jenkins, etc). What Lukasz Cwik said about mvn >>> archetypes and how easy the examples can be to get starter examples from a >>> common repo only strengthens my opinion. >>> > >>> > Regarding 2: I think it's not good to have some official examples here >>> and some there - IMO it can make a false impression (user experience) that >>> some examples are less important than the others. Maybe a good idea is to >>> encourage users to share their (independent, non official) examples and >>> create a list of such on the beam site instead of 2? >>> > >>> > Łukasz >>> > >>> > czw., 9 sie 2018 o 11:35 Alexey Romanenko <[email protected]> >>> napisał(a): >>> >> >>> >> 3 - I agree with JB, Charles and Lukasz arguments above saying why we >>> need to have examples and main code in the same repository (+ website code >>> base will move there soon). I don’t see any huge benefits to have examples >>> aside and, at the same time, it will bring additional complexity and burden >>> for project support. >>> >> >>> >> On 9 Aug 2018, at 08:18, Jean-Baptiste Onofré <[email protected]> >>> wrote: >>> >> >>> >> Hi guys, >>> >> >>> >> For this kind of discussion, I would prefer to avoid Google Doc and >>> >> directly put the point/proposal on the mailing list. >>> >> >>> >> It's easier for the community to follow. >>> >> >>> >> The statement is more for 3 because it's more convenient for users to >>> >> easily find the examples and include in the distribution. >>> >> >>> >> Regards >>> >> JB >>> >> >>> >> On 08/08/2018 23:25, Charles Chen wrote: >>> >> >>> >> It looks like the main claim is that 1 and 2 have the benefit of >>> >> increasing visibility for examples on the Beam site. I agree with >>> >> Robert's comments on the doc which claim that this is orthogonal to >>> >> whether a separate repository is created (the comments are unresolved: >>> >> >>> https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=AAAABzifZxY >>> ). >>> >> >>> >> I would add that the maintenance and testing burden has not been >>> >> adequately addressed in the proposal (i.e. are we creating new Jenkins >>> >> jobs?; will postcommits on the main Beam repo run examples tests?; are >>> >> we releasing artifacts--if so, is this together with the main package >>> or >>> >> separately in new packages?). If we go with the half-way solution in >>> >> (2), there is also the issue of where the threshold is--for example, >>> if >>> >> a user-contributed example is particularly useful, do we move it to >>> the >>> >> main repo? >>> >> >>> >> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas <[email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> I'd vote for 2. >>> >> >>> >> Giving independence to an example repository and creating the right >>> >> infrastructure to maintain them will give visibility to the efforts >>> >> our users are creating to solve their uses cases with Beam. I also >>> >> want to make the process of sharing common work more easily. >>> >> >>> >> Re:The examples that will remain in core, I agree that it's crucial >>> >> to keep some examples for testing. >>> >> >>> >> >>> >> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik <[email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> I would vote for 3. >>> >> >>> >> My reasoning is that Java has a good mechanism to get a >>> >> starter/example project going by using the the maven archetypes >>> >> already. Our quickstart guide for Apache Beam for the Java SDK >>> >> already covers generating the examples archetype. >>> >> We could point users to the starter project at the end of the >>> >> java quickstart. >>> >> >>> >> If python/go have a similar mechanism that is commonly used, I >>> >> would go with those over creating a separate repo for examples >>> >> and adding the maintenance burden involved. >>> >> >>> >> >>> >> >>> >> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang <[email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> 2 - examples that rely on experimental API can still stay >>> in >>> >> where they are because such examples could be changed. >>> >> >>> >> -Rui >>> >> >>> >> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen < >>> [email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> 3 - We benefit from increased test coverage by having >>> >> examples together with the rest of the code. As Robert >>> >> mentions in the doc, hosting the Beam examples in the >>> >> main repository is the best way to keep the examples >>> >> visible, tested and maintained. Given that we recently >>> >> moved to a single repository for the website since that >>> >> previously caused a lot of pain, it makes sense to be >>> >> consistent here. >>> >> >>> >> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay >>> >> <[email protected] <mailto:[email protected]>> wrote: >>> >> >>> >> 2 - Similar to Huygaa, I see value in keeping a >>> core >>> >> set of examples tested and maintained against head. >>> >> At the same time I understand the value of a >>> growing >>> >> set of community grown examples that are targeted >>> >> against a pre-defined versions of Beam and not >>> >> necessarily updated at every release. >>> >> >>> >> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan >>> >> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >> >>> >> 2 - I like the idea of having a separate repo >>> >> where we can have more freedom to check in >>> >> examples. However, we benefit from having >>> >> immediate core examples in Beam for testing >>> >> purposes. >>> >> >>> >> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos >>> >> <[email protected] >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> Hi everyone! >>> >> >>> >> We discussed several options as well as >>> some >>> >> of the implications of each option. Please >>> >> vote for your favorite option, feel free to >>> >> back it up with any reasons that make you >>> >> feel that way. >>> >> >>> >> 1) Move *all* samples to a *new >>> >> *examples*repository* >>> >> 2) Move *some* samples to a *new >>> >> *examples*repository* >>> >> 3) Leave samples where they are >>> >> >>> >> Some implications to creating a new >>> repository: >>> >> - Every example would be independent from >>> >> every other example, so tests can be run in >>> >> parallel >>> >> - Examples would now show how to use Beam >>> >> /externally/ >>> >> - The examples repository would need a >>> >> testing infrastructure >>> >> - Decoupling makes examples easier to test >>> >> on different versions >>> >> - Easier to copy-paste an existing example >>> >> and start from there, almost like a >>> template >>> >> - Smaller size for the core Beam library >>> >> - Two different repositories to maintain >>> >> - Versioning could mirror Beam's current >>> version >>> >> >>> >> Link to proposal >>> >> < >>> https://docs.google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?usp=sharing >>> > >>> >> >>> >> >>> >> >>> >> -- >>> >> Jean-Baptiste Onofré >>> >> [email protected] >>> >> http://blog.nanthrax.net >>> >> Talend - http://www.talend.com >>> >> >>> >> >>> >> >
