Re: [VOTE] Apache Beam release 0.3.0-incubating
Hi John Rat is supposed to run with the release profile. We are going to check that and why DEPENDENCIES file has not been checked. Regarding Kinesis, the dependency should not be embedded in any Beam jar or distribution. The user has to explicitly define the dependency to be able to use the IO. So it should not be an issue. Let me check if the scope is actually provided there. Thanks Regards JB On Oct 29, 2016, 02:05, at 02:05, "John D. Ament"wrote: >Hi, > >mvn apache-rat:check fails on your release due to the DEPENDENCIES file >not >having a header. If you don't need this file, please remove it. I >would >also recommend leaving apache-rat running all the time to avoid newly >introduced issues. > >In addition, I notice that your build output includes dependencies on >aws-kinesis-client, which is Amazon Software Licensed. Have you >received >clarification on whether you can include or not? > >John > > > >On Fri, Oct 28, 2016 at 4:49 AM Aljoscha Krettek >wrote: > >> Hi everyone, >> Please review and vote on the release candidate #1 for the Apache >Beam >> version 0.3.0-incubating, as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> >> The complete staging area is available for your review, which >includes: >> * JIRA release notes [1], >> * the official Apache source release to be deployed to >dist.apache.org >> [2], >> * all artifacts to be deployed to the Maven Central Repository [3], >> * source code tag "v0.3.0-incubating-RC1" [4], >> * website pull request listing the release and publishing the API >reference >> manual [5]. >> >> The Apache Beam community has unanimously approved this release [6]. >> >> As customary, the vote will be open for at least 72 hours. It is >adopted by >> a majority approval with at least three PMC affirmative votes. If >approved, >> we will proceed with the release. >> >> Thanks! >> >> [1] >> >> >https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051 >> [2] >> >https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/ >> [3] >> >https://repository.apache.org/content/repositories/staging/org/apache/beam/ >> [4] >> >> >https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=5d86ff7f04862444c266142b0d5acecb5a6b7144 >> [5] https://github.com/apache/incubator-beam-site/pull/52 >> [6] >> >> >https://lists.apache.org/thread.html/b3736acb5edcea247a5a6a64c09ecacab794461bf1ea628152faeb82@%3Cdev.beam.apache.org%3E >>
Re: [VOTE] Apache Beam release 0.3.0-incubating
Hi, mvn apache-rat:check fails on your release due to the DEPENDENCIES file not having a header. If you don't need this file, please remove it. I would also recommend leaving apache-rat running all the time to avoid newly introduced issues. In addition, I notice that your build output includes dependencies on aws-kinesis-client, which is Amazon Software Licensed. Have you received clarification on whether you can include or not? John On Fri, Oct 28, 2016 at 4:49 AM Aljoscha Krettekwrote: > Hi everyone, > Please review and vote on the release candidate #1 for the Apache Beam > version 0.3.0-incubating, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > * all artifacts to be deployed to the Maven Central Repository [3], > * source code tag "v0.3.0-incubating-RC1" [4], > * website pull request listing the release and publishing the API reference > manual [5]. > > The Apache Beam community has unanimously approved this release [6]. > > As customary, the vote will be open for at least 72 hours. It is adopted by > a majority approval with at least three PMC affirmative votes. If approved, > we will proceed with the release. > > Thanks! > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051 > [2] > https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/ > [3] > https://repository.apache.org/content/repositories/staging/org/apache/beam/ > [4] > > https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=5d86ff7f04862444c266142b0d5acecb5a6b7144 > [5] https://github.com/apache/incubator-beam-site/pull/52 > [6] > > https://lists.apache.org/thread.html/b3736acb5edcea247a5a6a64c09ecacab794461bf1ea628152faeb82@%3Cdev.beam.apache.org%3E >
Re: Intro + getting started
Hi Nick, Welcome. There are a list of starter jira issues (labeled starter), that could help you to get familiar with the code base. Feel free to grab and assign to yourself, and start contributing. https://issues.apache.org/jira/browse/BEAM-407?filter= 12315361=project%20%3D%20BEAM%20AND%20labels%20%3D%20starter On Fri, Oct 28, 2016 at 2:32 PM, Nick Traverswrote: > Oh cool. I haven't checked the site in a while. Nice additions! > > On Fri, Oct 28, 2016 at 2:09 PM, Dan Halperin > > wrote: > > > Hey Nick, > > > > Awesome! Welcome. > > > > http://beam.incubator.apache.org/contribute/contribution-guide/ is the > > place to start (have you seen it yet? if so, send more specific > questions?) > > > > Dan > > > > On Fri, Oct 28, 2016 at 1:33 PM, Nick Travers > > wrote: > > > > > Hi Beamers, > > > > > > I've been following along the lists for a while now ("long time > listener, > > > first time caller"), and met a few of you at Strata in September. Loved > > all > > > the talks. > > > > > > After using Beam (+ Dataflow runner) for some of my side projects, I'd > > love > > > to jump in and start contributing. Never having worked on libraries > (more > > > application development) or OSS for that matter, it would be good to > pick > > > up some of the smaller tickets as I feel my way around the codebase. > > > > > > Is it possible to be added to the Jira? And is it just a matter of > > picking > > > up unassigned tickets? I see that there are some "newbie" / "starter" > > > tickets that are already assigned but have been open for a while now. > The > > > AutoValue tickets looked like a good place to start. > > > > > > Also, is the Slack channel active? > > > > > > Thanks! Looking forward to getting stuck in! > > > > > > - nick > > > > > > -- Neelesh Srinivas Salian Engineer
Re: Intro + getting started
Oh cool. I haven't checked the site in a while. Nice additions! On Fri, Oct 28, 2016 at 2:09 PM, Dan Halperinwrote: > Hey Nick, > > Awesome! Welcome. > > http://beam.incubator.apache.org/contribute/contribution-guide/ is the > place to start (have you seen it yet? if so, send more specific questions?) > > Dan > > On Fri, Oct 28, 2016 at 1:33 PM, Nick Travers > wrote: > > > Hi Beamers, > > > > I've been following along the lists for a while now ("long time listener, > > first time caller"), and met a few of you at Strata in September. Loved > all > > the talks. > > > > After using Beam (+ Dataflow runner) for some of my side projects, I'd > love > > to jump in and start contributing. Never having worked on libraries (more > > application development) or OSS for that matter, it would be good to pick > > up some of the smaller tickets as I feel my way around the codebase. > > > > Is it possible to be added to the Jira? And is it just a matter of > picking > > up unassigned tickets? I see that there are some "newbie" / "starter" > > tickets that are already assigned but have been open for a while now. The > > AutoValue tickets looked like a good place to start. > > > > Also, is the Slack channel active? > > > > Thanks! Looking forward to getting stuck in! > > > > - nick > > >
Re: Intro + getting started
Hey Nick, Awesome! Welcome. http://beam.incubator.apache.org/contribute/contribution-guide/ is the place to start (have you seen it yet? if so, send more specific questions?) Dan On Fri, Oct 28, 2016 at 1:33 PM, Nick Traverswrote: > Hi Beamers, > > I've been following along the lists for a while now ("long time listener, > first time caller"), and met a few of you at Strata in September. Loved all > the talks. > > After using Beam (+ Dataflow runner) for some of my side projects, I'd love > to jump in and start contributing. Never having worked on libraries (more > application development) or OSS for that matter, it would be good to pick > up some of the smaller tickets as I feel my way around the codebase. > > Is it possible to be added to the Jira? And is it just a matter of picking > up unassigned tickets? I see that there are some "newbie" / "starter" > tickets that are already assigned but have been open for a while now. The > AutoValue tickets looked like a good place to start. > > Also, is the Slack channel active? > > Thanks! Looking forward to getting stuck in! > > - nick >
Intro + getting started
Hi Beamers, I've been following along the lists for a while now ("long time listener, first time caller"), and met a few of you at Strata in September. Loved all the talks. After using Beam (+ Dataflow runner) for some of my side projects, I'd love to jump in and start contributing. Never having worked on libraries (more application development) or OSS for that matter, it would be good to pick up some of the smaller tickets as I feel my way around the codebase. Is it possible to be added to the Jira? And is it just a matter of picking up unassigned tickets? I see that there are some "newbie" / "starter" tickets that are already assigned but have been open for a while now. The AutoValue tickets looked like a good place to start. Also, is the Slack channel active? Thanks! Looking forward to getting stuck in! - nick
Re: [PROPOSAL] New Beam website design?
Thanks a lot for your replies. Could you please provide me the location of the source code of the site.? Then I could look into it. BR On Thu, Oct 27, 2016 at 9:52 PM, Davor Bonaciwrote: > The best place to learn how to get started is the Contribution Guide [1]. > The list of pending JIRA issues related to the website is also available > [2]. > > I think BEAM-752 would be the best to get your feet wet. Other good > candidates are 516, 268, 776. If someone knows a good (non-fragile) > solution to 751, that would be a great contribution! > > Davor > > [1] http://beam.incubator.apache.org/contribute/contribution-guide/ > [2] > https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress% > 22%2C%20Reopened)%20AND%20component%20%3D%20website > > On Thu, Oct 27, 2016 at 5:20 AM, Jean-Baptiste Onofré > wrote: > > > Great !! Thanks. > > > > You can take a look on BEAM-500 and 501 and also the PR I did last week. > > > > I plan to submit new PRs during the week end. So please let me know how > we > > can sync. > > > > Thanks > > Regards > > JB > > > > On Oct 27, 2016, at 14:04, Minudika Malshan > wrote: > >> > >> Hi all, > >> > >> I would like to join for the development of the new site. > >> Is there any issue tracking method for this? (Are there any jirra > issues) > >> > >> Thank you! > >> > >> > >> > >> On Thu, Oct 27, 2016 at 4:01 PM, Jean-Baptiste Onofré > >> wrote: > >> > >> > >> > >>>Hi > >>> > >>> > >>> > >>> You can propose a PR on this Jira. > >>> > >>> > >>> > >>> We will be more than happy to review it. > >>> > >>> > >>> > >>> Thanks > >>> > >>> Regards > >>> > >>> JB > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Oct 27, 2016, 11:26, at 11:26, Abdullah Bashir < > mabdullah...@gmail.com> > >>> > >>> wrote: > >>> > >>> > >>> > Thank you very much for taking time to respond Davor :) > > > > Regarding BEAM-752, i can work on that, i have already built some > > Dataflow > > Piplines on Google Cloud in Python language. > > > > Again Can you tell me where to start for BEAM-752. I am new to ASF > > contribution, so onboarding steps are kind of a black box to me :). > > > > On Thu, Oct 27, 2016 at 11:34 AM, Davor Bonaci > > wrote: > > > > > > > Absolutely! > > > > > > > > I'm currently reviewing JB's PR #51, and that should go in shortly. > > > > > >Within > > > > > a day or so, I should have a better idea about future work in > this > > > > > >specific > > > > > area; please stay tuned. > > > > > > > > There are also separate things that are ready to be started at any > > > > > >time. > > > > > BEAM-752 comes to mind first. Is this something you'd be > interested > > > > > >in? > > > > > > > On Wed, Oct 26, 2016 at 11:17 PM, Abdullah Bashir > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi Davor, > >> > >> > >> > >> I am done with my local setup to start contributing, I have forked > >> > >> > >> > >and > > > > > > >> merged pull request *(**pull/51)* into my local repo. Then I > read > >> > >> > >> > >the > > > > > > >> > >> google docs, their are two tasks mentioned in it, as [Beam-500] and > >> > >> [Beam-501]. > >> > >> I found out that [Beam-500] is closed in JIRA and [Beam-501] is > >> > >> assigned to Jean-Baptiste > >> > >> Onofré, Is their any task that you can assign to me ? > >> > >> > >> > >> Thanks. > >> > >> > >> > >> Regards, > >> > >> Abdullah Bashir > >> > >> > >> > >> > >> > >> On Tue, Oct 25, 2016 at 1:50 AM, Davor Bonaci > >> > >> > >> > >wrote: > > > > > > >> > >> > >> > >>>Abdullah, welcome! > >>> > >>> > >>> > >>> I think it's rather clear we've been struggling with the website, > >>> > >>> > >>> > >> > >so any > > > > > > >> > >>>help is very welcome. It is a little bit messy right now -- > there > >>> > >>> > >>> > >> > >are a > > > > > > >> few > >> > >> > >> > >>>outstanding pull requests and forked branches. I'm trying > to get > >>> > >>> > >>> > >> > >all > > > > > > >>
Re: Why does `Combine.perKey(SerializableFunction)` require same input and output type
For it to be considered a combiner, the function needs to be associative and commutative. The issue is that from an API perspective it would be easy to have a Combine.perKey(SerializableFunction). But many people in the data processing world expect that this parallelization/optimization is performed and thus exposing such a method would be dangerous as it would be breaking users expectations so from the design perspective it is a hard requirement. If PCollections ever become ordered or gain other properties, these requirements may loosen but it seems unlikely in the short term. At this point, I think your looking for a MapElements which you pass in a SerializableFunction >. Creating a wrapper SerializableFunction > which can delegate to a SerializableFunction should be trivial. On Thu, Oct 27, 2016 at 6:17 PM, Manu Zhang wrote: > Thanks for the thorough explanation. I see the benefits for such a > function. > My follow-up question is whether this is a hard requirement. > There are computations that don't satisfy this (I think it's monoid rule) > but possible and easier to write with Combine.perKey( > SerializableFunction ). It's not difficult to > provide an underlying CombineFn. > > > On Thu, Oct 27, 2016 at 9:47 AM Lukasz Cwik > wrote: > >> Combine.perKey takes a single SerializableFunction which knows how to >> convert from Iterable to V. >> >> It turns out that many runners implement optimizations which allow them to >> run the combine operation across several machines to parallelize the work >> and potentially reduce the amount of data they store during a GBK. >> To be able to do such an optimization, it requires you to actually have >> three functions: >> InputT -> AccumulatorT : Creates the intermediate representation which >> allows for associative combining >> Iterable -> AccumulatorT: Performs the actual combining >> AccumT -> OutputT: Extracts the output >> >> In the case of Combine.perKey with a SerializableFunction, your providing >> Iterable -> AccumulatorT and the other two functions are the >> identity functions. >> >> To be able to support a Combine.perKey which can go from Iterable >> -> OutputT would require that this occurred within a single machine >> removing the parallelization benefits that runners provide and for almost >> all cases is not a good idea. >> >> On Wed, Oct 26, 2016 at 6:23 PM, Manu Zhang >> wrote: >> >> > Hi all, >> > >> > I'm wondering why `Combine.perKey(SerializableFunction)` requires input >> > and >> > output to be of the same type while `Combine.PerKey` doesn't have this >> > restriction. >> > >> > Thanks, >> > Manu >> > >> >
[VOTE] Apache Beam release 0.3.0-incubating
Hi everyone, Please review and vote on the release candidate #1 for the Apache Beam version 0.3.0-incubating, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], * all artifacts to be deployed to the Maven Central Repository [3], * source code tag "v0.3.0-incubating-RC1" [4], * website pull request listing the release and publishing the API reference manual [5]. The Apache Beam community has unanimously approved this release [6]. As customary, the vote will be open for at least 72 hours. It is adopted by a majority approval with at least three PMC affirmative votes. If approved, we will proceed with the release. Thanks! [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051 [2] https://dist.apache.org/repos/dist/dev/incubator/beam/0.3.0-incubating/ [3] https://repository.apache.org/content/repositories/staging/org/apache/beam/ [4] https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=tag;h=5d86ff7f04862444c266142b0d5acecb5a6b7144 [5] https://github.com/apache/incubator-beam-site/pull/52 [6] https://lists.apache.org/thread.html/b3736acb5edcea247a5a6a64c09ecacab794461bf1ea628152faeb82@%3Cdev.beam.apache.org%3E
[RESULT] [VOTE] Release 0.3.0-incubating, release candidate #1
I'm happy to announce that we have unanimously approved this release. There are 10 approving votes 6 of which are binding: * Jean-Baptiste (binding) * Ismaël * Sergio * Seetharam * Davor (binding) * Dan (binding) * Kenneth (binding) * Maximilian (binding) * Amit (binding) * Neelesh There are no disapproving votes. As Sergio pointed out, the PPMC votes are not really binding while we are still in incubator, only the IPMC votes will be. I'm nevertheless listing them here as binding because we also did it on past votes. I'll now go ahead and post this result to the IPMC for final voting. Thanks everyone! On Fri, 28 Oct 2016 at 09:09 Aljoscha Krettekwrote: > The voting time has elapsed. I'm hereby closing this vote and will tally > the results in a separate thread. > > On Thu, 27 Oct 2016 at 17:38 Neelesh Salian wrote: > > +1 (non-binding) > Thank you for putting this together > > > On Thu, Oct 27, 2016 at 12:00 AM, Jean-Baptiste Onofré > wrote: > > > No problem for the vote. > > > > For graduation, we are already thinking about it yes. > > > > Regards > > JB > > > > > > > > On Oct 27, 2016, 08:54, at 08:54, "Sergio Fernández" > > wrote: > > >Hi JB, > > > > > >On Tue, Oct 25, 2016 at 12:00 PM, Jean-Baptiste Onofré > > > > > >wrote: > > > > > >> Thanks Sergio ;) > > >> > > > > > >You are welcome. > > > > > > > > >> Just tried to explain to the others what is a binding vote ;) > > >> > > > > > >It's a common mistake in many podlings that PPMC members thing they > > >have > > >binding votes over developers who are not part of the project. But > > >during > > >incubation only IPMC are binding votes. I hope that's clear. > > > > > >In theory it's simple. So sorry if I've made some noise with that. I'll > > >repeat my vote later at general@incubator if you prefer it in that way. > > > > > >Cheers, > > > > > >P.S.: after 0.3.0-incubating, are you thinking about graduation? I > > >think > > >you should ;-) > > > > > > > > > > > >On Oct 25, 2016, 11:53, at 11:53, "Sergio Fernández" > > > > > >> wrote: > > >> >On Tue, Oct 25, 2016 at 11:36 AM, Jean-Baptiste Onofré > > >> > > > >> >wrote: > > >> > > > >> >> By the way, your vote is not binding from a podling perspective > > >(you > > >> >are > > >> >> not PPMC). Your vote is binding from IPMC perspective (so when you > > >> >will > > >> >> vote on the incubator mailing list). > > >> >> > > >> > > > >> >Well, PPMC are never binding votes, only IPMC are actually binding. > > >> >That > > >> >I'm not part of the PPMC is not much relevant. Therefore I think my > > >> >vote is > > >> >still a valid binding one; but I can vote again on > > >general@incubator, > > >> >no > > >> >problem. > > >> > > > >> >Sorry for jumping-in too early. Besides a IPMC, I'm also a developer > > >> >interested in Beam ;-) > > >> > > > >> >Cheers, > > >> > > > >> > > > >> > > > >> > > > >> >> On Oct 25, 2016, 11:33, at 11:33, "Sergio Fernández" > > >> > > > >> >> wrote: > > >> >> >+1 (binding) > > >> >> > > > >> >> >So far I've successfully checked: > > >> >> >* signatures and digests > > >> >> >* source releases file layouts > > >> >> >* matched git tags and commit ids > > >> >> >* incubator suffix and disclaimer > > >> >> >* NOTICE and LICENSE files > > >> >> >* license headers > > >> >> >* clean build (Java 1.8.0_91, Scala, 2.11.7, SBT 0.13.9, Debian > > >> >amd64) > > >> >> > > > >> >> > > > >> >> >Couple of minor issues I've seen it'd be great to have fixed in > > >> >> >upcoming > > >> >> >releases: > > >> >> >* MongoDbIOTest fails (addr already in use) when a Mongo service > > >is > > >> >> >locally > > >> >> >running. I'd say the port should be random in the test suite. > > >> >> >* How did you generated the checksums? Because both SHA1/MD5 > > >can't > > >> >be > > >> >> >automatically checked because "no properly formatted SHA1/MD5 > > >> >checksum > > >> >> >lines found". > > >> >> > > > >> >> >Great to see the project moving forward at this speed :-) > > >> >> > > > >> >> >Cheers, > > >> >> > > > >> >> > > > >> >> > > > >> >> >On Mon, Oct 24, 2016 at 11:30 PM, Aljoscha Krettek > > >> >> > > > >> >> >wrote: > > >> >> > > > >> >> >> Hi Team! > > >> >> >> > > >> >> >> Please review and vote at your leisure on release candidate #1 > > >for > > >> >> >version > > >> >> >> 0.3.0-incubating, as follows: > > >> >> >> [ ] +1, Approve the release > > >> >> >> [ ] -1, Do not approve the release (please provide specific > > >> >comments) > > >> >> >> > > >> >> >> The complete staging area is available for your review, which > > >> >> >includes: > > >> >> >> * JIRA release notes [1], > > >> >> >> * the official Apache source release to be deployed to > > >> >> >dist.apache.org > > >> >> >> [2], > > >> >> >> * all artifacts to be deployed to the Maven Central Repository > > >> >[3], > > >> >> >> *