Re: [PROPOSAL] State and Timers for DoFn (aka per-key workflows)

Hi all, I thought I would loop back on this proposal and email thread with an FYI that coding has begun for this design. Here are some recent PRs for your perusal, if you are interested. https://github.com/apache/incubator-beam/pull/10 44 "Refac

Adding display data to Python SDK

Hello there, I started working on adding display data to the Python SDK. This feature is under Jira issue BEAM-722 . I have a small commit that adds the infrastructure for this (linked in the JIRA issue). Feedback is welcome. If everyone is okay with

Re: Documentation for IDE setup

Thanks guys for doing this! Friction-free contributor experience would be really beneficial. On Fri, Oct 14, 2016 at 8:51 AM, Jean-Baptiste Onofré wrote: > I gonna merge. > > Thanks. > > > On 10/14/2016 05:37 PM, Daniel Kulp wrote: > >> >> On Oct 14, 2016, at 10:06 AM, Jesse Anderson >>> wrote:

Re: Specifying type arguments for generic PTransform builders

On Thu, Oct 13, 2016 at 10:36 PM, Eugene Kirpichov wrote: > I think the choice between #1 or #3 is a red herring - the cases where #3 > is a better choice than #1 are few and far between, and probably not at all > controversial (e.g. ParDo). So I suggest we drop this part of the > discussion. I d

Re: [PROPOSAL] Introduce review mailing list and provide update on open discussion

> On Oct 14, 2016, at 7:46 AM, Jean-Baptiste Onofré wrote: > I think we agreed on most of the points. We also agreed that points 4 & 5 > should be a best effort and not "enforced”. 4 and 5 are really just needed when any “significant change” are part of the discussion. Things like whitespace

Re: Documentation for IDE setup

I gonna merge. Thanks. On 10/14/2016 05:37 PM, Daniel Kulp wrote: On Oct 14, 2016, at 10:06 AM, Jesse Anderson wrote: Last week I imported Beam with IntelliJ and everything worked. That said, I tried to import the Eclipse project and that doesn't compile anymore. I didn't have time to figu

Re: [DISCUSS] Introduce DoFnWithStore

Hi JB, Can you clarify more what you want to achieve by "writing an unbounded PCollection to a bounded sink": I see several options: 1. Read some amount of data from the collection (limited by number of elements or by time), and write just that, and stop reading from it - as if the collection was

Re: Documentation for IDE setup

> On Oct 14, 2016, at 10:06 AM, Jesse Anderson wrote: > > Last week I imported Beam with IntelliJ and everything worked. > > That said, I tried to import the Eclipse project and that doesn't compile > anymore. I didn't have time to figure out what happened though. > I have a pull request http

Re: Documentation for IDE setup

Btw. I finished the documentation now and created a PR: https://github.com/apache/incubator-beam-site/pull/44 While testing the documentation I also found this issue: https://github.com/apache/incubator-beam-site/pull/45 Christian On 14.10.2016 10:17, Christian Schneider wrote: Hello all, I a

Re: Documentation for IDE setup

I rely on having the Maven Eclipse integration and m2e-apt and do a maven import of a project. On Fri, Oct 14, 2016 at 8:10 AM, Jesse Anderson wrote: > I did a "mvn eclipse:eclipse" to generate the Eclipse projects and imported > them. That didn't compile either. > > On Fri, Oct 14, 2016 at 8:06

Re: [DISCUSS] Introduce DoFnWithStore

The only way we have today is to use BoundedReadFromUnboundedSource or use a side input to bridge an unbounded portion of the pipeline with a bounded portion of the pipeline. The model allows the side input bridge between these two portions of the pipeline to happen but I can't comment as to how we

Re: Documentation for IDE setup

I use Eclipse for development but always defer to maven since its the source of truth in the end. I also have issues with getting it to compile on import and it has to do with annotation processing and generally requires m2e-apt to be installed and configured correctly. On Fri, Oct 14, 2016 at 7:2

Re: Documentation for IDE setup

I did a "mvn eclipse:eclipse" to generate the Eclipse projects and imported them. That didn't compile either. On Fri, Oct 14, 2016 at 8:06 AM Lukasz Cwik wrote: > I use Eclipse for development but always defer to maven since its the > source of truth in the end. > I also have issues with getting

Re: [DISCUSS] Introduce DoFnWithStore

Thanks for the update Lukasz. How would you implement a "transform" from unbounded PCollection to bounded PCollection ? Even if I use a GroupByKey with something like KV>, it doesn't change the type of the PCollection. You are right with State API. My proposal is more a way to implicitly u

Re: [DISCUSS] Introduce DoFnWithStore

SplittableDoFn is about taking a single element and turning it into potentially many in a parallel way by allowing an element to be split across bundles. I believe a user could do what you describe by using a GBK to group their data how they want. In your example it would be a single key, then the

Re: Documentation for IDE setup

I was looking for the same couple of days ago. But IntelliJ is less worrisome than Eclipse. Straight Import. No Hassle. +1 to docs, though. On Fri, Oct 14, 2016 at 7:19 AM, Jean-Baptiste Onofré wrote: > [Troll] Who's using Eclipse anymore ? [/Troll] > > ;) > > Regards > JB > > > On 10/14/2016 0

Re: Documentation for IDE setup

[Troll] Who's using Eclipse anymore ? [/Troll] ;) Regards JB On 10/14/2016 04:06 PM, Jesse Anderson wrote: Last week I imported Beam with IntelliJ and everything worked. That said, I tried to import the Eclipse project and that doesn't compile anymore. I didn't have time to figure out what ha

Re: Documentation for IDE setup

Last week I imported Beam with IntelliJ and everything worked. That said, I tried to import the Eclipse project and that doesn't compile anymore. I didn't have time to figure out what happened though. On Fri, Oct 14, 2016 at 1:21 AM Jean-Baptiste Onofré wrote: > Hi Christian, > > IntelliJ doesn

Re: [PROPOSAL] New Beam website design?

Hi James, just to let you know that I did good progress on the website mockup. I should be able to propose a PR very soon. Thanks for your patience ;) Regards JB On 06/06/2016 05:29 PM, James Malone wrote: Hello everyone! The current design of the Apache Beam website[1] is based on the a ba

[DISCUSS] Introduce DoFnWithStore

Hi guys, When testing the different IOs, we want to have the best possible coverage and be able to test with different use cases. We create integration test pipelines, and, one "classic" use case is to implement a pipeline starting from an unbounded source (providing an unbounded PCollection

Re: [PROPOSAL] Introduce review mailing list and provide update on open discussion

Hi guys, I think we agreed on most of the points. We also agreed that points 4 & 5 should be a best effort and not "enforced". If there's no objection, I will create the review mailing list and update the github integration configuration. Thanks all for your comments and feebacks ! Regards

Re: Documentation for IDE setup

Hi Christian, IntelliJ doesn't need any special config (maybe the code style can be documented or imported). Anyway, agree to add such on website in the contribute directory. I think it could be part of the contribution-guide as it's first setup step. Regards JB On 10/14/2016 10:17 AM, Chr

Documentation for IDE setup

Hello all, I am new to the beam community and currently start making myself familiar with the code. I quickly found the contribution guide and was able to clone the code and build beam using maven. The first obstacle I faced was getting the code build in eclipse. I naively imported as exist