Re: [ANNOUNCE] New committer: Rui Wang

Congrats Rui!! On Wed, Aug 7, 2019 at 12:54 PM Rui Wang wrote: > Thank you guys! Looking forward to contributing more to Beam community! > > > -Rui > > On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 11:28 PM Robin Qiu wrote: >> >>

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats Kyle!! On Wed, Aug 7, 2019 at 11:30 AM Heejong Lee wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 11:27 PM Robin Qiu wrote: >> >>> Congratulations, Kyle! >>> >>> On Wed, Aug 7, 2019 at 5:04 A

Re: [ANNOUNCE] New committer: Jan Lukavský

Congratulations Jan!! On Wed, Aug 7, 2019 at 5:05 AM Valentyn Tymofieiev wrote: > Congrats, Jan! > > On Wed, Aug 7, 2019 at 8:01 AM Connell O'Callaghan > wrote: > >> Well done Jan!!! >> >> On Tue, Aug 6, 2019 at 15:44 Chamikara Jayalath >> wrote: >> >>> Congrats Jan! >>> >>> On Thu, Aug 1, 201

[BEAM-6855] Please review merge request.

Hi All, Please review https://github.com/apache/beam/pull/9140 Thanks. -- *Regards,Muhammad Talha Jamil* *Computer Systems Engineer*

Re: Python DoFn.setup()

Replied on the JIRA issue[1] you filed. Let's keep the conversation in one place in the JIRA. [1] https://issues.apache.org/jira/browse/BEAM-7885 Ahmet On Wed, Aug 7, 2019 at 1:19 PM Niklas Hansson wrote: > Hi! > > I have a streaming (unbound) pipeline in which i like to use a sklearn( > http:

Re: Jira email notifications

Yes, I am already on the "Watchers" list On Wed, Aug 7, 2019 at 7:13 PM Pablo Estrada wrote: > Have you tried "watching" the particular JIRA issue? There's a "Watch" > thing on the right-hand side of an issue page. > > Happy to help more if that's not helpful : ) > Best > -P. > > On Wed, Aug 7,

Re: Jira email notifications

Have you tried "watching" the particular JIRA issue? There's a "Watch" thing on the right-hand side of an issue page. Happy to help more if that's not helpful : ) Best -P. On Wed, Aug 7, 2019 at 5:09 PM sridhar inuog wrote: > Hi, >Is there a way to get notifications whenever a jira issu

Jira email notifications

Hi, Is there a way to get notifications whenever a jira issue is updated? The only place I can see this can be enabled is profile -> Preferences -> My Changes (Notify me) Even though the description seems a little bit misleading I don't see any other place to make any changes. --- W

Re: Brief of interactive Beam

Ning, thank you for the heads up. All, this is a proposed work for improving interactive Beam experience. As mentioned in Ning's email, new concepts are being introduced. And in addition iBeam as a name is used as a new reference. I hope that bringing the discussion to the mailing list will give i

Re: [Discuss] Ideas for Apache Beam presence in social media

Thanks Thomas, changed the doc to view only and granted you and Ahmet edit access. @all - please send requests for access with your google accounts. I will update the thread once I document the process and submit the PR to the website. Thank you, Aizhamal On Wed, Aug 7, 2019 at 3:12 PM Thomas Wei

Re: Java 11 compatibility question

On Wed, Aug 7, 2019 at 12:21 PM Elliotte Rusty Harold wrote: > gRPC bug here: https://github.com/grpc/grpc-java/issues/3522 > > google-cloud-java bug: > https://github.com/googleapis/google-cloud-java/issues/5760 > > Neither has a cheap or easy fix, I'm afraid. Commenting on these > issues might

Re: [Discuss] Ideas for Apache Beam presence in social media

I was able to subscribe now. Reminder for others that the spreadsheet of interest can be found here: s.apache.org/beam-tweets Aizhamal, Can you help with a couple changes to bring this closer to how similar gdoc resources are handled? * Make the document view only. *PMC members* that care to he

Re: [Update] Beam 2.15 Release Progress

Thanks Udi. On Wed, Aug 7, 2019 at 2:58 PM Udi Meiri wrote: > https://github.com/apache/beam/pull/9240 has been merged > > On Wed, Aug 7, 2019 at 12:33 PM Anton Kedin wrote: > >> Perf regression is seemingly gone now. If this is caused by a PR we might >> want to find out which one and cherry-p

Re: [Update] Beam 2.15 Release Progress

https://github.com/apache/beam/pull/9240 has been merged On Wed, Aug 7, 2019 at 12:33 PM Anton Kedin wrote: > Perf regression is seemingly gone now. If this is caused by a PR we might > want to find out which one and cherry-pick it into the release. > > Regards, > Anton > > On Tue, Aug 6, 2019 a

Re: [Discuss] Ideas for Apache Beam presence in social media

I am able to subscribe to notifications now. Thomas does it work for you? On Wed, Aug 7, 2019 at 2:23 PM Aizhamal Nurmamat kyzy wrote: > Hi all, > > I set the access to 'anyone can edit'. Let me know if notifications work > now. > > Thanks, > > On Wed, Aug 7, 2019 at 2:00 PM Ahmet Altay wrote:

Re: [Discuss] Ideas for Apache Beam presence in social media

Hi all, I set the access to 'anyone can edit'. Let me know if notifications work now. Thanks, On Wed, Aug 7, 2019 at 2:00 PM Ahmet Altay wrote: > You are probably right and it is an access issue. > > Aizhamal, could you give us edit access? And we can see if notifications > work after that. >

Re: [PROPOSAL] An initial Schema API in Python

Thanks for all the suggestions, I've added responses inline. On Wed, Aug 7, 2019 at 12:52 PM Chad Dombrova wrote: > There’s a lot of ground to cover here, so I’m going to pull from a few > different responses. > -- > > numpy ints > > A properly written library should

Re: [Discuss] Ideas for Apache Beam presence in social media

You are probably right and it is an access issue. Aizhamal, could you give us edit access? And we can see if notifications work after that. On Wed, Aug 7, 2019 at 1:41 PM Thomas Weise wrote: > The use of JIRA was also suggested before, but why do the notifications > not work? I wasn't able to s

Re: [Discuss] Ideas for Apache Beam presence in social media

The use of JIRA was also suggested before, but why do the notifications not work? I wasn't able to subscribe and I suspect that was due to not having sufficient access to the spreadsheet? On Wed, Aug 7, 2019 at 1:26 PM Ahmet Altay wrote: > As far as I understand we have not resolved this discu

Re: Python postcommits broken: crossLanguagePythonJavaFlink

Thank you for filing. Assigned to +Heejong Lee , he was working on this test. On Wed, Aug 7, 2019 at 1:21 PM Udi Meiri wrote: > I opened a bug here: https://issues.apache.org/jira/browse/BEAM-7924 > but I don't know who's the best person to take a look. > Could someone assign this please? >

Re: [Discuss] Ideas for Apache Beam presence in social media

As far as I understand we have not resolved this discussion and the sticking issue is that there is no good way of subscribing to changes (i.e. proposals for tweets) for interested parties. The method suggested in this thread (e.g. Tools and then Notification rules.) does not work for some reason f

Python postcommits broken: crossLanguagePythonJavaFlink

I opened a bug here: https://issues.apache.org/jira/browse/BEAM-7924 but I don't know who's the best person to take a look. Could someone assign this please? smime.p7s Description: S/MIME Cryptographic Signature

Python DoFn.setup()

Hi! I have a streaming (unbound) pipeline in which i like to use a sklearn( http://scikit-learn.github.io/stable) model for making predictions based upon input from a pubsub stream. In order to avoid loading the model multiple times I would like to use the DoFn.Setup() feature that where implement

Re: Beam Python Portable Runner - Adding timeout to JobServer grpc calls

Could we pick a default timeout value instead of introducing a flag? We use 60 seconds as the default timeout for http client [1], we can do the same here. [1] https://github.com/apache/beam/blob/3a182d64c86ad038692800f5c343659ab0b935b0/sdks/python/apache_beam/internal/http_client.py#L32 On Wed,

Re: Support ZetaSQL as a new SQL dialect in BeamSQL

Thanks to highlight the parts of types/operators/functions/..., that does make things more complicated. +1 that as a short/middle term solution, the proposal is reasonable. We could follow up in future to handle it in Calcite Babel if possible. Mingmin On Tue, Aug 6, 2019 at 3:57 PM Rui Wang wro

Re: [ANNOUNCE] New committer: Rui Wang

Thank you guys! Looking forward to contributing more to Beam community! -Rui On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 11:28 PM Robin Qiu wrote: > >> Congratulations, Rui! >> >> On Wed, Aug 7, 2019 at 5:03 AM Valentyn Tymofieiev >

Re: [PROPOSAL] An initial Schema API in Python

There’s a lot of ground to cover here, so I’m going to pull from a few different responses. -- numpy ints A properly written library should accept any type implementing the *int* (or *index*) methods in place of an int, rather than doing explicit type checks Yes, but

Brief of interactive Beam

To whom may concern, This is Ning from Google. We are currently making efforts to leverage an interactive runner under python beam sdk. There is already an interactive Beam (iBeam for short) runner with jupyter notebook in the repo

Re: [Update] Beam 2.15 Release Progress

Perf regression is seemingly gone now. If this is caused by a PR we might want to find out which one and cherry-pick it into the release. Regards, Anton On Tue, Aug 6, 2019 at 4:52 PM Yifan Zou wrote: > Hi, > > There is a perf regression on SQL Query3 on dataflow runner. This was > treated as a

Re: Java 11 compatibility question

gRPC bug here: https://github.com/grpc/grpc-java/issues/3522 google-cloud-java bug: https://github.com/googleapis/google-cloud-java/issues/5760 Neither has a cheap or easy fix, I'm afraid. Commenting on these issues might help us prove that there's a demand to priorotize these compared to other w

Hardening BigQuery file loads in Python SDK

Hi everyone, Load jobs into BigQuery are subject to various quotas and limitations. In the Python SDK, the BigQuery sink that uses file loads does not work well with these quotas and limitations. Improvements are needed in the following areas: 1. Handle per load job limitations on the total si

Re: [PROPOSAL] An initial Schema API in Python

On Tue, Aug 6, 2019 at 3:55 AM Robert Bradshaw wrote: > On Sun, Aug 4, 2019 at 12:03 AM Chad Dombrova wrote: > > > > Hi, > > > > This looks like a great feature. > > > > Is there a plan to eventually support custom field types? > > > > I assume adding support for dataclasses in python 3.7+ shoul

Proposal for SDFs in the Go SDK

Hello Beam devs, I've been working on a proposal for implementing SDFs in the Go SDK. For those who were unaware, the Go SDK hasn't supported SDFs in any capacity yet, so my proposal covers the user-facing API and a basic look into how it will work under the hood. I'd appreciate it if anyone inte

Beam Python Portable Runner - Adding timeout to JobServer grpc calls

Hello, I noticed that the calls to the JobServer from the Python SDK do not have timeouts. If I'm not mistaken that means that the call to pipeline.run() could hang forever if the JobServer is not running (or failing to start). E.g. https://github.com/apache/beam/blob/master/sdks/python/apache_bea

Re: [PROPOSAL] An initial Schema API in Python

On Wed, Aug 7, 2019 at 10:51 AM Brian Hulette wrote: > If it is not a big deal supporting both sounds good. I was actually >> referring to your comment about typing.Collection not being available on >> python 2. >> > > Oh, of course, sorry somehow that completely slipped my mind. Now that I'm >

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations! On Wed, Aug 7, 2019 at 11:05 AM Tanay Tummalapalli wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 11:27 PM Robin Qiu wrote: > >> Congratulations, Kyle! >> >> On Wed, Aug 7, 2019 at 5:04 AM Valentyn Tymofieiev >> wrote: >> >>> Congrats, Kyle! >>> >>> On Wed, Aug 7, 2019 a

Re: Java 11 compatibility question

Since java8 -> java11 is similar to python2 -> python3 migration, what was the acceptance criteria there? On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold wrote: > > > On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia > wrote: > >> >> Are these tests sufficient to say that we’re java 11 compati

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

I wanted to add some more details about the state discussion. BEAM-7000 is about adding support for a gRPC message saying that the SDK is now blocked on one of its requests. This would allow for an easy optimization on the runner side where it gathers requests and is able to batch them knowing tha

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations! On Wed, Aug 7, 2019 at 11:27 PM Robin Qiu wrote: > Congratulations, Kyle! > > On Wed, Aug 7, 2019 at 5:04 AM Valentyn Tymofieiev > wrote: > >> Congrats, Kyle! >> >> On Wed, Aug 7, 2019 at 1:01 PM Ismaël Mejía wrote: >> >>> Congrats Kyle, well deserved :clap: ! >>> >>> On Wed,

Re: [ANNOUNCE] New committer: Rui Wang

Congratulations! On Wed, Aug 7, 2019 at 11:28 PM Robin Qiu wrote: > Congratulations, Rui! > > On Wed, Aug 7, 2019 at 5:03 AM Valentyn Tymofieiev > wrote: > >> Congrats, Rui! >> >> On Wed, Aug 7, 2019 at 1:00 PM Ismaël Mejía wrote: >> >>> Congrats Rui! >>> >>> On Wed, Aug 7, 2019, 11:37 AM Gleb

Re: [ANNOUNCE] New committer: Rui Wang

Congratulations, Rui! On Wed, Aug 7, 2019 at 5:03 AM Valentyn Tymofieiev wrote: > Congrats, Rui! > > On Wed, Aug 7, 2019 at 1:00 PM Ismaël Mejía wrote: > >> Congrats Rui! >> >> On Wed, Aug 7, 2019, 11:37 AM Gleb Kanterov wrote: >> >>> Congratulations Rui! Well done! >>> >>> On Wed, Aug 7, 2019

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations, Kyle! On Wed, Aug 7, 2019 at 5:04 AM Valentyn Tymofieiev wrote: > Congrats, Kyle! > > On Wed, Aug 7, 2019 at 1:01 PM Ismaël Mejía wrote: > >> Congrats Kyle, well deserved :clap: ! >> >> On Wed, Aug 7, 2019, 11:22 AM Gleb Kanterov wrote: >> >>> Congratulations! >>> >>> On Wed,

Re: Waht would be the best place for performance tests documentation?

I also think confluence makes the most sense. On Wed, Aug 7, 2019 at 11:57 AM Alexey Romanenko wrote: > I agree with Cyrus that Confluence page should a good place for that > since, seems, it will be very dev oriented documentation. > > > On 7 Aug 2019, at 16:31, Cyrus Maden wrote: > > Hi Łukas

Re: Java 11 compatibility question

On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia wrote: > > Are these tests sufficient to say that we’re java 11 compatible? What > other aspects do we need to test to be able to say that? > > Are any packages split across multiple jar files, including packages beam dependns on? That's the one that'

Re: [PROPOSAL] An initial Schema API in Python

> > If it is not a big deal supporting both sounds good. I was actually > referring to your comment about typing.Collection not being available on > python 2. > Oh, of course, sorry somehow that completely slipped my mind. Now that I'm actually thinking it through, you're right there are several

Re: (mini-doc) Beam (Flink) portable job templates

> For example, there could be access to a file system or other service to fetch metadata that is required to build the pipeline. That's a good point. It's totally up to users to decide how they want to deploy. I just think the jar solution would provide a useful option for many, but not all use ca

Re: (mini-doc) Beam (Flink) portable job templates

--> > > > > * The pipeline construction code itself may need access to cluster > resources. In such cases the jar file cannot be created offline. > > Could you elaborate? > The entry point is arbitrary code written by the user, not limited to Beam pipeline construction alone. For example, there c

Re: Waht would be the best place for performance tests documentation?

I agree with Cyrus that Confluence page should a good place for that since, seems, it will be very dev oriented documentation. > On 7 Aug 2019, at 16:31, Cyrus Maden wrote: > > Hi Łukasz, > > This sounds perfect for the confluence, since we already have the testing > guide >

Re: Collecting metrics in JobInvocation - BEAM-4775

Thanks, Kenn and Robert! It got me thinking and digging. In terms of BEAM-4775 I'm currently working on, it seems the only thing I can do now is preparing a version without collecting metrics when after cancel(). I think I'll go this direction firs

Re: (mini-doc) Beam (Flink) portable job templates

Added comment access to the doc link. On Wed, Aug 7, 2019 at 3:00 AM Robert Bradshaw wrote: > On Wed, Aug 7, 2019 at 6:20 AM Thomas Weise wrote: > > > > Hi Kyle, > > > > [document doesn't have comments enabled currently] > > > > As noted, worker deployment is an open question. I believe pipelin

Re: Waht would be the best place for performance tests documentation?

Hi Łukasz, This sounds perfect for the confluence, since we already have the testing guide in there. There's already a link to the testing guide in the beam.apache.org/contribute section, which might help direct folks to

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

The list looks good to me. Thanks for summarizing. Feel free to dive into any of these issues yourself :). On Fri, Aug 2, 2019 at 6:24 PM jincheng sun wrote: > > Hi all, > > > Thanks a lot for sharing your thoughts! > > > It seems that we have already reached consensus for the following items. >

Java 11 compatibility question

Hi everyone, I want to gather the collective knowledge here about Java 11 compatibility and ask about the tests needed to deem Beam compatible with JDK 11. Right now concerning testing JDK 11 compatibility I implemented: - Jenkins jobs for running ValidatesRunner test sets in both Direct and

Re: Collecting metrics in JobInvocation - BEAM-4775

I think the question here is whether PipelineRunner::run is allowed to be blocking. If it is, then the futures make sense (but there's no way to properly cancel it). I'm OK with not being able to return metrics on cancel in this case, or the case the pipeline didn't even start up yet. Otherwise, we

Re: [ANNOUNCE] New committer: Jan Lukavský

Congrats, Jan! On Wed, Aug 7, 2019 at 8:01 AM Connell O'Callaghan wrote: > Well done Jan!!! > > On Tue, Aug 6, 2019 at 15:44 Chamikara Jayalath > wrote: > >> Congrats Jan! >> >> On Thu, Aug 1, 2019 at 11:10 AM Thomas Weise wrote: >> >>> Congrats! >>> >>> >>> On Thu, Aug 1, 2019 at 8:44 AM jinc

Re: [ANNOUNCE] New committer: Rui Wang

Congrats, Rui! On Wed, Aug 7, 2019 at 1:00 PM Ismaël Mejía wrote: > Congrats Rui! > > On Wed, Aug 7, 2019, 11:37 AM Gleb Kanterov wrote: > >> Congratulations Rui! Well done! >> >> On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan >> wrote: >> >>> Well done Rui!!! >>> >>> On Tue, Aug 6, 2019 a

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats, Kyle! On Wed, Aug 7, 2019 at 1:01 PM Ismaël Mejía wrote: > Congrats Kyle, well deserved :clap: ! > > On Wed, Aug 7, 2019, 11:22 AM Gleb Kanterov wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan >> wrote: >> >>> Well done congratulations Kyle!!! >>

Waht would be the best place for performance tests documentation?

Hi all, I'm currently working on documenting the load tests of Core Apache Beam operations as we have some Jenkins jobs running and several dashboards for that. This is what I've got so far (work in progress b

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats Kyle, well deserved :clap: ! On Wed, Aug 7, 2019, 11:22 AM Gleb Kanterov wrote: > Congratulations! > > On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan > wrote: > >> Well done congratulations Kyle!!! >> >> On Tue, Aug 6, 2019 at 21:58 Thomas Weise wrote: >> >>> Congrats! >>> >>> On

Re: (mini-doc) Beam (Flink) portable job templates

On Wed, Aug 7, 2019 at 6:20 AM Thomas Weise wrote: > > Hi Kyle, > > [document doesn't have comments enabled currently] > > As noted, worker deployment is an open question. I believe pipeline > submission and worker execution need to be considered together for a complete > deployment story. The i

Re: [ANNOUNCE] New committer: Rui Wang

Congrats Rui! On Wed, Aug 7, 2019, 11:37 AM Gleb Kanterov wrote: > Congratulations Rui! Well done! > > On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan > wrote: > >> Well done Rui!!! >> >> On Tue, Aug 6, 2019 at 15:41 Chamikara Jayalath >> wrote: >> >>> Congrats Rui. >>> >>> On Tue, Aug 6, 2

Re: [ANNOUNCE] New committer: Rui Wang

Congratulations Rui! Well done! On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan wrote: > Well done Rui!!! > > On Tue, Aug 6, 2019 at 15:41 Chamikara Jayalath > wrote: > >> Congrats Rui. >> >> On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak >> wrote: >> >>> Congrats Rui! >>> >>> On Tue, Aug

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations! On Wed, Aug 7, 2019 at 7:01 AM Connell O'Callaghan wrote: > Well done congratulations Kyle!!! > > On Tue, Aug 6, 2019 at 21:58 Thomas Weise wrote: > >> Congrats! >> >> On Tue, Aug 6, 2019, 7:24 PM Reza Rokni wrote: >> >>> Congratz! >>> >>> On Wed, 7 Aug 2019 at 06:40, Chamikar

Re: [ANNOUNCE] New committer: Rui Wang

Congrats Rui! On 8/7/19 7:01 AM, Connell O'Callaghan wrote: Well done Rui!!! On Tue, Aug 6, 2019 at 15:41 Chamikara Jayalath > wrote: Congrats Rui. On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak mailto:meliss...@google.com>> wrote: Congrats

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats Kyle! On 8/7/19 7:00 AM, Connell O'Callaghan wrote: Well done congratulations Kyle!!! On Tue, Aug 6, 2019 at 21:58 Thomas Weise > wrote: Congrats! On Tue, Aug 6, 2019, 7:24 PM Reza Rokni mailto:r...@google.com>> wrote: Congratz! On W