Re: Beam's recent community development work

Dang. I missed that. Ross is exactly right here. GREAT idea. I am going to push this all over. On Mon, Jul 2, 2018 at 8:27 PM wrote: > There is one insight here that I particularly like and I believe helps me > find a good compromise that I’ve struggled with for years. I’m a fan of CTR > rat

RE: Beam's recent community development work

There is one insight here that I particularly like and I believe helps me find a good compromise that I’ve struggled with for years. I’m a fan of CTR rather than RTC for committers. However, I recognize that a number of projects don’t share my views on this. I ***love*** your solution and will q

Re: Invite to comment on the @RequiresStableInput design doc

The common use case for a timer is to read in data that was stored using the state API in processElement. There is no guarantee that is stable, and I believe no runner currently guarantees this. For example: class MyDoFn extends DoFn { @StateId("bag") private final StateSpec> buffer = StateSpec.

Re: Invite to comment on the @RequiresStableInput design doc

Could you give an example of such a usecase? (I suppose I'm not quite following what it means for a timer to be unstable...) On Mon, Jul 2, 2018 at 6:20 PM Reuven Lax wrote: > One issue: we definitely have some strong use cases where we want this on > ProcessTimer but not on ProcessElement. Sinc

Re: Invite to comment on the @RequiresStableInput design doc

One issue: we definitely have some strong use cases where we want this on ProcessTimer but not on ProcessElement. Since both are on the same DoFn, I'm not sure how you would represent this as a separate transform. On Mon, Jul 2, 2018 at 5:05 PM Robert Bradshaw wrote: > Thanks for the writeup. >

Re: Invite to comment on the @RequiresStableInput design doc

Thanks for the writeup. I'm wondering with, rather than phrasing this as an annotation on DoFn methods that gets plumbed down through the portability representation, if it would make more sense to introduce a new, primitive "EnsureStableInput" transform. For those runners whose reshuffle provide s

Re: Could any PMC member add me as Beam contributor?

Done. On Mon, Jul 2, 2018 at 4:52 PM Robin Qiu wrote: > Could any PMC member add me as a Beam contributor, such that JIRA issued > can be assigned to me? My JIRA user name is robinyqiu, and name is Yueyang > Qiu. > > Thanks! > Robin > >

Could any PMC member add me as Beam contributor?

Could any PMC member add me as a Beam contributor, such that JIRA issued can be assigned to me? My JIRA user name is robinyqiu, and name is Yueyang Qiu. Thanks! Robin

Re: Beam's recent community development work

Thanks for the guidance Ted, All of your points are well taken. I/we will definitely stay careful about phrasing encouragement emails and our guidelines. Kenn On Sat, Jun 30, 2018 at 8:45 AM Ted Dunning wrote: > > Ken, > > This is really good. > > I would like to emphasize one nuance, however.

Re: Unbounded source translation for portable pipelines

Updated the Flink section. To run a basic Python wordcount (sent to you in a separate thread, but repeating here too for others to play with): Step 1: Run once to build a container: "./gradlew -p sdks/python/container docker" Step 2: ./gradlew :beam-runners-flink_2.11-job-server:runShadow - this

Re: Parallelizing test runs

Done! On Mon, Jul 2, 2018 at 4:10 PM Scott Wegner wrote: > Hey Rafael, looks like we need more 'INSTANCE_TEMPLATES' quota [1]. Can > you take a look? I've filed [BEAM-4722]: > https://issues.apache.org/jira/browse/BEAM-4722 > > [1] https://github.com/apache/beam/pull/5861#issuecomment-401963630

Jenkins build is back to normal : beam_Release_Gradle_NightlySnapshot #88

See

Re: Parallelizing test runs

Hey Rafael, looks like we need more 'INSTANCE_TEMPLATES' quota [1]. Can you take a look? I've filed [BEAM-4722]: https://issues.apache.org/jira/browse/BEAM-4722 [1] https://github.com/apache/beam/pull/5861#issuecomment-401963630 On Mon, Jul 2, 2018 at 11:33 AM Rafael Fernandez wrote: > OK, Scot

Re: Invite to comment on the @RequiresStableInput design doc

Hi everyone, Thanks for your feedback on the doc. I have revamped it according to all of the comments. The major changes I have made are: * The problem description should be more general and accurate now. * I added more background information, such as details about Reshuffle, so I should be easier

Re: Python 3 support in the Python SDK

Hi Charles, Thanks for the heads up. Looking at BEAM-2784, most of the sub-tickets are either DONE or IN PROGRESS, meaning that the futurization is almost finished, right? Should I wait a bit, and then help to port/debug the test code? Sergei On Mon, Jul 2, 2018 at 10:43 PM Charles Chen wrote:

Re: Python 3 support in the Python SDK

Hi Sergei, Matthias and Robbe are actively working on this support. Their plan is to futurize all relevant modules and then work on Beam Python 3 tests; this is being tracked in https://issues.apache.org/jira/browse/BEAM-2784 and I added https://issues.apache.org/jira/browse/BEAM-4715 as well. W

Python 3 support in the Python SDK

Hello, The Beam Python SDK does not currently support Python 3. This limits the use of Beam itself, as well as some other projects depending on it (e.g. TensorFlow Model Analysis [1]). There is an ongoing effort on making the SDK Python 3-compatible (see e.g. [2]). However, there is no up-to-date

Re: Parallelizing test runs

OK, Scott just sent https://github.com/apache/beam/pull/5860 . Quotas should not be a problem, if they are, please file a JIRA under gcp-quota. Cheers, r On Mon, Jul 2, 2018 at 10:06 AM Kenneth Knowles wrote: > One thing that is nice when you do this is to be able to share your > results. Thoug

Re: Help! Beam SQL needs more committer support

Haha! I don't even know who I should @ now because there are so many helpful hands! -Rui On Mon, Jul 2, 2018 at 9:24 AM Andrew Pilloud wrote: > Thanks to everyone who's volunteered to help review SQL PRs. Sounds like > we will be in good hands while Kenn is out! > > Andrew > > On Fri, Jun 29, 2

Re: Parallelizing test runs

One thing that is nice when you do this is to be able to share your results. Though if all you are sharing is "they passed" then I guess we don't have to insist on evidence. Kenn On Mon, Jul 2, 2018 at 9:25 AM Scott Wegner wrote: > A few thoughts: > > * The Jenkins job getting backed up > is be

Re: Parallelizing test runs

A few thoughts: * The Jenkins job getting backed up is beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR [1]. Since Mikhail refactored Jenkins jobs, this only runs when explicitly requested via "Run Dataflow ValidatesRunner", and only has 8 total runs. So this job is idle more often than bac

Re: Help! Beam SQL needs more committer support

Thanks to everyone who's volunteered to help review SQL PRs. Sounds like we will be in good hands while Kenn is out! Andrew On Fri, Jun 29, 2018 at 7:45 AM Alexey Romanenko wrote: > I’d be happy to learn and help with Beam SQL as well! > > Alexey > > On 28 Jun 2018, at 22:12, Ahmet Altay wrote

Re: Parallelizing test runs

The validates runner test parallelism is controlled here and is currently set to be "unlimited": https://github.com/apache/beam/blob/fbfe6ceaea9d99cb1c8964087aafaa2bc2297a03/runners/google-cloud-dataflow-java/build.gradle#L115 Each test fork is run on a different gradle worker, so the number of pa

Jenkins build is back to normal : beam_SeedJob #2137

See

Build failed in Jenkins: beam_SeedJob #2136

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam14 (beam) in workspace > git rev-parse --

Build failed in Jenkins: beam_SeedJob #2135

See -- GitHub pull request #5852 of commit 98f13829508155a5a612f5a533b2a8d3f1b19e63, no merge conflicts. Setting status of 98f13829508155a5a612f5a533b2a8d3f1b19e63 to PENDING with url https

Jenkins build is back to normal : beam_SeedJob #2134

See

Build failed in Jenkins: beam_SeedJob #2133

See Changes: [lukasz.gajowy] Fixes -- GitHub pull request #5831 of commit ee9b8df78fad0c1235cc1b7a86627283f447dc75, has merge conflicts. Setting status of ee9b8df78fad0c1235c

Beam Dependency Check Report (2018-07-02)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release dill 0.2.6 0.2.8.2 2017-02-01 2018-06-25

Build failed in Jenkins: beam_SeedJob #2132

See Changes: [altay] Add custom sources and sinks for Python page [dhalperi] [BEAM-475] update release instructions for javadoc [dhalperi] Minor tweaks [melissapa] Fix broken links due to code path changes [davor

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #87

See Changes: [github] Return timeout value to 180 [klk] Convert Beam Row to Avatica Row in BeamEnumerableCollector -- [...truncated 1.59 MB...] :beam-sdk