Re: Beam Summits!

Hi All, PMC approval still pending for Summit in SF (so things may change), but wanted to get a preliminary CfP out there to start to get sense of interest -- giving the targeted dates are approaching. Much of this delay/uncertainty my fault and I should have done more before the holidays and my

Re: Cross-language pipelines

On Wed, Jan 23, 2019 at 1:03 PM Robert Bradshaw wrote: > On Wed, Jan 23, 2019 at 6:38 PM Maximilian Michels wrote: > > > > Thank you for starting on the cross-language feature Robert! > > > > Just to recap: Each SDK runs an ExpansionService which can be contacted > during > > pipeline

Re: Enforce javadoc comments in public methods?

Our recent change is on "JavaDocMethod", which not turned on yet. Not relevant to this error here. The one throws error is "javaDocType". it has been there for a while , which is for

Precomit broken due to style violation. Why are failures getting past precommit?

See: https://issues.apache.org/jira/browse/BEAM-6500 I think that this PR introduced the issue. Though I am not sure how to read the test status. It looks like its marked with an X for the postcommit status, but presumably the precommit was okay even though java precommit appears to be broken

Re: Enforce javadoc comments in public methods?

Did their happen to be a short time window where some missing Javadoc comments went in? I am now seeing precommit fail due to code I didn't modify. https://scans.gradle.com/s/nwgb7xegklwqo/console-log?task=:beam-runners-direct-java:checkstyleMain On Wed, Jan 23, 2019 at 2:34 PM Ruoyun Huang

Re: [PROPOSAL] Prepare Beam 2.10.0 release

Python precommit cron job ( https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Python_Cron/) seems to be healthy. Could you share one of the PRs that you noticed the flakiness, I would be interested in debugging. On Wed, Jan 23, 2019 at 12:25 PM Kenneth Knowles

Re: ContainerLaunchException in precommit [BEAM-6497]

Thank you Gleb, appreciate it. On Wed, Jan 23, 2019 at 2:40 PM Gleb Kanterov wrote: > I'm looking into it. This image exists in docker hub [1], but for some > reason, it wasn't picked up. > > [1] https://hub.docker.com/r/yandex/clickhouse-server/tags > > On Wed, Jan 23, 2019 at 10:01 PM Alex

Re: [DISCUSSION] ParDo Async Java API

> > I'd love to see something like this as well. Also +1 to process(@Element > InputT element, @Output OutputReceiver>). I > don't know if there's much benefit to passing a future in, since the > framework itself could hook up the process function to complete when the > future completes. > One

Re: ContainerLaunchException in precommit [BEAM-6497]

I'm looking into it. This image exists in docker hub [1], but for some reason, it wasn't picked up. [1] https://hub.docker.com/r/yandex/clickhouse-server/tags On Wed, Jan 23, 2019 at 10:01 PM Alex Amato wrote: > >1. > See: BEAM-6497 >

Re: Enforce javadoc comments in public methods?

Trying to understand your suggestion. By saying "break that dependency", do you mean moving checkstyle out of Java PreCommit? currently we do have checkstyle as part of ":check". It seems to me "check" does minimal amount of essential works (correct me If I am wrong), much less than what

Re: Confusing sentence in Windowing section in Beam programming guide

Thanks all! The documentation has been updated. -Rui On Tue, Jan 22, 2019 at 12:24 PM Reuven Lax wrote: > Ah yes, Kenn is correct, and i forget we made that change. > > To clarify - Beam does not expose late elements as a concept, rather it > exposes late panes on its triggering API. The

Re: Our jenkins beam1 server is down

Hi, seems like both Beam4[1] and Beam9[2] are down. -Rui [1]: https://builds.apache.org/computer/beam4/ [2]: https://builds.apache.org/computer/beam9/ On Wed, Jan 23, 2019 at 8:15 AM Yifan Zou wrote: > Looking. The following errors happened consistently. > > Jan 23 16:05:55

Re: Confluence wiki edit access request

Thank you! On Wed, Jan 23, 2019 at 12:49 PM Ismaël Mejía wrote: > Done. > > On Tue, Jan 22, 2019 at 8:53 PM Udi Meiri wrote: > > > > bump > > > > On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: > >> > >> username: udim > >> > >> Thanks! > smime.p7s Description: S/MIME Cryptographic

Re: Cross-language pipelines

On Wed, Jan 23, 2019 at 6:38 PM Maximilian Michels wrote: > > Thank you for starting on the cross-language feature Robert! > > Just to recap: Each SDK runs an ExpansionService which can be contacted during > pipeline translation to expand transforms that are unknown to the SDK. The > service

ContainerLaunchException in precommit [BEAM-6497]

1. See: BEAM-6497 1. This is also causing issues blocking precommits. 2. Seems to be caused by this failure to locate the image. Are these stored somewhere or built by the build process? Any idea why these are

Re: Confluence wiki edit access request

Done. On Tue, Jan 22, 2019 at 8:53 PM Udi Meiri wrote: > > bump > > On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: >> >> username: udim >> >> Thanks!

Re: [PROPOSAL] Prepare Beam 2.10.0 release

I've seen many PRs that I am reviewing with flakiness problems in the python precommit IT. Anyone have any insights? Kenn On Wed, Jan 23, 2019 at 12:23 PM Kenneth Knowles wrote: > Thanks for the reminder. I was reading from > https://issues.apache.org/jira/projects/BEAM/versions/12344540. RC1

Re: [PROPOSAL] Prepare Beam 2.10.0 release

Thanks for the reminder. I was reading from https://issues.apache.org/jira/projects/BEAM/versions/12344540. RC1 also gated on clearing https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.10.0 . Kenn On Wed, Jan 23, 2019 at 9:44 AM Maximilian Michels wrote: >

Re: compileJava broken on master see: BEAM-6495

This might connect to vendoring Calcite. It will be easiest, and have the best incremental build, if we separate the generated code into its own module that has relocation to match the vendored Calcite. Kenn On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin wrote: > We don't pre-generate the code

Re: Enforce javadoc comments in public methods?

It is always a bummer when the Java PreCommit fails due to style checking. Can we get this to run separately and quicker? I notice it depends on compileJava. I cannot remember why that is, but I recall it is a legitimate reason. Nonetheless, can we break that dependency somehow? Kenn On Wed, Jan

Re: [DISCUSSION] ParDo Async Java API

I think your concerns are valid but i want to clarify about "first class async APIs". Does "first class" mean that it is a well-encapsulated abstraction? or does it mean that the user can more or less do whatever they want? These are opposite but both valid meanings for "first class", to me. I

Re: compileJava broken on master see: BEAM-6495

We don't pre-generate the code as a separate step. Code gen from the SQL parser syntax spec and its compilation happens both during the Beam SQL build task. Splitting the code generation and compilation might not be trivial. We definitely should look into fixing this though. Regards, Anton On

Re: compileJava broken on master see: BEAM-6495

https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/build.gradle#L95 After a glance on sql build code, seems like it does some file copy operations and then compile. It might be the source of race condition. -Rui On Wed, Jan 23, 2019 at 11:17 AM Ruoyun Huang wrote: > I ran

Re: compileJava broken on master see: BEAM-6495

On Wed, Jan 23, 2019 at 11:13 AM Alex Amato wrote: > Okay, make sense perhaps we can somehow make it fail when it fails to > generate the dep, rather than when compiling the java code later on > That would be a good improvement on error message. :-) Or, does it make sense to manually check-in

Re: compileJava broken on master see: BEAM-6495

Okay, make sense perhaps we can somehow make it fail when it fails to generate the dep, rather than when compiling the java code later on On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin wrote: > ParserImpl is autogenerated by Calcite at build time. It seems that > there's a race condition there

Re: compileJava broken on master see: BEAM-6495

This kind of dependency can cause a flakey test? Seems like this kind of thing would be a consistent failure, at least that's what I am seeing as well On Wed, Jan 23, 2019 at 11:11 AM Ruoyun Huang wrote: > I ran into the same issue. it is flaky, due to some depended package. > > short term

Re: compileJava broken on master see: BEAM-6495

ParserImpl is autogenerated by Calcite at build time. It seems that there's a race condition there and it sometimes fails. Rerunning the build works for me. Regards, Anton On Wed, Jan 23, 2019, 11:06 AM Alex Amato wrote: > https://jira.apache.org/jira/browse/BEAM-6495?filter=-2 > > Any ideas,

Re: compileJava broken on master see: BEAM-6495

I ran into the same issue. it is flaky, due to some depended package. short term solution is to rerun precommit. For me it was gone on second try. On Wed, Jan 23, 2019 at 11:06 AM Alex Amato wrote: > https://jira.apache.org/jira/browse/BEAM-6495?filter=-2 > > Any ideas, how this got through

Re: [DISCUSSION] ParDo Async Java API

A related question is how to make execution observable such that a runner can make proper scaling decisions. Runners decide how to schedule bundles within and across multiple worker instances, and can use information about execution to make dynamic scaling decisions. First-class async APIs seem

Re: Dealing with expensive jenkins + Dataflow jobs

Agreeing with Robert about "what is it we're trying to test?". Would a smaller performance test find the same issues, faster and more reliably? We have seen issues with the apache-beam-testing project exceeding quota during dataflow jobs, resulting in spurious failures during precommits and

Re: [PROPOSAL] Prepare Beam 2.10.0 release

What about the revert of "Parse SDK-unknown pipeline options"? https://github.com/apache/beam/pull/7564 Should we merge this for the release? On 23.01.19 11:56, Scott Wegner wrote: Cherry-pick PR for the last-remaining issue: https://github.com/apache/beam/pull/7603 On Wed, Jan 23, 2019 at

Re: Cross-language pipelines

Thank you for starting on the cross-language feature Robert! Just to recap: Each SDK runs an ExpansionService which can be contacted during pipeline translation to expand transforms that are unknown to the SDK. The service returns the Proto definitions to the querying process. There will be

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Thanks for the review, Alex. I pushed an additional commit to add commentary and to explicitly pass a copy option that indicates we want to preserve attributes like timestamps. On Wed, Jan 23, 2019 at 12:23 PM Alex Amato wrote: > Thanks Jeff, I reviewed your PR with one suggestion to add a

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Thanks Jeff, I reviewed your PR with one suggestion to add a comment to make the test more clear. I am assuming the modified times get copied, not re-timestamped on copy, which is why your method works. Otherwise looks good to me On Wed, Jan 23, 2019 at 5:49 AM Jeff Klukas wrote: > Posted

Re: [PROPOSAL] Prepare Beam 2.10.0 release

Cherry-pick PR for the last-remaining issue: https://github.com/apache/beam/pull/7603 On Wed, Jan 23, 2019 at 7:15 AM Kenneth Knowles wrote: > The last remaining issue is split into a non-blocker and a revert that is > confirmed to fix the issue. > > Once

Re: Our jenkins beam1 server is down

Looking. The following errors happened consistently. Jan 23 16:05:55 apache-beam-jenkins-slave-group-51fn systemd[1]: Started Session 72 of user jenkins. Jan 23 16:06:03 apache-beam-jenkins-slave-group-51fn snmpd[16379]: error on subcontainer 'ia_addr' insert (-1) Jan 23 16:08:33

Re: [spark runner based on dataset POC] your opinion

Hi all ,Thanks for your feedback! I was indeed thinking about Reuven's work around Schema PCollections, hence my email to the community. I don't see how it fits considering that, as I'm wrapping a source, I need to store both the timestamp and the value hence the use of WindowedValue (as the

Re: Our jenkins beam1 server is down

Looks like beam9 is now gone. On Tue, Jan 22, 2019 at 8:57 PM Yifan Zou wrote: > > The inventory test on the beam1 passed. The beam1 is back to normal. > https://builds.apache.org/job/beam_Inventory_beam1/303/ > > On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou wrote: >> >> Thanks for reporting the

Re: [PROPOSAL] Prepare Beam 2.10.0 release

The last remaining issue is split into a non-blocker and a revert that is confirmed to fix the issue. Once https://github.com/apache/beam/pull/7600 is merged and cherry-picked, I will cut RC1 today. Kenn On Tue, Jan 22, 2019 at 6:03 PM Kenneth Knowles wrote: > OK. There is just one release

Re: Dealing with expensive jenkins + Dataflow jobs

I like the idea of creating separate project(s) for load tests so as to not compete with other tests and the standard development cycle. As for how many workers is too many, I would take the track "what is it we're trying to test?" Unless your stress-testing the shuffle itself, much of what Beam

Re: Dealing with expensive jenkins + Dataflow jobs

Hi, pinging this thread (maybe some folks missed it). What do you think about those concerns/ideas? Łukasz pon., 14 sty 2019 o 17:11 Łukasz Gajowy napisał(a): > Hi all, > > one problem we need to solve while working with load tests we currently > develop is that we don't really know how much

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Posted https://github.com/apache/beam/pull/7599 That PR follows suggestion #4. I chose that route because it maintains the PAssert containsInAnyOrder check which seems easier to read and more straight-forward than PAssert satisfies. Do let me know if you disagree and I can switch back to

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Suggestion #4: Create source files outside the writer thread, and then copy them from a source directory to the watched directory. That should atomically write the file with the already known lastModificationTime. On Wed, Jan 23, 2019 at 7:37 AM Jeff Klukas wrote: > I'll work on getting a PR

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

I'll work on getting a PR together this morning, probably following Eugene's suggestion #1. On Tue, Jan 22, 2019 at 8:34 PM Udi Meiri wrote: > Alex, the only way to implement my suggestion #1 (that I know of) would be > to write to a file and read it back. > I don't have good example for #2. >

Re: How to use "PortableRunner" in Python SDK?

We should probably make the job endpoint mandatory for PortableRunner, and offer a separate FlinkRunner (and others) that provides a default endpoint and otherwise delegates everything down. On Thu, Nov 15, 2018 at 12:07 PM Maximilian Michels wrote: > > > 1) The default behavior, where

Re: [PROPOSAL] allow the users to anticipate the support of features in the targeted runner.

Hi thanks for the proposal and not abandoning this thread. This topic is very important. I left some comments. Thanks, Łukasz śr., 23 sty 2019 o 10:00 Etienne Chauchot napisał(a): > HI guys, > > As part of our user growth, I'd like to revive this subject. > I have sketched up a 2 pages

Re: Cross-language pipelines

No, this PR simply takes an endpoint address as a parameter, expecting it to already be up and available. More convenient APIs, e.g. ones that spin up and endpoint and tear it down, or catalog and locate code and services offering these endpoints, could be provided as wrappers on top of or

Re: [PROPOSAL] allow the users to anticipate the support of features in the targeted runner.

HI guys, As part of our user growth, I'd like to revive this subject.I have sketched up a 2 pages proposal on this: https://docs.google.com/document/d/1eXt54ht0h7-pPbP-MJR0N5nzmxRRlAwbFod-LXI1x0A/edit?usp=sharing Unfortunately I have no knowledge on IDE plugin developement. Does someone have