Re: Java PortableRunner GBK load test fails

Thanks for the reply, I added a link to the Jenkins job in the JIRA, here it is as well: https://builds.apache.org/job/beam_LoadTests_Java_GBK_Flink_Batch_PR/28/console The second stacktrace I pasted there comes from a Flink cluster I used for investigation - I forgot to mention it yesterday. On

Re: Artifact staging in cross-language pipelines

Thanks! I've left some comments on the doc. On Tue, Dec 17, 2019, 5:03 PM Heejong Lee wrote: > Hi, > > I wrote the draft on implementation plan[1]. The summary is on the first > page. Any help would be appreciated! > > [1]: > https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYO

Re: Artifact staging in cross-language pipelines

Hi, I wrote the draft on implementation plan[1]. The summary is on the first page. Any help would be appreciated! [1]: https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog/edit?usp=sharing On Thu, Dec 12, 2019 at 5:02 PM Heejong Lee wrote: > I'm brushing up memory by

Re: Root logger configuration

The generally expected behavior is that if you don't do anything, logging goes to stderr. Logging to non-root loggers breaks this. (Arguably it's a bug in the Python logging libraries to have this inconsistency, but so be it...) On the other hand, if you do set something up, that is respected. I

Re: Request for review of PR [Beam-8564]

Sorry for the long delay (was on vacation). Using org.apache.hadoop isn't part of the Apache Beam Core module but is a dependency for those who depend on the Apache Beam Hadoop module. So I don't think swapping the com.facebook.presto.hadoop version for the org.apache.hadoop version will address I

Re: Root logger configuration

The Python basicConfig[1] sets up a StreamHandler[2], which by default publishes to stderr. This configuration is applied by default whenever someone logs anything on the root logger (e.g. logging.info('abc')). With the module-based logging changes, the basic config is never called on the pipeline

Re: Root logger configuration

In Beam Java, the expectation has always been that pipeline authors are responsible for setting up logging correctly during pipeline construction time and that the Beam SDK is responsible for setting up logging at pipeline execution time. Is this something we can solve by documenting and telling u

Re: Root logger configuration

It should not affect debuggability at pipeline runtime, as the sdk_worker already does the appropriate setup of handlers, but it may affect debugging of e.g. ptransform expansion, and pipeline construction issues. Best -P. On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri wrote: > Pablo, does the issue

Re: Root logger configuration

Pablo, does the issue affect debuggability of pipelines? On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova wrote: > > > On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada wrote: > >> +chad...@gmail.com is this consistent with behavior >> that you observed? >> > > I honestly can't recall, sorry. I just

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 Em sex., 13 de dez. de 2019 às 01:58, Kenneth Knowles escreveu: > Please vote on the proposal for Beam's mascot to be the Firefly. This > encompasses the Lampyridae family of insects, without specifying a genus or > species. > > [ ] +1, Approve Firefly being the mascot > [ ] -1, Disapprove Fi

[VOTE] Release 2.17.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.17.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (non-binding) -Rui On Tue, Dec 17, 2019 at 8:50 AM Luke Cwik wrote: > +1 (binding) > > On Mon, Dec 16, 2019 at 1:18 PM Chamikara Jayalath > wrote: > >> +1 (non-binding) >> >> On Mon, Dec 16, 2019 at 1:12 PM Mark Liu wrote: >> >>> +1 >>> >>> On Mon, Dec 16, 2019 at 11:31 AM Daniel Oliveira

Re: Java PortableRunner GBK load test fails

The connection closing is a red herring as that error gets printed when SDK Harness dies. More logs/jenkins link would be useful to understand what going on with the pipeline. On Tue, Dec 17, 2019 at 5:53 AM Michał Walenia wrote: > Hi there, > I'm trying to add a Jenkins job for a load test of G

Re: [DISCUSS] BIP reloaded

Hi, I feel a "soft consensus" :) that people see some benefits of introducing (possibly optional) process of proposing new features. I think that in order to proceed with this we need to agree on goals that we want to achieve. Whether the process should or should not be optional, which form

Re: [DISCUSS] BIP reloaded

It seems that lots of people see benefit in a more formalized BIP process. I think that makes sense, though I'd like to give people the freedom to choose the medium for their design discussions. The projects I'm aware of usually do this through wiki-type mediums. We have cwiki, though lots of peop

Re: [DISCUSS] BIP reloaded

The main benefit of BIPs I see is the visibility they create for the project users and contributors. Right now, we have a long unordnered list of design documents. Some of the documents are not even in that list. With BIPs, we would end up with an ordered list "BIP-1, BIP-2, .." which reflects

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (binding) On Mon, Dec 16, 2019 at 1:18 PM Chamikara Jayalath wrote: > +1 (non-binding) > > On Mon, Dec 16, 2019 at 1:12 PM Mark Liu wrote: > >> +1 >> >> On Mon, Dec 16, 2019 at 11:31 AM Daniel Oliveira >> wrote: >> >>> +1 (non-binding) >>> >>> On Sat, Dec 14, 2019 at 5:24 PM Kyle Weaver wr

Java PortableRunner GBK load test fails

Hi there, I'm trying to add a Jenkins job for a load test of GBK on portable Flink in Java. I encountered a problem - the test fails with an exception that doesn't say much (Exception in thread "main" java.lang.RuntimeException: Invalid job state: FAILED.) After some investigation, I found where t

Re: [Proposal] Slowly Changing Dimensions and Distributed Map Side Inputs (in Dataflow)

Hi Mikhail, On 12/17/19 10:43 AM, Mikhail Gryzykhin wrote: inline On Tue, Dec 17, 2019 at 12:59 AM Jan Lukavský > wrote: Hi, I actually thought that the proposal refers to Dataflow only. If this is supposed to be general, can we remove the Dataflow/Win

Re: [Proposal] Slowly Changing Dimensions and Distributed Map Side Inputs (in Dataflow)

inline On Tue, Dec 17, 2019 at 12:59 AM Jan Lukavský wrote: > Hi, > > I actually thought that the proposal refers to Dataflow only. If this is > supposed to be general, can we remove the Dataflow/Windmill specific parts > and replace them with generic ones? > I'll look into rephrasing doc to ke

Re: [Proposal] Slowly Changing Dimensions and Distributed Map Side Inputs (in Dataflow)

Hi, I actually thought that the proposal refers to Dataflow only. If this is supposed to be general, can we remove the Dataflow/Windmill specific parts and replace them with generic ones? I'd have two more questions:  a) the proposal is named "Slowly changing", why is the rate of change ess