Re: Help measuring upcoming performance increase in flink runner on production systems

2020-12-15 Thread Ahmet Altay
Hi Teodor, Thank you for working on this. If I remember correctly, there were some opportunities to improve in the previous paper (e.g. not focusing deprecated runners, long running benchmarks, varying data sizes). And I am excited that you are keeping the community as part of your research proces

Re: BeamSql query failure on complex data structures

2020-12-15 Thread Kyle Weaver
Hi Aspen, if the issue has been fixed on Calcite's side, it is waiting for an overdue upgrade of the Calcite version Beam uses (tracked by [1]). Currently Beam depends on Calcite 1.20, and it looks like that issue was fixed in Calcite 1.21 [2]. [1] https://issues.apache.org/jira/browse/BEAM-9379 [

Re: BeamSql query failure on complex data structures

2020-12-15 Thread Ahmet Altay
+Andrew Pilloud +Robin Qiu On Tue, Dec 15, 2020 at 12:42 PM Aspen Barnes wrote: > Hello, I have recently encountered an issue with BeamSql queries against > complex data structures. See > https://issues.apache.org/jira/browse/BEAM-11140 for info. Essentially, > the query code generation is fa

Re: PR with a fix stays unchecked(java)(xml)(BEAM-11002)

2020-12-15 Thread Ahmet Altay
I can review this, added a comment in the PR. On Tue, Dec 15, 2020 at 9:51 AM Николай Романов wrote: > Dear beam developers, > The pull request I created hasn't received any comment for 6 days. > I'm sorry for pushing, but it affects the system I maintain: > https://github.com/apache/beam/pull/1

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

2020-12-15 Thread Kyle Weaver
Figured this out. The newly explicit dependency on io.confluent:kafka-schema-registry-client:5.3.2 pulled in a dependency on org.apache.kafka:kafka-clients:5.3.2-ccs, which we didn't want. We fixed it by excluding kafka-clients from kafka-schema-registry-client's dependencies. On Mon, Dec 14, 202

BeamSql query failure on complex data structures

2020-12-15 Thread Aspen Barnes
Hello, I have recently encountered an issue with BeamSql queries against complex data structures. See https://issues.apache.org/jira/browse/BEAM-11140 for info. Essentially, the query code generation is failing past a certain depth, which is not hard to achieve with subqueries and nested data struc

PR with a fix stays unchecked(java)(xml)(BEAM-11002)

2020-12-15 Thread Николай Романов
Dear beam developers, The pull request I created hasn't received any comment for 6 days. I'm sorry for pushing, but it affects the system I maintain: https://github.com/apache/beam/pull/13513 Please let me know I did something wrong when I was creating that PR. Looking forward to getting any notes

Re: Help measuring upcoming performance increase in flink runner on production systems

2020-12-15 Thread Teodor Spæren
Hey! Yeah, that paper was what prompted my master thesis! I definitivly will post here, once I get more data :) Teodor On Mon, Dec 14, 2020 at 06:56:30AM -0600, Rion Williams wrote: Hi Teodor, Although I’m sure you’ve come across it, this might have some valuable resources or methodologies