Re: [DISCUSS] Detection Flink Backpressure

2019-01-07 Thread zhijiang
Hi all, The current thread strace sample can not cover all the cases. And the output queue length might not be very accurate sometimes, because the last BufferConsumer in each subpartition might not be fulfilled and still be writable although the outqueu length is already equal or more (conside

Re: [DISCUSS] Support Interactive Programming in Flink Table API

2019-01-07 Thread Becket Qin
Happy New Year, everybody! I would like to resume this discussion thread. At this point, We have agreed on the first step goal of interactive programming. The open discussion is the exact API. More specifically, what should *cache()* method return and what is the semantic. There are three options:

[jira] [Created] (FLINK-11272) Support for parsing multiple --yarnship parameters

2019-01-07 Thread leesf (JIRA)
leesf created FLINK-11272: - Summary: Support for parsing multiple --yarnship parameters Key: FLINK-11272 URL: https://issues.apache.org/jira/browse/FLINK-11272 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2019-01-07 Thread Timo Walther
Hi Eron, thank you very much for the contributions. I merged the first little bug fixes. For the remaining PRs I think we can review and merge them soon. As you said, the code is agnostic to the details of the ExternalCatalog interface and I don't expect bigger merge conflicts in the near futu

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2019-01-07 Thread Timo Walther
Hi everyone, Xuefu and I had multiple iterations over the catalog design document [1]. I believe that it is in a good shape now to be converted into FLIP. Maybe we need a bit more explanation at some places but the general design would be ready now. The design document covers the following c

[jira] [Created] (FLINK-11273) Queryable state (rocksdb) end-to-end test fails

2019-01-07 Thread Timo Walther (JIRA)
Timo Walther created FLINK-11273: Summary: Queryable state (rocksdb) end-to-end test fails Key: FLINK-11273 URL: https://issues.apache.org/jira/browse/FLINK-11273 Project: Flink Issue Type: B

Re: [DISCUSS] Support Interactive Programming in Flink Table API

2019-01-07 Thread Piotr Nowojski
Hi Becket! After further thinking I tend to agree that my previous proposal (*Option 2*) indeed might not be if would in the future introduce automatic caching. However I would like to propose a slightly modified version of it: *Option 4* Adding `cache()` method with following signature: Tabl

[jira] [Created] (FLINK-11274) Scala 2.12 Kryo serialization bug

2019-01-07 Thread Zhenhao Li (JIRA)
Zhenhao Li created FLINK-11274: -- Summary: Scala 2.12 Kryo serialization bug Key: FLINK-11274 URL: https://issues.apache.org/jira/browse/FLINK-11274 Project: Flink Issue Type: Bug Affects Ver

[jira] [Created] (FLINK-11275) Unified Catalog APIs

2019-01-07 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created FLINK-11275: --- Summary: Unified Catalog APIs Key: FLINK-11275 URL: https://issues.apache.org/jira/browse/FLINK-11275 Project: Flink Issue Type: Improvement Componen

[jira] [Created] (FLINK-11276) Slicing Window Optimization

2019-01-07 Thread Rong Rong (JIRA)
Rong Rong created FLINK-11276: - Summary: Slicing Window Optimization Key: FLINK-11276 URL: https://issues.apache.org/jira/browse/FLINK-11276 Project: Flink Issue Type: New Feature Rep

[jira] [Created] (FLINK-11277) Queryable state IT test failed

2019-01-07 Thread vinoyang (JIRA)
vinoyang created FLINK-11277: Summary: Queryable state IT test failed Key: FLINK-11277 URL: https://issues.apache.org/jira/browse/FLINK-11277 Project: Flink Issue Type: New Feature Comp

Re: [DISCUSS] Support Interactive Programming in Flink Table API

2019-01-07 Thread Becket Qin
Hi Piotr, Thanks for the proposal and detailed explanation. I like the idea of returning a new hinted Table without modifying the original table. This also leave the room for users to benefit from future implicit caching. Just to make sure I get the full picture. In your proposal, there will also

[jira] [Created] (FLINK-11278) Add documentation for TableAPI&SQL in scala-shell

2019-01-07 Thread sunjincheng (JIRA)
sunjincheng created FLINK-11278: --- Summary: Add documentation for TableAPI&SQL in scala-shell Key: FLINK-11278 URL: https://issues.apache.org/jira/browse/FLINK-11278 Project: Flink Issue Type: I

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2019-01-07 Thread Zhang, Xuefu
Thanks, Timo! I have started put the content from the google doc to FLIP-30 [1]. However, please still keep the discussion along this thread. Thanks, Xuefu [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs --

[jira] [Created] (FLINK-11279) The bug of Error parsing ExpressionParser

2019-01-07 Thread xuqianjin (JIRA)
xuqianjin created FLINK-11279: - Summary: The bug of Error parsing ExpressionParser Key: FLINK-11279 URL: https://issues.apache.org/jira/browse/FLINK-11279 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-11280) RocksDBSerializedCompositeKeyBuilder's key serializer is not being reconfigured properly

2019-01-07 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-11280: --- Summary: RocksDBSerializedCompositeKeyBuilder's key serializer is not being reconfigured properly Key: FLINK-11280 URL: https://issues.apache.org/jira/browse/FLINK-1

[jira] [Created] (FLINK-11281) Two test in StateBackendMigrationTestBase failes

2019-01-07 Thread BoWang (JIRA)
BoWang created FLINK-11281: -- Summary: Two test in StateBackendMigrationTestBase failes Key: FLINK-11281 URL: https://issues.apache.org/jira/browse/FLINK-11281 Project: Flink Issue Type: Bug