Re: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

2019-04-25 Thread vino yang
Hi Quan, Thanks for your reply. Actually, I did not try this way. But, there are two factors we should consider: 1. The local state storage is not equals to RocksDB, otherwise Flink does not need to provide a queryable state client. What's more, querying the RocksDB is still an

RE: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

2019-04-25 Thread Shi Quan
Hi, How about take states from RocksDB directly, in this case, TM host is unnecessary. Best Quan Shi From: vino yang Sent: Thursday, April 25, 2019 10:18:20 PM To: dev; user Cc: Stefan Richter; Aljoscha Krettek; kklou...@gmail.com Subject: [DISCUSS] Improve

[jira] [Created] (FLINK-12334) change to MockStreamTask breaks OneInputStreamOperatorTestHarness

2019-04-25 Thread Cliff Resnick (JIRA)
Cliff Resnick created FLINK-12334: - Summary: change to MockStreamTask breaks OneInputStreamOperatorTestHarness Key: FLINK-12334 URL: https://issues.apache.org/jira/browse/FLINK-12334 Project: Flink

[jira] [Created] (FLINK-12333) Add documentation around save point with cancel lifecycle through REST

2019-04-25 Thread Vishal Santoshi (JIRA)
Vishal Santoshi created FLINK-12333: --- Summary: Add documentation around save point with cancel lifecycle through REST Key: FLINK-12333 URL: https://issues.apache.org/jira/browse/FLINK-12333

[DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

2019-04-25 Thread vino yang
Hi all, I want to share my thought with you about improving the queryable state and introducing a QueryServerProxy component. I think the current queryable state's client is hard to use. Because it needs users to know the TaskManager's address and proxy's port. Actually, some business users who

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

2019-04-25 Thread Jeff Zhang
Hi Beckett, Thanks for your feedback, See my comments inline >>> How do user specify the listener? * What I proposal is to register JobListener in ExecutionEnvironment. I don't think we should make ClusterClient as public api. >>> Where should the listener run? * I don't think it is proper to

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

2019-04-25 Thread Becket Qin
Thanks for the proposal, Jeff. Adding a listener to allow users handle events during the job lifecycle makes a lot of sense to me. Here are my two cents. * How do user specify the listener? * It is not quite clear to me whether we consider ClusterClient as a public interface? From what I

[jira] [Created] (FLINK-12332) Cancel running tasks if exist before shutting down TM

2019-04-25 Thread Biao Liu (JIRA)
Biao Liu created FLINK-12332: Summary: Cancel running tasks if exist before shutting down TM Key: FLINK-12332 URL: https://issues.apache.org/jira/browse/FLINK-12332 Project: Flink Issue Type:

[jira] [Created] (FLINK-12331) Introduce partition/gate setup to decouple task registration with NetworkEnvironment

2019-04-25 Thread zhijiang (JIRA)
zhijiang created FLINK-12331: Summary: Introduce partition/gate setup to decouple task registration with NetworkEnvironment Key: FLINK-12331 URL: https://issues.apache.org/jira/browse/FLINK-12331

Re: Introducing Flink's Plugin mechanism

2019-04-25 Thread Piotr Nowojski
Hi Rong, t is definitely do-able to use Plugins for TableFactories discovery, but it would require more extensive changes/discussions. Couple of challenges there: 1. Currently TableFactories come from user job’s jar, which would mean support for dynamically loaded plugins or Pluggable

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

2019-04-25 Thread Robert Bradshaw
On Thu, Apr 25, 2019 at 6:04 AM jincheng sun wrote: > > Hi Robert, > > In addition to the questions described by Dian, I also want to know what > difficult problems Py4j's solution will encounter in add UDF support, which > you mentioned as follows: > >> Using something like Py4j is an easy way

Re: kafka partitions, data locality

2019-04-25 Thread Dawid Wysakowicz
Hi Smirnov, Actually there is a way to tell Flink that data is already partitioned. You can try the reinterpretAsKeyedStream[1] method. I must warn you though this is an experimental feature. Best, Dawid [1]

[jira] [Created] (FLINK-12330) Add integrated Tox for ensuring compatibility with the python2/3 version

2019-04-25 Thread sunjincheng (JIRA)
sunjincheng created FLINK-12330: --- Summary: Add integrated Tox for ensuring compatibility with the python2/3 version Key: FLINK-12330 URL: https://issues.apache.org/jira/browse/FLINK-12330 Project: