[jira] [Created] (FLINK-12308) Support python language in Flink Table API

2019-04-23 Thread sunjincheng (JIRA)
sunjincheng created FLINK-12308: --- Summary: Support python language in Flink Table API Key: FLINK-12308 URL: https://issues.apache.org/jira/browse/FLINK-12308 Project: Flink Issue Type: New

[jira] [Created] (FLINK-12307) Support translation from StreamExecWindowJoin to StreamTransformation.

2019-04-23 Thread Jing Zhang (JIRA)
Jing Zhang created FLINK-12307: -- Summary: Support translation from StreamExecWindowJoin to StreamTransformation. Key: FLINK-12307 URL: https://issues.apache.org/jira/browse/FLINK-12307 Project: Flink

Re: [DISCUSS] Proposal of Flink Driver - Making Flink More Interactive

2019-04-23 Thread Shuiqiang Chen
Hi Till, Thanks for your insightful and valuable comments! The introduction of drive dispatcher is the functionality extension of current existing dispatchers, and has somewhat change to the runtime. It is mainly to manage the ephemeral session cluster lifecycle when user detachedly submits an

[jira] [Created] (FLINK-12306) Change the name of variable "log" to Upper case "LOG"

2019-04-23 Thread Kejian Li (JIRA)
Kejian Li created FLINK-12306: - Summary: Change the name of variable "log" to Upper case "LOG" Key: FLINK-12306 URL: https://issues.apache.org/jira/browse/FLINK-12306 Project: Flink Issue Type:

[jira] [Created] (FLINK-12305) Table API Clarification

2019-04-23 Thread Alex Barnes (JIRA)
Alex Barnes created FLINK-12305: --- Summary: Table API Clarification Key: FLINK-12305 URL: https://issues.apache.org/jira/browse/FLINK-12305 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

2019-04-23 Thread Stephan Ewen
Hi all! Below are my notes on the discussion last week on how to collaborate between Beam and Flink. The discussion was between Tyler, Kenn, Luke, Ahmed, Xiaowei, Shaoxuan, Jincheng, and me. This represents my understanding of the discussion, please augment this where I missed something or where

[DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

2019-04-23 Thread Gary Yao
Hi all, As the subject states, I am proposing to temporarily remove support for changing the parallelism of a job via the following syntax [1]: ./bin/flink modify [job-id] -p [new-parallelism] This is an experimental feature that we introduced with the first rollout of FLIP-6 (Flink 1.5).

[jira] [Created] (FLINK-12304) AvroInputFormat should support schema evolution

2019-04-23 Thread John (JIRA)
John created FLINK-12304: Summary: AvroInputFormat should support schema evolution Key: FLINK-12304 URL: https://issues.apache.org/jira/browse/FLINK-12304 Project: Flink Issue Type: Bug

Re: HA lock nodes, Checkpoints, and JobGraphs after failure

2019-04-23 Thread Till Rohrmann
It would be awesome to get the DEBUG logs for JobMaster, ZooKeeper, ZooKeeperCompletedCheckpointStore, ZooKeeperStateHandleStore, CheckpointCoordinator. Cheers, Till On Tue, Apr 23, 2019 at 2:37 PM Dyana Rose wrote: > may take me a bit to get the logs as we're not always in a situation where >

Apache Flink survey by Ververica

2019-04-23 Thread Robert Metzger
Hi everyone! Ververica is running a brief survey to understand Apache Flink usage and the needs of the community. We are hoping that this survey will help identify common usage patterns, as well as pinpoint what are the most needed features for Flink. We'll share a report with a summary of

Re: HA lock nodes, Checkpoints, and JobGraphs after failure

2019-04-23 Thread Dyana Rose
may take me a bit to get the logs as we're not always in a situation where we've got enough hands free to run through the scenarios for a day. Is that DEBUG JobManager, DEBUG ZooKeeper, or both you'd be interested in? Thanks, Dyana On Tue, 23 Apr 2019 at 13:23, Till Rohrmann wrote: > Hi

Re: HA lock nodes, Checkpoints, and JobGraphs after failure

2019-04-23 Thread Till Rohrmann
Hi Dyana, your analysis is almost correct. The only part which is missing is that the lock nodes are created as ephemeral nodes. This should ensure that if a JM process dies that the lock nodes will get removed by ZooKeeper. It depends a bit on ZooKeeper's configuration how long it takes until Zk

Re: [DISCUSS] Proposal of Flink Driver - Making Flink More Interactive

2019-04-23 Thread Till Rohrmann
Thanks for proposing this design document Shuiqiang. It is a very interesting idea how to solve the problem of running multiple Flink jobs as part of a single application. I like the idea since it does not require many runtime changes apart from a session concept on the Dispatcher and it would

[jira] [Created] (FLINK-12303) Scala 2.12 lambdas does not work in event classes inside streams.

2019-04-23 Thread JIRA
Matěj Novotný created FLINK-12303: - Summary: Scala 2.12 lambdas does not work in event classes inside streams. Key: FLINK-12303 URL: https://issues.apache.org/jira/browse/FLINK-12303 Project: Flink

[jira] [Created] (FLINK-12302) fix the finalStatus of application when job not finished

2019-04-23 Thread lamber-ken (JIRA)
lamber-ken created FLINK-12302: -- Summary: fix the finalStatus of application when job not finished Key: FLINK-12302 URL: https://issues.apache.org/jira/browse/FLINK-12302 Project: Flink Issue

[jira] [Created] (FLINK-12301) Scala value classes cannot be serialized anymore in case classes in Flink 1.8.0

2019-04-23 Thread Michael (JIRA)
Michael created FLINK-12301: --- Summary: Scala value classes cannot be serialized anymore in case classes in Flink 1.8.0 Key: FLINK-12301 URL: https://issues.apache.org/jira/browse/FLINK-12301 Project: Flink

[jira] [Created] (FLINK-12300) Unused Import warning with DataStream[Seq[...]]

2019-04-23 Thread Michael (JIRA)
Michael created FLINK-12300: --- Summary: Unused Import warning with DataStream[Seq[...]] Key: FLINK-12300 URL: https://issues.apache.org/jira/browse/FLINK-12300 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-12299) ExecutionConfig#setAutoWatermarkInterval should check param(interval should not less than zero)

2019-04-23 Thread shiwuliang (JIRA)
shiwuliang created FLINK-12299: -- Summary: ExecutionConfig#setAutoWatermarkInterval should check param(interval should not less than zero) Key: FLINK-12299 URL: https://issues.apache.org/jira/browse/FLINK-12299

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

2019-04-23 Thread Till Rohrmann
I think we should not expose the ClusterClient configuration via the ExecutionEnvironment (env.getClusterClient().addJobListener) because this is effectively the same as exposing the JobListener interface directly on the ExecutionEnvironment. Instead I think it could be possible to provide a

[jira] [Created] (FLINK-12298) Make column functions accept custom

2019-04-23 Thread Dawid Wysakowicz (JIRA)
Dawid Wysakowicz created FLINK-12298: Summary: Make column functions accept custom Key: FLINK-12298 URL: https://issues.apache.org/jira/browse/FLINK-12298 Project: Flink Issue Type:

[DISCUSS] Proposal of Flink Driver - Making Flink More Interactive

2019-04-23 Thread Shuiqiang Chen
Hi All, We would like to start a discussion thread about a new feature called Flink Driver. A brief summary is following. As mentioned in the discussion of Interactive Programming, user applications might consist of multiple jobs and take long to finish. Currently, when Flink runs applications

Re: Support for ProtoBuf data format in StreamingFileSink

2019-04-23 Thread Driesprong, Fokko
Thanks Kailash for bringing this up. I think this is a good idea. By passing the ParquetWriter we gain much more flexibility. I did a small PR on adding the ability to add compression to the Parquet writer: https://github.com/apache/flink/pull/7547 But I believe this is the wrong approach. For

[jira] [Created] (FLINK-12297) We should clean the closure for OutputTags

2019-04-23 Thread Dawid Wysakowicz (JIRA)
Dawid Wysakowicz created FLINK-12297: Summary: We should clean the closure for OutputTags Key: FLINK-12297 URL: https://issues.apache.org/jira/browse/FLINK-12297 Project: Flink Issue

[jira] [Created] (FLINK-12296) Data loss silently in RocksDBStateBackend when more than one operator chained in a single task

2019-04-23 Thread Congxian Qiu(klion26) (JIRA)
Congxian Qiu(klion26) created FLINK-12296: - Summary: Data loss silently in RocksDBStateBackend when more than one operator chained in a single task Key: FLINK-12296 URL:

[jira] [Created] (FLINK-12295) use retract aggregate function instead of regular aggregate function for retractable aggregate in code gen

2019-04-23 Thread godfrey he (JIRA)
godfrey he created FLINK-12295: -- Summary: use retract aggregate function instead of regular aggregate function for retractable aggregate in code gen Key: FLINK-12295 URL:

[jira] [Created] (FLINK-12294) kafka consumer, data locality

2019-04-23 Thread Sergey (JIRA)
Sergey created FLINK-12294: -- Summary: kafka consumer, data locality Key: FLINK-12294 URL: https://issues.apache.org/jira/browse/FLINK-12294 Project: Flink Issue Type: Improvement