[jira] [Created] (FLINK-14735) Improve batch schedule performance

2019-11-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14735: Summary: Improve batch schedule performance Key: FLINK-14735 URL: https://issues.apache.org/jira/browse/FLINK-14735 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-14734) Add a ResourceSpec in SlotSharingGroup to describe its overall resources

2019-11-12 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14734: --- Summary: Add a ResourceSpec in SlotSharingGroup to describe its overall resources Key: FLINK-14734 URL: https://issues.apache.org/jira/browse/FLINK-14734 Project: Flink

[jira] [Created] (FLINK-14733) Introduce ResourceProfile builder to enable flexible building

2019-11-12 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14733: --- Summary: Introduce ResourceProfile builder to enable flexible building Key: FLINK-14733 URL: https://issues.apache.org/jira/browse/FLINK-14733 Project: Flink Issue

[jira] [Created] (FLINK-14732) flink-release-1.9.x can NOT compile with CDH 5.16

2019-11-12 Thread Chen Shuai (Jira)
Chen Shuai created FLINK-14732: -- Summary: flink-release-1.9.x can NOT compile with CDH 5.16 Key: FLINK-14732 URL: https://issues.apache.org/jira/browse/FLINK-14732 Project: Flink Issue Type:

[jira] [Created] (FLINK-14731) LogicalWatermarkAssigner should use specified trait set when doing copy

2019-11-12 Thread Liya Fan (Jira)
Liya Fan created FLINK-14731: Summary: LogicalWatermarkAssigner should use specified trait set when doing copy Key: FLINK-14731 URL: https://issues.apache.org/jira/browse/FLINK-14731 Project: Flink

[jira] [Created] (FLINK-14730) Add pending slots for job

2019-11-12 Thread lining (Jira)
lining created FLINK-14730: -- Summary: Add pending slots for job Key: FLINK-14730 URL: https://issues.apache.org/jira/browse/FLINK-14730 Project: Flink Issue Type: Improvement Components:

[jira] [Created] (FLINK-14729) Multi-topics consuming from KafkaTableSource

2019-11-12 Thread Leo Zhang (Jira)
Leo Zhang created FLINK-14729: - Summary: Multi-topics consuming from KafkaTableSource Key: FLINK-14729 URL: https://issues.apache.org/jira/browse/FLINK-14729 Project: Flink Issue Type: New

[jira] [Created] (FLINK-14728) add reminder for users of potential thread safety issues of hive built-in function

2019-11-12 Thread Bowen Li (Jira)
Bowen Li created FLINK-14728: Summary: add reminder for users of potential thread safety issues of hive built-in function Key: FLINK-14728 URL: https://issues.apache.org/jira/browse/FLINK-14728 Project:

[jira] [Created] (FLINK-14727) update doc of supported Hive versions

2019-11-12 Thread Bowen Li (Jira)
Bowen Li created FLINK-14727: Summary: update doc of supported Hive versions Key: FLINK-14727 URL: https://issues.apache.org/jira/browse/FLINK-14727 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14726) avoid creating new HiveShim instance every time when calling HiveShimLoader.loadHiveShim()

2019-11-12 Thread Bowen Li (Jira)
Bowen Li created FLINK-14726: Summary: avoid creating new HiveShim instance every time when calling HiveShimLoader.loadHiveShim() Key: FLINK-14726 URL: https://issues.apache.org/jira/browse/FLINK-14726

Re: Flink Read thousands of files with batch

2019-11-12 Thread Dominik Wosiński
I have managed to locate the issue with timeout, changing `web.timeout` was the solution. However, now I am getting the following error : 019-11-12 16:58:00,741 INFO org.apache.parquet.hadoop.ParquetInputFormat - Total input paths to process : 671 2019-11-12 16:58:04,878 INFO

[jira] [Created] (FLINK-14725) Remove unused (anymore) TaskManagerOptions#EXIT_ON_FATAL_AKKA_ERROR

2019-11-12 Thread Yun Tang (Jira)
Yun Tang created FLINK-14725: Summary: Remove unused (anymore) TaskManagerOptions#EXIT_ON_FATAL_AKKA_ERROR Key: FLINK-14725 URL: https://issues.apache.org/jira/browse/FLINK-14725 Project: Flink

Re: Flink Read thousands of files with batch

2019-11-12 Thread Dominik Wosiński
Hey Jingsong, I will try to use the patch to verify. In the meantime, I have run the job with -D akka.ask.timeout and -D akka.client.timeout, both equal to 600s. But the stacktrace is the same : org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result.

[jira] [Created] (FLINK-14724) Join condition could be simplified in logical phase

2019-11-12 Thread godfrey he (Jira)
godfrey he created FLINK-14724: -- Summary: Join condition could be simplified in logical phase Key: FLINK-14724 URL: https://issues.apache.org/jira/browse/FLINK-14724 Project: Flink Issue Type:

[jira] [Created] (FLINK-14723) Improve some code about computed column in planner

2019-11-12 Thread Jark Wu (Jira)
Jark Wu created FLINK-14723: --- Summary: Improve some code about computed column in planner Key: FLINK-14723 URL: https://issues.apache.org/jira/browse/FLINK-14723 Project: Flink Issue Type:

Re: Flink Read thousands of files with batch

2019-11-12 Thread Jingsong Li
Hi Dominik: I found a problem too that it maybe your root cause.[1] JobConf in HadoopInputSplit may very big, contains hundreds of configurations, if it is serialized by every split, that will significantly reduce performance. Consider thousands of splits, the akka thread of JobMaster will all on

[jira] [Created] (FLINK-14722) Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable

2019-11-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-14722: Summary: Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable Key: FLINK-14722 URL: https://issues.apache.org/jira/browse/FLINK-14722

Re: Flink Read thousands of files with batch

2019-11-12 Thread Dominik Wosiński
Hey, I have increased the `akka.client.timeout` but it has not helped at all. Here is the log with callstack for AskTimeoutException: 019-11-12 10:19:17,425 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received JobGraph submission 81fbbc3f41ad5e08ac832d0e656478bc (Flink

Re: [DISCUSS] Stateful Functions - Contribution Details

2019-11-12 Thread Stephan Ewen
As mentioned before, the name was mainly chosen to resonate with developers form a different background (applications / services) and we checked it with some users. Unrelated to Flink and Stream Processing, it seemed to describe the target use case pretty well. What would you use as a name

[jira] [Created] (FLINK-14721) HiveTableSource should implement LimitableTableSource interface

2019-11-12 Thread Terry Wang (Jira)
Terry Wang created FLINK-14721: -- Summary: HiveTableSource should implement LimitableTableSource interface Key: FLINK-14721 URL: https://issues.apache.org/jira/browse/FLINK-14721 Project: Flink

[jira] [Created] (FLINK-14720) Bring down ExecutionVertex#deployToSlot access modifier and mark it with @VisibleForTesting annotation

2019-11-12 Thread vinoyang (Jira)
vinoyang created FLINK-14720: Summary: Bring down ExecutionVertex#deployToSlot access modifier and mark it with @VisibleForTesting annotation Key: FLINK-14720 URL: https://issues.apache.org/jira/browse/FLINK-14720

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-12 Thread tison
+1 (binding) Best, tison. Aljoscha Krettek 于2019年11月12日周二 下午4:37写道: > +1 (binding) > > > On 8. Nov 2019, at 09:45, Danny Chan wrote: > > > > Thanks for driving on this, David. > > > > +1(non-binding) from my side, we really need this feature ! > > > > Best, > > Danny Chan > > 在 2019年11月7日

Re: [DISCUSS] Stateful Functions - Contribution Details

2019-11-12 Thread Chesnay Schepler
I'm concerned both about the abbreviation and full name. a) It's not distinguishing enough from existing APIs, specifically the Streaming API, which already features stateful functions. b) It doesn't describe use-cases that the existing APIs cannot satisfy. On 11/11/2019 15:28, Stephan Ewen

[jira] [Created] (FLINK-14719) Making Semantic configurable in Flinkkafkaproducer to support exactly-once semantic in Table API

2019-11-12 Thread chaiyongqiang (Jira)
chaiyongqiang created FLINK-14719: - Summary: Making Semantic configurable in Flinkkafkaproducer to support exactly-once semantic in Table API Key: FLINK-14719 URL:

Re: [VOTE] FLIP-59: Enable execution configuration from Configuration object

2019-11-12 Thread Aljoscha Krettek
+1 (binding) > On 8. Nov 2019, at 09:45, Danny Chan wrote: > > Thanks for driving on this, David. > > +1(non-binding) from my side, we really need this feature ! > > Best, > Danny Chan > 在 2019年11月7日 +0800 PM5:04,Dawid Wysakowicz ,写道: >> Hello, >> >> please vote for FLIP-59 >>

[jira] [Created] (FLINK-14718) Refactor ComputedColumnToRelTest to resue TableTestBase

2019-11-12 Thread Jark Wu (Jira)
Jark Wu created FLINK-14718: --- Summary: Refactor ComputedColumnToRelTest to resue TableTestBase Key: FLINK-14718 URL: https://issues.apache.org/jira/browse/FLINK-14718 Project: Flink Issue Type:

[jira] [Created] (FLINK-14716) Cooperate computed column with project push down

2019-11-12 Thread Jark Wu (Jira)
Jark Wu created FLINK-14716: --- Summary: Cooperate computed column with project push down Key: FLINK-14716 URL: https://issues.apache.org/jira/browse/FLINK-14716 Project: Flink Issue Type: Bug