[jira] [Created] (FLINK-19654) Improve the execution time of PyFlink end-to-end tests

2020-10-14 Thread Dian Fu (Jira)
Dian Fu created FLINK-19654: --- Summary: Improve the execution time of PyFlink end-to-end tests Key: FLINK-19654 URL: https://issues.apache.org/jira/browse/FLINK-19654 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19653) HiveCatalogITCase fails on azure

2020-10-14 Thread Yun Gao (Jira)
Yun Gao created FLINK-19653: --- Summary: HiveCatalogITCase fails on azure Key: FLINK-19653 URL: https://issues.apache.org/jira/browse/FLINK-19653 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19652) Support pure string format for Kafka table connector

2020-10-14 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-19652: - Summary: Support pure string format for Kafka table connector Key: FLINK-19652 URL: https://issues.apache.org/jira/browse/FLINK-19652 Project: Flink Issue

[jira] [Created] (FLINK-19651) Support filter push down for the Jdbc connector

2020-10-14 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-19651: - Summary: Support filter push down for the Jdbc connector Key: FLINK-19651 URL: https://issues.apache.org/jira/browse/FLINK-19651 Project: Flink Issue

[jira] [Created] (FLINK-19650) Support the limit push down for the Jdbc connector

2020-10-14 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-19650: - Summary: Support the limit push down for the Jdbc connector Key: FLINK-19650 URL: https://issues.apache.org/jira/browse/FLINK-19650 Project: Flink Issue

[jira] [Created] (FLINK-19649) In Flink's module flink-sql-parser/sqlCreateTable/unparse, the tables without columns are not supported.

2020-10-14 Thread gsralex (Jira)
gsralex created FLINK-19649: --- Summary: In Flink's module flink-sql-parser/sqlCreateTable/unparse, the tables without columns are not supported. Key: FLINK-19649 URL: https://issues.apache.org/jira/browse/FLINK-19649

[jira] [Created] (FLINK-19648) Support nested projection push down for hbase connector

2020-10-14 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-19648: - Summary: Support nested projection push down for hbase connector Key: FLINK-19648 URL: https://issues.apache.org/jira/browse/FLINK-19648 Project: Flink

[jira] [Created] (FLINK-19647) Support limit push down for hbase connector

2020-10-14 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-19647: - Summary: Support limit push down for hbase connector Key: FLINK-19647 URL: https://issues.apache.org/jira/browse/FLINK-19647 Project: Flink Issue Type:

[DISCUSS] FLIP-148: Introduce Sort-Merge Based Blocking Shuffle to Flink

2020-10-14 Thread Yingjie Cao
Hi devs, Currently, Flink adopts a hash-style blocking shuffle implementation which writes data sent to different reducer tasks into separate files concurrently. Compared to sort-merge based approach writes those data together into a single file and merges those small files into bigger ones,

Re: [DISCUSS] Support KeyedSortedMapState in DataStream API

2020-10-14 Thread Jark Wu
Hi, Thanks for bringing this discussion. I think limiting the key type to Long can't resolve the comparison problem, because the bytes order and value order of negative numbers is different. Unless, we limit the key type to positive Long. But how to check this before submitting a job? In Blink

[jira] [Created] (FLINK-19646) StreamExecutionEnvironment support new Source interface based on FLIP-27

2020-10-14 Thread Nicholas Jiang (Jira)
Nicholas Jiang created FLINK-19646: -- Summary: StreamExecutionEnvironment support new Source interface based on FLIP-27 Key: FLINK-19646 URL: https://issues.apache.org/jira/browse/FLINK-19646

[jira] [Created] (FLINK-19645) ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable

2020-10-14 Thread Dian Fu (Jira)
Dian Fu created FLINK-19645: --- Summary: ShuffleCompressionITCase.testDataCompressionForBlockingShuffle is instable Key: FLINK-19645 URL: https://issues.apache.org/jira/browse/FLINK-19645 Project: Flink

Re: [DISCUSS] Support KeyedSortedMapState in DataStream API

2020-10-14 Thread Sean Z
Thanks for the reply! Look forward to learning more about this prototype. Is there any way that we could track this TemporalState like Jira issue? or should we start to create one in Jira? so anyone has interest like me, could be part of the loop. Besides, is there any written docs/code about the

Re: [DISCUSS] Support KeyedSortedMapState in DataStream API

2020-10-14 Thread David Anderson
I'm very interested in this topic, and have even done some prototyping of solution 1 -- limiting the key type to Long -- which Nico Kruber and I called TemporalState in our prototype. I look forward to sharing what we learned, and to discussing this further, but I am completely overwhelmed with

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

2020-10-14 Thread Satyaa Dixit
Hi Chesnay/Team Thank you so much.I have tried with the solution but it is not working as expected showing compilation issues and tried all the ways .Please find below code snippet : s3PathList.stream() .map(directory -> S3Service.customCreateInputStream(environment, directory,

[jira] [Created] (FLINK-19644) Support read specific partition of Hive table in temporal join

2020-10-14 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-19644: -- Summary: Support read specific partition of Hive table in temporal join Key: FLINK-19644 URL: https://issues.apache.org/jira/browse/FLINK-19644 Project: Flink

[jira] [Created] (FLINK-19643) Add Pandas UDAF Doc

2020-10-14 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-19643: Summary: Add Pandas UDAF Doc Key: FLINK-19643 URL: https://issues.apache.org/jira/browse/FLINK-19643 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19642) RocksDBListStatePerformanceTest.testRocksDbListStateAPIs is instable

2020-10-14 Thread Dian Fu (Jira)
Dian Fu created FLINK-19642: --- Summary: RocksDBListStatePerformanceTest.testRocksDbListStateAPIs is instable Key: FLINK-19642 URL: https://issues.apache.org/jira/browse/FLINK-19642 Project: Flink

[jira] [Created] (FLINK-19641) Optimize parallelism calculating of HiveTableSource by checking file number

2020-10-14 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-19641: --- Summary: Optimize parallelism calculating of HiveTableSource by checking file number Key: FLINK-19641 URL: https://issues.apache.org/jira/browse/FLINK-19641 Project:

[jira] [Created] (FLINK-19640) Expose to users the batch-style execution for bounded keyed streams

2020-10-14 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-19640: Summary: Expose to users the batch-style execution for bounded keyed streams Key: FLINK-19640 URL: https://issues.apache.org/jira/browse/FLINK-19640 Project:

[jira] [Created] (FLINK-19639) Support SupportsNestedProjectionPushDown in planner

2020-10-14 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-19639: - Summary: Support SupportsNestedProjectionPushDown in planner Key: FLINK-19639 URL: https://issues.apache.org/jira/browse/FLINK-19639 Project: Flink Issue

[jira] [Created] (FLINK-19638) AvroUnionLogicalSerializerTest class compile error

2020-10-14 Thread shizhengchao (Jira)
shizhengchao created FLINK-19638: Summary: AvroUnionLogicalSerializerTest class compile error Key: FLINK-19638 URL: https://issues.apache.org/jira/browse/FLINK-19638 Project: Flink Issue

[jira] [Created] (FLINK-19637) Remove AllocationIdsExposingResourceManagerGateway

2020-10-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-19637: Summary: Remove AllocationIdsExposingResourceManagerGateway Key: FLINK-19637 URL: https://issues.apache.org/jira/browse/FLINK-19637 Project: Flink

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-10-14 Thread Aljoscha Krettek
Agreed! Aljoscha On 14.10.20 06:38, Jingsong Li wrote: Hi Aljoscha, Thanks for your feedback. Yes, we should add DataStream Providers to the table bridge module. I think your concerns are right, including the relationship between DataStream and table. My understanding is that the

[jira] [Created] (FLINK-19636) Add declarative SlotPool

2020-10-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-19636: Summary: Add declarative SlotPool Key: FLINK-19636 URL: https://issues.apache.org/jira/browse/FLINK-19636 Project: Flink Issue Type: Task

[DISCUSS] Support KeyedSortedMapState in DataStream API

2020-10-14 Thread Sean Z
Hi devs, Current DataStream API doesn't have SortedMapState supported. There are lots of use cases based on sorted time-series data like range-query or higher/lower key fetch, and ordered data seems like a nature of time-series stream processing. Therefore, we propose to support the

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-14 Thread Yun Gao
Hi Till, Very thanks for the feedbacks ! > 1) When restarting all tasks independent of the status at checkpoint time > (finished, running, scheduled), we might allocate more resources than we > actually need to run the remaining job. From a scheduling perspective it > would be easier if we

[jira] [Created] (FLINK-19635) HBaseConnectorITCase.testTableSourceSinkWithDDL is unstable with a result mismatch

2020-10-14 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19635: -- Summary: HBaseConnectorITCase.testTableSourceSinkWithDDL is unstable with a result mismatch Key: FLINK-19635 URL: https://issues.apache.org/jira/browse/FLINK-19635

[jira] [Created] (FLINK-19634) Official Flink visualizer display bug

2020-10-14 Thread appleyuchi (Jira)
appleyuchi created FLINK-19634: -- Summary: Official Flink visualizer display bug Key: FLINK-19634 URL: https://issues.apache.org/jira/browse/FLINK-19634 Project: Flink Issue Type: Bug

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-14 Thread Arvid Heise
I remember this conversation popping up a few times already and I'm in general a big fan of removing BucketingSink. However, until now there were a few features lacking in StreamingFileSink that are present in BucketingSink and that are being actively used (I can't exactly remember them now, but

[jira] [Created] (FLINK-19633) Fix ArrayIndexOutOfBoundsException in RetractableTopNFunction

2020-10-14 Thread dalongliu (Jira)
dalongliu created FLINK-19633: - Summary: Fix ArrayIndexOutOfBoundsException in RetractableTopNFunction Key: FLINK-19633 URL: https://issues.apache.org/jira/browse/FLINK-19633 Project: Flink