[DISCUSS] Introducing Backpressure for connected streams

2021-03-23 Thread Robin KC
Hi all, The issue has been discussed before here - http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Sharing-state-between-subtasks-td24489.html Our use case requires event time join of two streams and we use ConnectedStreams for the same. Within the CoProcessFunction, we buffer reco

[jira] [Created] (FLINK-21942) KubernetesLeaderRetrievalDriver not closed after terminated which lead to connection leak

2021-03-23 Thread Yi Tang (Jira)
Yi Tang created FLINK-21942: --- Summary: KubernetesLeaderRetrievalDriver not closed after terminated which lead to connection leak Key: FLINK-21942 URL: https://issues.apache.org/jira/browse/FLINK-21942 Proje

[jira] [Created] (FLINK-21941) testSavepointRescalingOutPartitionedOperatorStateList fail

2021-03-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21941: - Summary: testSavepointRescalingOutPartitionedOperatorStateList fail Key: FLINK-21941 URL: https://issues.apache.org/jira/browse/FLINK-21941 Project: Flink Issue Ty

[jira] [Created] (FLINK-21940) Rowtime/proctime should be obtained from getTimestamp instead of getLong

2021-03-23 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-21940: Summary: Rowtime/proctime should be obtained from getTimestamp instead of getLong Key: FLINK-21940 URL: https://issues.apache.org/jira/browse/FLINK-21940 Project: Fli

[jira] [Created] (FLINK-21939) Support batch mode in Python DataStream API for process operation

2021-03-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-21939: --- Summary: Support batch mode in Python DataStream API for process operation Key: FLINK-21939 URL: https://issues.apache.org/jira/browse/FLINK-21939 Project: Flink Issu

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-23 Thread chenqin
Also noticed the actual states stored in _metadata still contains entropy marker after we fix metadata directory issue. This issue seems related to code refactory as well as doesn't conveyed in tests. -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-21938) Add documentation about how to test Python UDFs

2021-03-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-21938: --- Summary: Add documentation about how to test Python UDFs Key: FLINK-21938 URL: https://issues.apache.org/jira/browse/FLINK-21938 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-21937) Support batch mode in Python DataStream API for basic operations

2021-03-23 Thread Dian Fu (Jira)
Dian Fu created FLINK-21937: --- Summary: Support batch mode in Python DataStream API for basic operations Key: FLINK-21937 URL: https://issues.apache.org/jira/browse/FLINK-21937 Project: Flink Issue

[jira] [Created] (FLINK-21936) Disable checkpointing of inflight data in pointwise connections for unaligned checkpoints

2021-03-23 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-21936: --- Summary: Disable checkpointing of inflight data in pointwise connections for unaligned checkpoints Key: FLINK-21936 URL: https://issues.apache.org/jira/browse/FLINK-21936

Re: [DISCUSS] Feature freeze date for 1.13

2021-03-23 Thread Thomas Weise
Hi Dawid, Thanks for the heads up. Regarding the "Rebase and merge" button. I find that merge option useful, especially for small simple changes and for backports. The following should help to safeguard from the issue encountered previously: https://github.com/jazzband/pip-tools/issues/1085 Than

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-23 Thread chenqin
make it easier to read @Nullable private static EntropyInjectingFileSystem getEntropyFs(FileSystem fs) { LOG.warn(fs.getClass().toGenericString()); if (fs instanceof EntropyInjectingFileSystem) { return (EntropyInjectingFileSy

[jira] [Created] (FLINK-21935) Remove "state.backend.async" option.

2021-03-23 Thread Stephan Ewen (Jira)
Stephan Ewen created FLINK-21935: Summary: Remove "state.backend.async" option. Key: FLINK-21935 URL: https://issues.apache.org/jira/browse/FLINK-21935 Project: Flink Issue Type: Improvement

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-23 Thread chenqin
Hi Till, Thanks for sharing pointers related to entropy injection feature on 1.11. We did some investigation and so far it seems like an edge case handling bug. Testing Environment: flink 1.11.2 release with plugins plugins/s3-fs-hadoop/flink-s3-fs-hadoop state.backend.rocksdb.timer-service.fac

Re: [jira] [Created] (FLINK-21820) JDBC connector shouldn't read all rows in per statement by default

2021-03-23 Thread ding peng
unsubscribe Leonard Xu (Jira) 于2021年3月16日周二 下午8:05写道: > Leonard Xu created FLINK-21820: > -- > > Summary: JDBC connector shouldn't read all rows in per > statement by default > Key: FLINK-21820 > URL: https://issues.a

[jira] [Created] (FLINK-21934) Add new StreamTableEnvironment.toDataStream

2021-03-23 Thread Timo Walther (Jira)
Timo Walther created FLINK-21934: Summary: Add new StreamTableEnvironment.toDataStream Key: FLINK-21934 URL: https://issues.apache.org/jira/browse/FLINK-21934 Project: Flink Issue Type: Sub-t

[jira] [Created] (FLINK-21933) [kinesis][efo] EFO consumer treats interrupts as retryable exceptions

2021-03-23 Thread Danny Cranmer (Jira)
Danny Cranmer created FLINK-21933: - Summary: [kinesis][efo] EFO consumer treats interrupts as retryable exceptions Key: FLINK-21933 URL: https://issues.apache.org/jira/browse/FLINK-21933 Project: Flin

[jira] [Created] (FLINK-21932) Add a Python SDK showcase to the playground

2021-03-23 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-21932: Summary: Add a Python SDK showcase to the playground Key: FLINK-21932 URL: https://issues.apache.org/jira/browse/FLINK-21932 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-21931) Watermark documentation needs to be fixed

2021-03-23 Thread Matthias (Jira)
Matthias created FLINK-21931: Summary: Watermark documentation needs to be fixed Key: FLINK-21931 URL: https://issues.apache.org/jira/browse/FLINK-21931 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-21930) There are some error for hive_read_writer.md

2021-03-23 Thread hehuiyuan (Jira)
hehuiyuan created FLINK-21930: - Summary: There are some error for hive_read_writer.md Key: FLINK-21930 URL: https://issues.apache.org/jira/browse/FLINK-21930 Project: Flink Issue Type: Wish

[jira] [Created] (FLINK-21929) flink-statebackend-rocksdb crashes with Error occurred in starting fork

2021-03-23 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-21929: -- Summary: flink-statebackend-rocksdb crashes with Error occurred in starting fork Key: FLINK-21929 URL: https://issues.apache.org/jira/browse/FLINK-21929 Project:

[DISCUSS] Feature freeze date for 1.13

2021-03-23 Thread Dawid Wysakowicz
Hi devs, users! 1. *Feature freeze date* We are approaching the end of March which we agreed would be the time for a Feature Freeze. From the knowledge I've gather so far it still seems to be a viable plan. I think it is a good time to agree on a particular date, when it should happen. We suggest

Re: Re: Re: [DISCUSSION] Introduce a separated memory pool for the TM merge shuffle

2021-03-23 Thread Guowei Ma
Hi, I discussed with Xingtong and Yingjie offline and we agreed that the name `taskmanager.memory.framework.off-heap.batch-shuffle.size` can better reflect the current memory usage. So we decided to use the name Till suggested. Thank you all for your valuable feedback. Best, Guowei On Mon, Mar 22

[jira] [Created] (FLINK-21928) DuplicateJobSubmissionException after JobManager failover

2021-03-23 Thread Ufuk Celebi (Jira)
Ufuk Celebi created FLINK-21928: --- Summary: DuplicateJobSubmissionException after JobManager failover Key: FLINK-21928 URL: https://issues.apache.org/jira/browse/FLINK-21928 Project: Flink Issue

[jira] [Created] (FLINK-21927) ExpressionReducer compile fail when running long term

2021-03-23 Thread runsun (Jira)
runsun created FLINK-21927: -- Summary: ExpressionReducer compile fail when running long term Key: FLINK-21927 URL: https://issues.apache.org/jira/browse/FLINK-21927 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-21926) Document Fine Grained Resource Management

2021-03-23 Thread Xintong Song (Jira)
Xintong Song created FLINK-21926: Summary: Document Fine Grained Resource Management Key: FLINK-21926 URL: https://issues.apache.org/jira/browse/FLINK-21926 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-21925) Support configuring fine grained resource requirements via DataStream API

2021-03-23 Thread Xintong Song (Jira)
Xintong Song created FLINK-21925: Summary: Support configuring fine grained resource requirements via DataStream API Key: FLINK-21925 URL: https://issues.apache.org/jira/browse/FLINK-21925 Project: Fl

Re: [VOTE] Move Flink ML pipeline API and library code to a separate repository named flink-ml

2021-03-23 Thread Dong Lin
Thanks everyone for your votes! The proposal has passed with the following votes: +1 (Binding): 3 (Becket, Till, Dian) +1 (Non-binding): 0 -1: 0 Thanks, Dong On Mon, Mar 22, 2021 at 7:34 PM Dian Fu wrote: > +1 > > Regards, > Dian > > On Fri, Mar 19, 2021 at 6:39 PM Till Rohrmann > wrote: > >

[jira] [Created] (FLINK-21924) Fine Grained Resource Management

2021-03-23 Thread Xintong Song (Jira)
Xintong Song created FLINK-21924: Summary: Fine Grained Resource Management Key: FLINK-21924 URL: https://issues.apache.org/jira/browse/FLINK-21924 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Apache Flink Jira Process

2021-03-23 Thread Konstantin Knauf
Hi everyone, The discussion has stalled a bit on this thread. I would proceed to a vote on the currently documented proposal tomorrow if there are no further concerns or opinions. Best, Konstantin On Fri, Mar 12, 2021 at 5:24 PM Konstantin Knauf wrote: > Hi Leonard, > > Thank you for your fee