Re: [DISCUSS] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-09 Thread Wenhao Ji
Thank everyone for participating in this discussion! The improvement has been accepted now. Can someone kindly assign this ticket FLINK-22452 [1] to me? I would like to start to implement this feature. [1] https://issues.apache.org/jira/browse/FLINK-22452 Thanks, Wenhao On Mon, Jun 28, 2021 at

[RESULT] [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-09 Thread Wenhao Ji
Hi everyone, I am happy to announce that FLIP-172 [1] is approved. The vote [2] is now closed. There were five +1 votes, three of them were binding: - Dawid Wysakowicz (binding) - Piotr Nowojski (binding) - Arvid Heise (binding) - Yuan Mei (non-binding) - Daniel Lorych (non-binding) There was

[jira] [Created] (FLINK-23344) State ownership: track and discard private state (rocks, heap)

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23344: - Summary: State ownership: track and discard private state (rocks, heap) Key: FLINK-23344 URL: https://issues.apache.org/jira/browse/FLINK-23344 Project:

[jira] [Created] (FLINK-23343) State ownership: prevent shared state deletion on TM

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23343: - Summary: State ownership: prevent shared state deletion on TM Key: FLINK-23343 URL: https://issues.apache.org/jira/browse/FLINK-23343 Project: Flink

[jira] [Created] (FLINK-23342) State ownership: compute shared state on recovery

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23342: - Summary: State ownership: compute shared state on recovery Key: FLINK-23342 URL: https://issues.apache.org/jira/browse/FLINK-23342 Project: Flink

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-09 Thread Piotr Nowojski
Hey, Thanks for writing this down. +1 from my side. Best, Piotrek pt., 9 lip 2021 o 17:00 Anton,Kalashnikov napisał(a): > Hey! > > There is a wish to decrease amount of in-flight data which can improve > aligned checkpoint time(fewer in-flight data to process before > checkpoint can complete)

[jira] [Created] (FLINK-23341) AkkaRpcActorTest.testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23341: - Summary: AkkaRpcActorTest.testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure Key: FLINK-23341 URL:

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-09 Thread Seth Wiesman
Happy to help clarify, and certainly, I don't think anyone would reject the contribution if you wanted to add that functionality. The important point we want to communicate is that the table API has matured to a level where we do not see it as supplementary to DataStream but as an equal

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-09 Thread Etienne Chauchot
Hi Seth, Thanks for your comment. That is perfect then ! If the community agrees to give precedence to the table API connectors over the DataStream connectors as discussed here, all the features are already available then. I will not need to port my latest additions to table connectors

Re: [DISCUSS] FLIP-182: Watermark alignment

2021-07-09 Thread Seth Wiesman
+1 In my opinion, this limitation is perfectly fine for the MVP. Watermark alignment is a long-standing issue and this already moves the ball so far forward. I don't expect this will cause many issues in practice, as I understand it the FileSource always processes one split at a time, and in my

[DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-09 Thread Anton,Kalashnikov
Hey! There is a wish to decrease amount of in-flight data which can improve aligned checkpoint time(fewer in-flight data to process before checkpoint can complete) and improve the behaviour and performance of unaligned checkpoints (fewer in-flight data that needs to be persisted in every

[jira] [Created] (FLINK-23340) Improve development instructions for flink-training exercises

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23340: --- Summary: Improve development instructions for flink-training exercises Key: FLINK-23340 URL: https://issues.apache.org/jira/browse/FLINK-23340 Project: Flink

[jira] [Created] (FLINK-23339) Disable flink-training exercises in Scala by default

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23339: --- Summary: Disable flink-training exercises in Scala by default Key: FLINK-23339 URL: https://issues.apache.org/jira/browse/FLINK-23339 Project: Flink Issue

[jira] [Created] (FLINK-23338) Use Spotless for flink-training exercises as well

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23338: --- Summary: Use Spotless for flink-training exercises as well Key: FLINK-23338 URL: https://issues.apache.org/jira/browse/FLINK-23338 Project: Flink Issue Type:

[jira] [Created] (FLINK-23337) Properly use the 'shadow' plugin and remove flinkShadowJar

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23337: --- Summary: Properly use the 'shadow' plugin and remove flinkShadowJar Key: FLINK-23337 URL: https://issues.apache.org/jira/browse/FLINK-23337 Project: Flink

[jira] [Created] (FLINK-23336) Use the same log4j version as in Flink 1.13

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23336: --- Summary: Use the same log4j version as in Flink 1.13 Key: FLINK-23336 URL: https://issues.apache.org/jira/browse/FLINK-23336 Project: Flink Issue Type:

[jira] [Created] (FLINK-23335) Add a separate 'runSolution' gradle task

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23335: --- Summary: Add a separate 'runSolution' gradle task Key: FLINK-23335 URL: https://issues.apache.org/jira/browse/FLINK-23335 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-23334) Move 'application' implementation decision to subprojects

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23334: --- Summary: Move 'application' implementation decision to subprojects Key: FLINK-23334 URL: https://issues.apache.org/jira/browse/FLINK-23334 Project: Flink

[jira] [Created] (FLINK-23333) Improve gradle setup for flink-training exercises

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-2: --- Summary: Improve gradle setup for flink-training exercises Key: FLINK-2 URL: https://issues.apache.org/jira/browse/FLINK-2 Project: Flink Issue Type:

[jira] [Created] (FLINK-23332) Update flink-training exercises gradle version

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23332: --- Summary: Update flink-training exercises gradle version Key: FLINK-23332 URL: https://issues.apache.org/jira/browse/FLINK-23332 Project: Flink Issue Type:

[jira] [Created] (FLINK-23331) FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails on Azure

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23331: --- Summary: FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails on Azure Key: FLINK-23331 URL: https://issues.apache.org/jira/browse/FLINK-23331 Project:

[DISCUSS] FLIP-182: Watermark alignment

2021-07-09 Thread Piotr Nowojski
Hey! A couple of weeks ago me and Arvid Heise played around with an idea to address a long standing issue of Flink: lack of watermark/event time alignment between different parallel instances of sources, that can lead to ever growing state size for downstream operators like WindowOperator. We

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-09 Thread Benchao Li
Congratulations Yuan~ Jun Qin 于2021年7月9日周五 下午4:27写道: > Congratulations, Yuan! > > > On Jul 8, 2021, at 11:49 AM, Jingsong Li wrote: > > > > Congratulations Yuan! > > > > Best, > > Jingsong > > > > On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise wrote: > > > >> Yay! > >> > >> On Thu, Jul 8, 2021 at

[jira] [Created] (FLINK-23330) Deprecate toAppendStream and toRetractStream

2021-07-09 Thread Timo Walther (Jira)
Timo Walther created FLINK-23330: Summary: Deprecate toAppendStream and toRetractStream Key: FLINK-23330 URL: https://issues.apache.org/jira/browse/FLINK-23330 Project: Flink Issue Type:

[jira] [Created] (FLINK-23329) Bump flink-shaded to 14.0

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23329: Summary: Bump flink-shaded to 14.0 Key: FLINK-23329 URL: https://issues.apache.org/jira/browse/FLINK-23329 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-23328) UDF Improvement

2021-07-09 Thread waywtdcc (Jira)
waywtdcc created FLINK-23328: Summary: UDF Improvement Key: FLINK-23328 URL: https://issues.apache.org/jira/browse/FLINK-23328 Project: Flink Issue Type: Improvement Components: Table

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-09 Thread Jun Qin
Congratulations, Yuan! > On Jul 8, 2021, at 11:49 AM, Jingsong Li wrote: > > Congratulations Yuan! > > Best, > Jingsong > > On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise wrote: > >> Yay! >> >> On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao >> wrote: >> >>> Congratulations Yuan! >>> >>> Best,

[jira] [Created] (FLINK-23327) Spark Structured Streaming runner sources should be put back to main src folder

2021-07-09 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-23327: Summary: Spark Structured Streaming runner sources should be put back to main src folder Key: FLINK-23327 URL: https://issues.apache.org/jira/browse/FLINK-23327

Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

2021-07-09 Thread Yingjie Cao
Hi Zhu, Thanks for the reply. > One question which might be out of the scope is that whether we should do similar things for ShuffleEnvironment? I agree we should also consider fatal error handling for ShuffleEnvironment eventually. > I think the proposal does not conflict with this target.

Re: Job Recovery Time on TM Lost

2021-07-09 Thread Till Rohrmann
Gen is right with his explanation why the dead TM discovery can be faster with Flink < 1.12. Concerning flaky TaskManager connections: 2.1 I think the problem is that the receiving TM does not know the container ID of the sending TM. It only knows its address. But this is something one could

[jira] [Created] (FLINK-23326) Bump flink-shaded-guava to 30.1.1

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23326: Summary: Bump flink-shaded-guava to 30.1.1 Key: FLINK-23326 URL: https://issues.apache.org/jira/browse/FLINK-23326 Project: Flink Issue Type:

[jira] [Created] (FLINK-23325) Bump flink-shaded-netty to 4.1.65

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23325: Summary: Bump flink-shaded-netty to 4.1.65 Key: FLINK-23325 URL: https://issues.apache.org/jira/browse/FLINK-23325 Project: Flink Issue Type:

[jira] [Created] (FLINK-23324) Postgres of JDBC Connector enable case-sensitive.

2021-07-09 Thread Ada Wong (Jira)
Ada Wong created FLINK-23324: Summary: Postgres of JDBC Connector enable case-sensitive. Key: FLINK-23324 URL: https://issues.apache.org/jira/browse/FLINK-23324 Project: Flink Issue Type: Bug