Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

2021-07-08 Thread Zhu Zhu
Thanks for starting this discussion. Here are some of my thoughts regarding the proposal and discussions above. *+1 to enable ShuffleMaster to stop track partitions proactively* In production we have encountered problems that it needs *hours* to recover from a remote shuffle worker lost problem.

[jira] [Created] (FLINK-23323) HAQueryableStateRocksDBBackendITCase failed due to heap OOM

2021-07-08 Thread Xintong Song (Jira)
Xintong Song created FLINK-23323: Summary: HAQueryableStateRocksDBBackendITCase failed due to heap OOM Key: FLINK-23323 URL: https://issues.apache.org/jira/browse/FLINK-23323 Project: Flink

Re: Job Recovery Time on TM Lost

2021-07-08 Thread Gen Luo
@刘建刚 Welcome to join the discuss and thanks for sharing your experience. I have a minor question. In my experience, network failures in a certain cluster usually takes a time to recovery, which can be measured as p99 to guide configuring. So I suppose it would be better to use time than attempt

[jira] [Created] (FLINK-23322) RMQSourceITCase.testStopWithSavepoint fails on azure due to timeout

2021-07-08 Thread Xintong Song (Jira)
Xintong Song created FLINK-23322: Summary: RMQSourceITCase.testStopWithSavepoint fails on azure due to timeout Key: FLINK-23322 URL: https://issues.apache.org/jira/browse/FLINK-23322 Project: Flink

[jira] [Created] (FLINK-23321) Make REST Netty thread county configurable

2021-07-08 Thread Nicolas Raga (Jira)
Nicolas Raga created FLINK-23321: Summary: Make REST Netty thread county configurable Key: FLINK-23321 URL: https://issues.apache.org/jira/browse/FLINK-23321 Project: Flink Issue Type:

Re: Job Recovery Time on TM Lost

2021-07-08 Thread Lu Niu
Thanks everyone! This is a great discussion! 1. Restarting takes 30s when throwing exceptions from application code because the restart delay is 30s in config. Before lots of related config are 30s which lead to the confusion. I redo the test with config:

Re: [ANNOUNCE] New PMC member: Guowei Ma

2021-07-08 Thread Steven Wu
Awesome! Congratulations, Guowei! On Wed, Jul 7, 2021 at 4:25 AM Jingsong Li wrote: > Congratulations, Guowei! > > Best, > Jingsong > > On Wed, Jul 7, 2021 at 6:36 PM Arvid Heise wrote: > > > Congratulations! > > > > On Wed, Jul 7, 2021 at 11:30 AM Till Rohrmann > > wrote: > > > > >

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-08 Thread Seth Wiesman
Hi Etienne, The `toDataStream` method supports converting to concrete Java types, not just Row, which can include your Avro specific-records. See example 2: https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/data_stream_api/#examples-for-todatastream On Thu, Jul 8, 2021 at

[jira] [Created] (FLINK-23320) Support semi/anti temporal joins

2021-07-08 Thread Timo Walther (Jira)
Timo Walther created FLINK-23320: Summary: Support semi/anti temporal joins Key: FLINK-23320 URL: https://issues.apache.org/jira/browse/FLINK-23320 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-23319) Support semi/anti lookup joins

2021-07-08 Thread Timo Walther (Jira)
Timo Walther created FLINK-23319: Summary: Support semi/anti lookup joins Key: FLINK-23319 URL: https://issues.apache.org/jira/browse/FLINK-23319 Project: Flink Issue Type: New Feature

[DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-08 Thread Arvid Heise
Dear devs, As a continuation and generalization of FLIP-33 (Standardize Connector Metrics) [1], we'd like to discuss how we actually expose the standardized operator metrics to users in terms of changes to the API. Please check out the FLIP [2] and provide feedback. Best, Arvid [1]

[jira] [Created] (FLINK-23318) AkkaRpcActorTest#testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure

2021-07-08 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-23318: Summary: AkkaRpcActorTest#testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure Key: FLINK-23318 URL: https://issues.apache.org/jira/browse/FLINK-23318

[jira] [Created] (FLINK-23317) Only keep the latest checkpoint in CompletedCheckpointStore

2021-07-08 Thread Jira
David Morávek created FLINK-23317: - Summary: Only keep the latest checkpoint in CompletedCheckpointStore Key: FLINK-23317 URL: https://issues.apache.org/jira/browse/FLINK-23317 Project: Flink

[jira] [Created] (FLINK-23316) There's no test for custom PartitionCommitPolicy

2021-07-08 Thread Rui Li (Jira)
Rui Li created FLINK-23316: -- Summary: There's no test for custom PartitionCommitPolicy Key: FLINK-23316 URL: https://issues.apache.org/jira/browse/FLINK-23316 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Feedback Collection Jira Bot

2021-07-08 Thread Timo Walther
Hi everyone, let me also post some feedback here. As someone who monitors and maintains the SQL component JIRA issues, I'm quite unhappy with the current bot behavior. I'm watching quite a few issues and SQL has a lot of pretty major if not critical topics. My mailbox is full of bot messages

[jira] [Created] (FLINK-23315) Bump log4j to 2.14.1 for version 1.13.2

2021-07-08 Thread Guilaume Kermorgant (Jira)
Guilaume Kermorgant created FLINK-23315: --- Summary: Bump log4j to 2.14.1 for version 1.13.2 Key: FLINK-23315 URL: https://issues.apache.org/jira/browse/FLINK-23315 Project: Flink Issue

[jira] [Created] (FLINK-23314) State left behind for short lived keys

2021-07-08 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-23314: Summary: State left behind for short lived keys Key: FLINK-23314 URL: https://issues.apache.org/jira/browse/FLINK-23314 Project: Flink Issue Type:

[jira] [Created] (FLINK-23313) Reintroduce temporal table function documentation

2021-07-08 Thread Timo Walther (Jira)
Timo Walther created FLINK-23313: Summary: Reintroduce temporal table function documentation Key: FLINK-23313 URL: https://issues.apache.org/jira/browse/FLINK-23313 Project: Flink Issue

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-08 Thread Etienne Chauchot
Hi Timo, Thanks for your answers, no problem with the delay, I was in vacation too last week :) My comments are inline Best, Etienne On 07/07/2021 16:48, Timo Walther wrote: Hi Etienne, sorry for the late reply due to my vacation last week. Regarding: "support of aggregations in batch

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Arvid Heise
Yay! On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao wrote: > Congratulations Yuan! > > Best, > Jiayi Liao > > On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan wrote: > > > Congratulations Yuan! > > > > Regards, > > Roman > > > > On Thu, Jul 8, 2021 at 6:02 AM Yang Wang wrote: > > > > > >

[jira] [Created] (FLINK-23312) Use -Dfast for building e2e tests on AZP

2021-07-08 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23312: --- Summary: Use -Dfast for building e2e tests on AZP Key: FLINK-23312 URL: https://issues.apache.org/jira/browse/FLINK-23312 Project: Flink Issue Type:

[jira] [Created] (FLINK-23311) Improve PojoSerializer test

2021-07-08 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23311: --- Summary: Improve PojoSerializer test Key: FLINK-23311 URL: https://issues.apache.org/jira/browse/FLINK-23311 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-23310) Correct the `ModifyKindSetTrait` for `GroupWindowAggregate` when the input is an update stream

2021-07-08 Thread Shuo Cheng (Jira)
Shuo Cheng created FLINK-23310: -- Summary: Correct the `ModifyKindSetTrait` for `GroupWindowAggregate` when the input is an update stream Key: FLINK-23310 URL: https://issues.apache.org/jira/browse/FLINK-23310

[jira] [Created] (FLINK-23309) Optimize the finish bundle logic in PyFlink

2021-07-08 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-23309: Summary: Optimize the finish bundle logic in PyFlink Key: FLINK-23309 URL: https://issues.apache.org/jira/browse/FLINK-23309 Project: Flink Issue Type:

[jira] [Created] (FLINK-23308) Performance regression on 06.07

2021-07-08 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-23308: Summary: Performance regression on 06.07 Key: FLINK-23308 URL: https://issues.apache.org/jira/browse/FLINK-23308 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-23307) Rowtime attributes are not properly resolved for views

2021-07-08 Thread Jira
Ingo Bürk created FLINK-23307: - Summary: Rowtime attributes are not properly resolved for views Key: FLINK-23307 URL: https://issues.apache.org/jira/browse/FLINK-23307 Project: Flink Issue Type:

[jira] [Created] (FLINK-23306) FlinkRelMdUniqueKeys causes exception when used with new Schema

2021-07-08 Thread Timo Walther (Jira)
Timo Walther created FLINK-23306: Summary: FlinkRelMdUniqueKeys causes exception when used with new Schema Key: FLINK-23306 URL: https://issues.apache.org/jira/browse/FLINK-23306 Project: Flink

[jira] [Created] (FLINK-23305) Support semi/anti interval joins

2021-07-08 Thread Timo Walther (Jira)
Timo Walther created FLINK-23305: Summary: Support semi/anti interval joins Key: FLINK-23305 URL: https://issues.apache.org/jira/browse/FLINK-23305 Project: Flink Issue Type: New Feature

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Jiayi Liao
Congratulations Yuan! Best, Jiayi Liao On Thu, Jul 8, 2021 at 3:55 PM Roman Khachatryan wrote: > Congratulations Yuan! > > Regards, > Roman > > On Thu, Jul 8, 2021 at 6:02 AM Yang Wang wrote: > > > > Congratulations Yuan! > > > > Best, > > Yang > > > > XING JIN 于2021年7月8日周四 上午11:46写道: > > >

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-08 Thread Roman Khachatryan
Congratulations Yuan! Regards, Roman On Thu, Jul 8, 2021 at 6:02 AM Yang Wang wrote: > > Congratulations Yuan! > > Best, > Yang > > XING JIN 于2021年7月8日周四 上午11:46写道: > > > Congratulations Yuan~! > > > > Roc Marshal 于2021年7月8日周四 上午11:28写道: > > > > > Congratulations, Yuan! > > > > > > > > > > >

Re: [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-08 Thread Yuan Mei
+1 Best, Yuan On Thu, Jul 8, 2021 at 3:03 PM Dawid Wysakowicz wrote: > +1 (binding) > > Best, > > Dawid > > On 28/06/2021 16:00, Wenhao Ji wrote: > > Hi everyone, > > > > I would like to start a vote on FLIP-172 [1] which was discussed in > > this thread [2]. > > The vote will be open for at

Re: [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-08 Thread Dawid Wysakowicz
+1 (binding) Best, Dawid On 28/06/2021 16:00, Wenhao Ji wrote: > Hi everyone, > > I would like to start a vote on FLIP-172 [1] which was discussed in > this thread [2]. > The vote will be open for at least 72 hours until July 1 unless there > is an objection or not enough votes. > > Thanks, >