Re: [DISCUSS] Change some default config values of blocking shuffle

2021-12-10 Thread Yingjie Cao
Hi dev & users: I have created a FLIP [1] for it, feedbacks are highly appreciated. Best, Yingjie [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-199%3A+Change+some+default+config+values+of+blocking+shuffle+for+better+usability Yingjie Cao 于2021年12月3日周五 17:02写道: > Hi dev & users, >

[jira] [Created] (FLINK-25243) Fabric8FlinkKubeClientITCase.testCheckAndUpdateConfigMapConcurrently

2021-12-10 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25243: - Summary: Fabric8FlinkKubeClientITCase.testCheckAndUpdateConfigMapConcurrently Key: FLINK-25243 URL: https://issues.apache.org/jira/browse/FLINK-25243 Project: Flink

Re: [VOTE] Deprecate Java 8 support

2021-12-10 Thread Chesnay Schepler
The vote has passed unanimously. Binding votes: * Thomas * Till * Ingo * Matthias * Arvid * Timo * Konstantion Non-binding votes: * Martijn * David Thank you for voting! I will file a ticket for the deprecation steps. On 08/12/2021 17:15, Thomas Weise wrote: +1 (binding) On Wed, Dec 8, 2021

[jira] [Created] (FLINK-25244) Deprecate Java 8 support

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25244: Summary: Deprecate Java 8 support Key: FLINK-25244 URL: https://issues.apache.org/jira/browse/FLINK-25244 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-25245) Allow Kubernetes E2E tests to run on Java 11

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25245: Summary: Allow Kubernetes E2E tests to run on Java 11 Key: FLINK-25245 URL: https://issues.apache.org/jira/browse/FLINK-25245 Project: Flink Issue Ty

[jira] [Created] (FLINK-25246) Run benchmarks on Java 11

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25246: Summary: Run benchmarks on Java 11 Key: FLINK-25246 URL: https://issues.apache.org/jira/browse/FLINK-25246 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25247) Inform users about deprecation

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25247: Summary: Inform users about deprecation Key: FLINK-25247 URL: https://issues.apache.org/jira/browse/FLINK-25247 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25248) Deprecate components that inherently require Java 8

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25248: Summary: Deprecate components that inherently require Java 8 Key: FLINK-25248 URL: https://issues.apache.org/jira/browse/FLINK-25248 Project: Flink I

[jira] [Created] (FLINK-25249) Reimplement KafkaSourceTestEnv with KafkaContainer

2021-12-10 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-25249: - Summary: Reimplement KafkaSourceTestEnv with KafkaContainer Key: FLINK-25249 URL: https://issues.apache.org/jira/browse/FLINK-25249 Project: Flink Issue Ty

[jira] [Created] (FLINK-25250) Consider Java 11 as default version on CI

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25250: Summary: Consider Java 11 as default version on CI Key: FLINK-25250 URL: https://issues.apache.org/jira/browse/FLINK-25250 Project: Flink Issue Type:

[jira] [Created] (FLINK-25251) Docker image should use Java 11 by default

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25251: Summary: Docker image should use Java 11 by default Key: FLINK-25251 URL: https://issues.apache.org/jira/browse/FLINK-25251 Project: Flink Issue Type

[jira] [Created] (FLINK-25253) An ClassNotFoundException of missing Hadoop class occurred when submitting to yarn

2021-12-10 Thread ghost synth (Jira)
ghost synth created FLINK-25253: --- Summary: An ClassNotFoundException of missing Hadoop class occurred when submitting to yarn Key: FLINK-25253 URL: https://issues.apache.org/jira/browse/FLINK-25253 Proj

[jira] [Created] (FLINK-25252) Check whether Kafka E2E tests could be run on Java 11

2021-12-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-25252: Summary: Check whether Kafka E2E tests could be run on Java 11 Key: FLINK-25252 URL: https://issues.apache.org/jira/browse/FLINK-25252 Project: Flink

Re: [DISCUSS] Change some default config values of blocking shuffle

2021-12-10 Thread Yun Gao
Hi Yingjie, Very thanks for drafting the FLIP and initiating the discussion! May I have a double confirmation for taskmanager.network.sort-shuffle.min-parallelism that since other frameworks like Spark have used sort-based shuffle for all the cases, does our current circumstance still have dif

[jira] [Created] (FLINK-25254) Default Completed Job List is not ordered by time in web page

2021-12-10 Thread Liu (Jira)
Liu created FLINK-25254: --- Summary: Default Completed Job List is not ordered by time in web page Key: FLINK-25254 URL: https://issues.apache.org/jira/browse/FLINK-25254 Project: Flink Issue Type: Impr

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

2021-12-10 Thread Timo Walther
Hi Wenlong, also thought about adding a `flinkVersion` field per ExecNode. But this is not necessary, because the `version` of the ExecNode has the same purpose. The plan version just encodes that: "plan has been updated in Flink 1.17" / "plan is entirely valid for Flink 1.17" The ExecNode

Re: [DISCUSS] Shall casting functions return null or throw exceptions for invalid input

2021-12-10 Thread Francesco Guardiani
> 3) Ease the management overhead via upgrade story and layered configuration. Maybe even with unified failure management. I'm not sure I get how this is supposed to work. Can you go a bit more in the details? On Thu, Dec 9, 2021 at 3:39 PM Timo Walther wrote: > Hi everyone, > > thanks for th

[jira] [Created] (FLINK-25255) Consider/design implementing State Processor API (FC)

2021-12-10 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-25255: -- Summary: Consider/design implementing State Processor API (FC) Key: FLINK-25255 URL: https://issues.apache.org/jira/browse/FLINK-25255 Project: Flink Iss

Re: [DISCUSS] Change some default config values of blocking shuffle

2021-12-10 Thread 刘建刚
Glad to see the suggestion. In our test, we found that small jobs with the changing configs can not improve the performance much just as your test. I have some suggestions: - The config can affect the memory usage. Will the related memory configs be changed? - Can you share the tpcds resu

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

2021-12-10 Thread wenlong.lwl
hi, Timo, thanks for the explanation. I totally agree with what you said. My actual question is: Will the version of an exec node be serialised in the Json Plan? In my understanding, it is not in the former design. If it is yes, my question is solved already. Best, Wenlong On Fri, 10 Dec 2021 a

[jira] [Created] (FLINK-25256) Savepoints do not work with ExternallyInducedSources

2021-12-10 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-25256: Summary: Savepoints do not work with ExternallyInducedSources Key: FLINK-25256 URL: https://issues.apache.org/jira/browse/FLINK-25256 Project: Flink

[DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Yunfeng Zhou
Hi all, I'm opening this thread to propose the design to support multiple rule & dynamic rule changing in the Flink-CEP project, as described in FLIP-200 [1] . Currently Flink CEP only supports having a single pattern inside a CepOperator and does not support changing the pattern dynamically. In

[jira] [Created] (FLINK-25257) Make the TaskExecutor not depend on concrete BlobServiceCache implementation

2021-12-10 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25257: - Summary: Make the TaskExecutor not depend on concrete BlobServiceCache implementation Key: FLINK-25257 URL: https://issues.apache.org/jira/browse/FLINK-25257 Projec

[jira] [Created] (FLINK-25258) Update log4j2 version to avoid

2021-12-10 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-25258: -- Summary: Update log4j2 version to avoid Key: FLINK-25258 URL: https://issues.apache.org/jira/browse/FLINK-25258 Project: Flink Issue Type: Bug Affects Versi

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

2021-12-10 Thread Zhilong Hong
Congratulations, Ingo! Best, Zhilong On Wed, Dec 8, 2021 at 10:18 PM godfrey he wrote: > Congratulations, Ingo! > > Best, > Godfrey > > Roman Khachatryan 于2021年12月6日周一 下午6:07写道: > > > > Congratulations, Ingo! > > > > Regards, > > Roman > > > > > > On Mon, Dec 6, 2021 at 11:05 AM Yang Wang wro

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

2021-12-10 Thread Zhilong Hong
Congratulations, Matthias! Best, Zhilong On Wed, Dec 8, 2021 at 10:18 PM godfrey he wrote: > Congratulations, Matthias! > > Best, > Godfrey > > Roman Khachatryan 于2021年12月6日周一 下午6:07写道: > > > > Congratulations, Matthias! > > > > Regards, > > Roman > > > > > > On Mon, Dec 6, 2021 at 11:04 AM Ya

Re: [DISCUSS][FLINK-24427] Hide Scala from table planner

2021-12-10 Thread Konstantin Knauf
Hi Francesco, Thanks for this summary. When would the follow ups (mentioned under out of scope) be done? I am asking because the intermediate state of the uber JAR described in the document seems a bit messy and I fear that users will stumble across that. For the old type system for UDFs: naive q

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Martijn Visser
Hi Yunfeng, Thanks for creating the FLIP. I don't see any mention of SQL's MATCH_RECOGNIZE implementation in the FLIP and I think that any change in CEP should be available to both DataStream and SQL/Table API users. Can you elaborate on that? Best regards, Martijn On Fri, 10 Dec 2021 at 12:16,

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Martijn Visser
Apologies, I do see SQL mentioned at the bottom for new rules, but I don't think it's a good idea to have these changes only for the DataStream API in the beginning. This would increase sparsity in Flink, which we should avoid. Best regards, Martijn On Fri, 10 Dec 2021 at 13:19, Martijn Visser

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

2021-12-10 Thread Timo Walther
Hi Wenlong, yes it will. Sorry for the confusion. This is a logical consequence of the assumption: The JSON plan contains no implementation details (i.e. no classes) and is fully declarative. I will add a remark. Thanks, Timo On 10.12.21 11:43, wenlong.lwl wrote: hi, Timo, thanks for th

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Ingo Bürk
Hi, I agree with Martijn on this. The lack of parity across major APIs is a frequent cause of user questions and friction, forcing users to switch between APIs etc. I would therefore also suggest expanding the scope to cover Table API + SQL as well. In general we should probably split FLIPs o

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Timo Walther
Hi all, I don't think it is easy to integrate this kind of functionality in SQL/Table API. The CEP library can be more powerful than MATCH_RECOGNIZE. I haven't taken a look at the FLIP yet. But I would be fine to leave Table API/SQL up for future work and a separate FLIP. Regards, Timo On

Re: [DISCUSS][FLINK-24427] Hide Scala from table planner

2021-12-10 Thread Francesco Guardiani
> When would the follow ups (mentioned under out of scope) be done? For PyFlink, I think we can reasonably get it done soon-ish. What needs to be done is for PyFlink to start using the new type system and then the planner needs to expose a way to plug in rules and to use the code generator. But we

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

2021-12-10 Thread Martijn Visser
Hi Timo, My expectation is that we should at least think of what and how it should work for SQL/Table API, especially since that's our recommendation for new users.That doesn't mean that everything needs to be implemented at the same time. My worry is that in order for a SQL / Table implementatio

About FLIP Page Edit Permission

2021-12-10 Thread Yuepeng Pan
Hi, Community. I want to contribute to Flink. Could someone help me to get the 'FLIP Page Edit Permission' ? JIRA Username: RocMarshal Confluence Username: RocMarshal Thank you. Best, Roc.

[DISCUSS] FLIP-198: Working directory for Flink processes

2021-12-10 Thread Till Rohrmann
Hi everyone, I would like to start a discussion about introducing an explicit working directory for Flink processes that can be used to store information [1]. Per default this working directory will reside in the temporary directory of the node Flink runs on. However, if configured to reside on a

Re: [DISCUSS] FLIP-196: Source API stability guarantees

2021-12-10 Thread Till Rohrmann
Thanks a lot for all your input. I hope that I could resolve most of your concerns. I will now start a vote on this FLIP. Cheers, Till On Tue, Dec 7, 2021 at 4:28 PM Till Rohrmann wrote: > Hi Fabian, > > Thanks for your input. Adding the definition of the different labels to > our documentation

[VOTE] FLIP-196: Source API stability guarantees

2021-12-10 Thread Till Rohrmann
Hi everyone, I'd like to start a vote on FLIP-196: Source API stability guarantees [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/x/IJeqCw [2] https://lists.apa

Re: [VOTE] FLIP-196: Source API stability guarantees

2021-12-10 Thread Till Rohrmann
+1 (binding) On Fri, Dec 10, 2021 at 6:07 PM Till Rohrmann wrote: > Hi everyone, > > I'd like to start a vote on FLIP-196: Source API stability guarantees [1] > which has been discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is an objection > or not enou

Re: [DISCUSS] FLIP-197: API stability graduation process

2021-12-10 Thread Till Rohrmann
Hi everyone, Thanks a lot for all your input. I hope that I could answer most of the questions. I would now start a vote on this FLIP. Cheers, Till On Tue, Dec 7, 2021 at 3:05 PM Till Rohrmann wrote: > Hi Timo, > > 1) I haven't fully thought where to put it but I guess that flink-core > might

[VOTE] FLIP-197: API stability graduation process

2021-12-10 Thread Till Rohrmann
Hi everyone, I'd like to start a vote on FLIP-197: API stability graduation process [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/x/J5eqCw [2] https://lists.ap

Re: [VOTE] FLIP-197: API stability graduation process

2021-12-10 Thread Till Rohrmann
+1 (binding) On Fri, Dec 10, 2021 at 6:09 PM Till Rohrmann wrote: > Hi everyone, > > I'd like to start a vote on FLIP-197: API stability graduation process [1] > which has been discussed in this thread [2]. > > The vote will be open for at least 72 hours unless there is an objection > or not eno

Re: [VOTE] FLIP-196: Source API stability guarantees

2021-12-10 Thread Ingo Bürk
Thanks for driving this, Till. I'm very happy to see the stability guarantees being formalized more, thus +1 (binding) Ingo On 10.12.21 18:07, Till Rohrmann wrote: Hi everyone, I'd like to start a vote on FLIP-196: Source API stability guarantees [1] which has been discussed in this thread

Re: [VOTE] FLIP-197: API stability graduation process

2021-12-10 Thread Ingo Bürk
+1 (binding) On 10.12.21 18:09, Till Rohrmann wrote: Hi everyone, I'd like to start a vote on FLIP-197: API stability graduation process [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://c

confluence permission apply

2021-12-10 Thread Yuepeng Pan
Hi, Till Rohrmann. I want to contribute to Flink. Would you please give me the confluence permission for flip? My Confluence ID is RocMarshal. My JIRA ID is RocMarshal. Thank you very much. Best, Roc.