Re: Checkpointing under backpressure

Hi Thomas, There are no Jira tickets or discussions at the moment. If there are any updates I would ping you. I agree that there is a benefit to firstly read other channels without barrier in high priority, otherwise it seems waste cpu resource to migrate blocked buffers to another cached queue

Unbearably slow Table API time-windowed stream join with RocksDBStateBackend

Example SQL: SELECT * FROM stream1 s1, stream2 s2 WHERE s1.id = s2.id AND s1.rowtime = s2.rowtime And we have lots of messages in stream1 and stream2 share a same rowtime. It runs fine when using heap as the state backend, but requires lots of heap memory sometimes (when upstream out of sync, et

[jira] [Created] (FLINK-13707) Make max parallelism configurable

xuekang created FLINK-13707: --- Summary: Make max parallelism configurable Key: FLINK-13707 URL: https://issues.apache.org/jira/browse/FLINK-13707 Project: Flink Issue Type: New Feature Co

Re: Checkpointing under backpressure

Great discussion! I'm excited that this is already under consideration! Are there any JIRAs or other traces of discussion to follow? Paris, if I understand correctly, then the proposal is to not block any input channel at all, but only log data from the backpressured channel (and make it part of t

Re: [DISCUSS] Merging new features post-feature-freeze

Hi, I agree with others in this thread that everyone should be focused on making the release happen and not push their own agenda. It is generally difficult to manage releases with so many interests and moving parts. Many feel the urgency to complete what they were originally hoping to complete. B

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Thanks for pushing this! Thanks, Biao /'bɪ.aʊ/ On Wed, 14 Aug 2019 at 09:37, Jark Wu wrote: > +1 (non-binding) > > Best, > Jark > > On Wed, 14 Aug 2019 at 09:22, Kurt Young wrote: > > > +1 (binding) > > > > Best, > > Kurt > > > > > > On Wed, Aug 14, 2019 at 1:34 AM Yun Tang

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Thomas, Thanks a lot the suggestions. Regarding to bundle processing, there is a section "Checkpoint"[1] in the design doc which talks about how to handle the checkpoint. However, I think you are right that we should talk more about it, such as what's bundle processing, how it affects the ch

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Best, Jark On Wed, 14 Aug 2019 at 09:22, Kurt Young wrote: > +1 (binding) > > Best, > Kurt > > > On Wed, Aug 14, 2019 at 1:34 AM Yun Tang wrote: > > > +1 (non-binding) > > > > But I have a minor question about "code change" action, for those > > "[hotfix]" github pull requests

Re: [VOTE] Flink Project Bylaws

+1 (binding) Best, Kurt On Wed, Aug 14, 2019 at 1:34 AM Yun Tang wrote: > +1 (non-binding) > > But I have a minor question about "code change" action, for those > "[hotfix]" github pull requests [1], the dev mailing list would not be > notified currently. I think we should change the descripti

[jira] [Created] (FLINK-13706) add documentation of using Hive functions in Flink

Bowen Li created FLINK-13706: Summary: add documentation of using Hive functions in Flink Key: FLINK-13706 URL: https://issues.apache.org/jira/browse/FLINK-13706 Project: Flink Issue Type: Sub-ta

Launching Flink server from IntelliJ

Hello, Is there any documentation around launching Flink service from Intellij (VM args, Main class etc)? I have found the documentation on remote debugging , but couldn't find steps to launch Flink from source,

Re: Re: [Discuss] What should the "Data Source" be translated into Chinese

Hi Hengwei, Since this is a discussion about how to translate "Data Sink" into Chinese, I suggest we keep the discussion in @user-zh mailing list. Or at least please translate the reply into English, although TBH I don't think people without knowledge of Chinese language would be interested in joi

Re: [DISCUSS] FLIP-50: Spill-able Heap Keyed State Backend

Sorry for forgetting to give the link of the FLIP, here it is: https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend Thanks! Best Regards, Yu On Tue, 13 Aug 2019 at 18:06, Yu Li wrote: > Hi All, > > We ever held a discussion about this feature before

[jira] [Created] (FLINK-13705) Broken links in Hive documentation

Till Rohrmann created FLINK-13705: - Summary: Broken links in Hive documentation Key: FLINK-13705 URL: https://issues.apache.org/jira/browse/FLINK-13705 Project: Flink Issue Type: Bug

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) But I have a minor question about "code change" action, for those "[hotfix]" github pull requests [1], the dev mailing list would not be notified currently. I think we should change the description of this action. [1] https://flink.apache.org/contributing/contribute-code.html

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Jincheng, Thanks for putting this together. The proposal is very detailed, thorough and for me as a Beam Flink runner contributor easy to understand :) One thing that you should probably detail more is the bundle processing. It is critically important for performance that multiple elements are

[DISCUSS] FLIP-50: Spill-able Heap Keyed State Backend

Hi All, We ever held a discussion about this feature before [1] but now opening another thread because after a second thought introducing a new backend instead of modifying the existing heap backend is a better option to prevent causing any regression or surprise to existing in-production usage. A

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Thanks Becket. I've learned a lot from current bylaws. Best, Jingsong Lee -- From:Yu Li Send Time:2019年8月13日(星期二) 17:48 To:dev Subject:Re: [VOTE] Flink Project Bylaws +1 (non-binding) Thanks for the efforts Beck

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Thanks for the efforts Becket! Best Regards, Yu On Tue, 13 Aug 2019 at 16:09, Xintong Song wrote: > +1 (non-binding) > > Thank you~ > > Xintong Song > > > > On Tue, Aug 13, 2019 at 1:48 PM Robert Metzger > wrote: > > > +1 (binding) > > > > On Tue, Aug 13, 2019 at 1:47 PM Bec

Re: Checkpointing under backpressure

Hi Paris, Thanks for the detailed sharing. And I think it is very similar with the way of overtaking we proposed before. There are some tiny difference: The way of overtaking might need to snapshot all the input/output queues. Chandy Lamport seems only need to snaphost (n-1) input channels afte

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Hi xintong,till > Native and Direct Memory My point is setting a very large max direct memory size when we do not differentiate direct and native memory. If the direct memory,including user direct memory and framework direct memory,could be calculated correctly,then i am in favor of setting dire

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Thank you~ Xintong Song On Tue, Aug 13, 2019 at 1:48 PM Robert Metzger wrote: > +1 (binding) > > On Tue, Aug 13, 2019 at 1:47 PM Becket Qin wrote: > > > Thanks everyone for voting. > > > > For those who have already voted, just want to bring this up to your > > attention th

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for replying, Till. About MemorySegment, I think you are right that we should not include this issue in the scope of this FLIP. This FLIP should concentrate on how to configure memory pools for TaskExecutors, with minimum involvement on how memory consumers use it. About direct memory, I t

[DISCUSS] Flink Python User-Defined Function for Table API

Hi all, The Python Table API(without Python UDF support) has already been supported and will be available in the coming release 1.9. As Python UDF is very important for Python users, we'd like to start the discussion about the Python UDF support in the Python Table API. Aljoscha Krettek, Dian Fu a

Re: Start contribution

Hi Fabian, Thank you. Sure, I will add comment in the issue with way of plan to fix and test the bug. -- *Maddila Rishindra Kumar* *Software Engineer* *Walmartlabs India* *Contact No: +919967379528 | Alternate E-mail ID: rishindra.madd...@walmartlabs.com *

Re:Re: [Discuss] What should the "Data Source" be translated into Chinese

我觉得《计算机科学技术名词》这个很有用，谢谢。 Flink 翻译规范 [ https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications ] 中有一些 Flink 专有名词的翻译规范，例如“State”翻译成“状态”，还有各种时间窗口，以及我们会把“Event”翻译成“事件”。 晦涩或者非常专用的词汇不做翻译比较合适，例如”State Backend“。但是如果可以找到比较优雅的中文词汇能够清晰代表含义的话，翻译过来有助于更好地理解，如果我们对于专有名词建立一个统一的译法就不

[jira] [Created] (FLINK-13704) TPC-H end-to-end test (Blink planner) fails on Travis

Till Rohrmann created FLINK-13704: - Summary: TPC-H end-to-end test (Blink planner) fails on Travis Key: FLINK-13704 URL: https://issues.apache.org/jira/browse/FLINK-13704 Project: Flink Issue

Re:回复：Re: [Discuss] What should the "Data Source" be translated into Chinese

漕是通过水道运输，而sink有沉淀、下降的意思，代表数据的终点和目的地，感觉漕的意义似乎不太合适。 在 2019-08-13 18:07:04，dafei1...@sina.com 写道： >How about translate "data sink" into “数据漕” > 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> > https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu > > > >- 原始邮件 - >发件人：Kurt You

Re: Start contribution

Hi Rishindra, Welcome to the Flink community. Thanks for reporting the bug and volunteering to fix it. Can you briefly discuss in the issue how you plan to fix and test the fix? Thank you, Fabian Am Mo., 12. Aug. 2019 um 23:48 Uhr schrieb Zili Chen : > Hi Rishindra, > > It would be helpful to

Re: Checkpointing under backpressure

yes! It’s quite similar I think. Though mind that the devil is in the details, i.e., the temporal order actions are taken. To clarify, let us say you have a task T with two input channels I1 and I2. The Chandy Lamport execution flow is the following: 1) T receives barrier from I1 and... 2) ..

Re: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release

Hi all, I've opened a pull request for the release announcement. I'm looking forward to your review & feedback: https://github.com/apache/flink-web/pull/244 On Fri, Aug 9, 2019 at 6:27 PM Tzu-Li (Gordon) Tai wrote: > Hi, > > Update: voting on RC2 for Apache Flink 1.9.0 has started: > > http://a

Re: [Discuss] What should the "Data Source" be translated into Chinese

Hi, 这里有翻译的Convention, 可以作为参考, 个人建议这里不翻译 https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications Zili Chen 于2019年8月13日周二 下午6:59写道： > Source 和 Sink 的通译是 源 和 汇，data source 和 data sink 因此分别对应 数据源 和 > 数据汇。出处可参见中国计算机协会的《计算机科学技术名词》，线上检索在这里[1] > > 但是具体到 FLINK 的情况，实际上我建议所有 F

Re: [VOTE] Flink Project Bylaws

+1 (binding) On Tue, Aug 13, 2019 at 1:47 PM Becket Qin wrote: > Thanks everyone for voting. > > For those who have already voted, just want to bring this up to your > attention that there is a minor clarification to the bylaws wiki this > morning. The change is in bold format below: > > one +1

Re: [VOTE] Flink Project Bylaws

Thanks everyone for voting. For those who have already voted, just want to bring this up to your attention that there is a minor clarification to the bylaws wiki this morning. The change is in bold format below: one +1 from a committer followed by a Lazy approval (not counting the vote > of the c

Re: [VOTE] Flink Project Bylaws

+1(non-binding) I really think this will improve many aspects of how our community operates. Thank you Becket for kicking it off! Best, Dawid On 11/08/2019 10:07, Becket Qin wrote: > Hi all, > > I would like to start a voting thread on the project bylaws of Flink. It > aims to help the communit

[jira] [Created] (FLINK-13703) AvroTypeInfo requires objects to be strict POJOs (mutable, with setters)

Alexander Fedulov created FLINK-13703: - Summary: AvroTypeInfo requires objects to be strict POJOs (mutable, with setters) Key: FLINK-13703 URL: https://issues.apache.org/jira/browse/FLINK-13703 Pr

Re: [DISCUSS] Flink project bylaws

Hi Robert, Thanks for help apply the changes. I agree we should freeze the change to the bylaws page starting from now. For this particular addition of clarification, I'll send a notice in the voting thread to let who have already voted to know. Thanks, Jiangjie (Becket) Qin On Tue, Aug 13, 201

Re: [VOTE] Flink Project Bylaws

+1 > On 13 Aug 2019, at 13:22, vino yang wrote: > > +1 > > Tzu-Li (Gordon) Tai 于2019年8月13日周二 下午6:32写道： > >> +1 >> >> On Tue, Aug 13, 2019, 12:31 PM Hequn Cheng wrote: >> >>> +1 (non-binding) >>> >>> Thanks a lot for driving this! Good job. @Becket Qin < >> becket@gmail.com >>>

Re: [DISCUSS] Flink project bylaws

Hi Becket, I've applied the proposed change to the document: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=120731026&selectedPageVersions=20&selectedPageVersions=19 I would propose to stop accepting changes to the document now, as it might start a discussion about the

Re: Checkpointing under backpressure

Thanks for the input. Regarding the Chandy-Lamport snapshots don’t you still have to wait for the “checkpoint barrier” to arrive in order to know when have you already received all possible messages from the upstream tasks/operators? So instead of processing the “in flight” messages (as the Flin

Re: [VOTE] Flink Project Bylaws

+1 Tzu-Li (Gordon) Tai 于2019年8月13日周二 下午6:32写道： > +1 > > On Tue, Aug 13, 2019, 12:31 PM Hequn Cheng wrote: > > > +1 (non-binding) > > > > Thanks a lot for driving this! Good job. @Becket Qin < > becket@gmail.com > > > > > > > Best, Hequn > > > > On Tue, Aug 13, 2019 at 6:26 PM Stephan Ewen

Re: Checkpointing under backpressure

Interesting problem! Thanks for bringing it up Thomas. Ignore/Correct me if I am wrong but I believe Chandy-Lamport snapshots [1] would help out solve this problem more elegantly without sacrificing correctness. - They do not need alignment, only (async) logging for in-flight records between th

Re: [Discuss] What should the "Data Source" be translated into Chinese

Source 和 Sink 的通译是 源 和 汇，data source 和 data sink 因此分别对应 数据源 和 数据汇。出处可参见中国计算机协会的《计算机科学技术名词》，线上检索在这里[1] 但是具体到 FLINK 的情况，实际上我建议所有 FLINK 专有名词都不要翻译，翻译反而增加理解难度。 Best, tison. [1] http://www.termonline.cn/index.htm hsw 于2019年8月13日周二 下午6:46写道： > 赞同，想了半天没想到合适的中文翻译data sink > > > > -

[DISCUSS] FLIP-51: Rework of the Expression Design

Hi everyone, We would like to start a discussion thread on "FLIP-51: Rework of the Expression Design"(Design doc: [1], FLIP: [2]), where we describe how to improve the new java Expressions to work with type inference and convert expression to the calcite RexNode. This is a follow-up plan for FL

回复: [Discuss] What should the "Data Source" be translated into Chinese

赞同，想了半天没想到合适的中文翻译data sink -- 发件人：Jeff Zhang 日　期：2019年08月13日 18:20:34 收件人： 抄　送：dafei1...@sina.com; dev 主　题：Re: [Discuss] What should the "Data Source" be translated into Chinese 不建议翻译 Simon Su 于2019年8月13日周二 下午6:18写道： > 更倾向不去翻译Da

Re: Apache flink 1.7.2 security issues

Hi! Thank you for reporting this! At the moment, the Flink REST endpoint is not secure in the way that you can expose it publicly. After all, you can submit Flink jobs to it which by definition support executing arbitrary code. Given that access to the REST endpoint allows by design arbitrary cod

Re: [VOTE] Flink Project Bylaws

+1 On Tue, Aug 13, 2019, 12:31 PM Hequn Cheng wrote: > +1 (non-binding) > > Thanks a lot for driving this! Good job. @Becket Qin > > > Best, Hequn > > On Tue, Aug 13, 2019 at 6:26 PM Stephan Ewen wrote: > > > +1 > > > > On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels > > wrote: > > > > >

Re: [VOTE] Flink Project Bylaws

+1 (non-binding) Thanks a lot for driving this! Good job. @Becket Qin Best, Hequn On Tue, Aug 13, 2019 at 6:26 PM Stephan Ewen wrote: > +1 > > On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels > wrote: > > > +1 It's good that we formalize this. > > > > On 13.08.19 10:41, Fabian Hueske wrot

Re: [VOTE] Flink Project Bylaws

+1 On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels wrote: > +1 It's good that we formalize this. > > On 13.08.19 10:41, Fabian Hueske wrote: > > +1 for the proposed bylaws. > > Thanks for pushing this Becket! > > > > Cheers, Fabian > > > > Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Me

回复：Re: [Discuss] What should the "Data Source" be translated into Chinese

How about translate "data sink" into “数据漕” 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu - 原始邮件 - 发件人：Kurt Young 收件人：dev , user-zh 主题：Re: [Discuss] What should the "Data Source" be translated into Chinese 日期：2019年08月13日

Re: [VOTE] Flink Project Bylaws

+1 It's good that we formalize this. On 13.08.19 10:41, Fabian Hueske wrote: > +1 for the proposed bylaws. > Thanks for pushing this Becket! > > Cheers, Fabian > > Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Metzger < > rmetz...@apache.org>: > > > I changed the permissions of the page. > > >

Re: [Discuss] What should the "Data Source" be translated into Chinese

不建议翻译 Simon Su 于2019年8月13日周二 下午6:18写道： > 更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 > > > Thanks, > SImon > > > On 08/13/2019 18:07， wrote： > How about translate "data sink" into “数据漕” > 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> > https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu > >

Re: [DISCUSS] Flink project bylaws

Hi Becket, Thanks for clarifying and updating the draft. The changes look good to me. I don't feel strong about a 2/3 majority in case of committer/PMC removal. Like you pointed out, both provide a significant hurdle due to possible vetos or a 2/3 majority. Thanks, Max On 13.08.19 10:36, Becket

Re: [Discuss] What should the "Data Source" be translated into Chinese

更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 Thanks, SImon On 08/13/2019 18:07， wrote： How about translate "data sink" into “数据漕” 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu - 原始邮件 - 发件人：Kurt Young 收件人：dev , user-zh 主题：

[jira] [Created] (FLINK-13702) BaseMapSerializerTest.testDuplicate fails on Travis

Till Rohrmann created FLINK-13702: - Summary: BaseMapSerializerTest.testDuplicate fails on Travis Key: FLINK-13702 URL: https://issues.apache.org/jira/browse/FLINK-13702 Project: Flink Issue T

Re: [DISCUSS] Best practice to run flink on kubernetes

Hi Yang, thanks for reviving the discussion about Flink's Kubernetes integration. In a nutshell, I think that Flink should support option 1) and 3). Concretely, option 1) would be covered by the reactive mode [1] which is not necessarily bound to Kubernetes and works in all environments equally we

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

Hi Richard, although I can see that it would be handy for users who have PubSub set up, I would rather not include examples which require an external dependency into the Flink distribution. I think examples should be self-contained. My concern is that we would bloat the distribution for many users

Re: [VOTE] Flink Project Bylaws

+1 for the proposed bylaws. Thanks for pushing this Becket! Cheers, Fabian Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Metzger < rmetz...@apache.org>: > I changed the permissions of the page. > > On Mon, Aug 12, 2019 at 4:21 PM Till Rohrmann > wrote: > > > +1 for the proposal. Thanks a lo

Re: [DISCUSS] Flink project bylaws

Piotr just reminded me that there was a previous suggestion to clarify a committer's responsibility when commit his/her own patch. So I'd like to incorporate that in the bylaws. The additional clarification is following in bold and italic font. one +1 from a committer followed by a Lazy approval (

Re: Checkpointing under backpressure

Hi Thomas, As Zhijiang has responded, we are now in the process of discussing how to address this issue and one of the solution that we are discussing is exactly what you are proposing: checkpoint barriers overtaking the in flight data and make the in flight data part of the checkpoint. If eve

Re: [Discuss] What should the "Data Source" be translated into Chinese

cc user-zh mailing list, since there are lots of chinese speaking people. Best, Kurt On Tue, Aug 13, 2019 at 4:02 PM WangHengwei wrote: > Hi all, > > > I'm working on [FLINK-13405] Translate "Basic API Concepts" page into > Chinese. I have a problem. > > Usually we translate "Data Sourc

[Discuss] What should the "Data Source" be translated into Chinese

Hi all, I'm working on [FLINK-13405] Translate "Basic API Concepts" page into Chinese. I have a problem. Usually we translate "Data Source" into "数据源" but there is no agreed translation for "Data Sink". Since it often appears in documents, I think we'd better to have a unified tr

[jira] [Created] (FLINK-13701) KafkaTableSourceSink doesn't support topic pattern

Shengnan YU created FLINK-13701: --- Summary: KafkaTableSourceSink doesn't support topic pattern Key: FLINK-13701 URL: https://issues.apache.org/jira/browse/FLINK-13701 Project: Flink Issue Type:

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

Hi Till, After thinking about we can use VARCHAR as an alternative of timestamp/time/date. I'm fine with not recognize it as a blocker issue. We can fix it into 1.9.1. Thanks, Jark On Tue, 13 Aug 2019 at 15:10, Richard Deurwaarder wrote: > Hello all, > > I noticed the PubSub example jar is n

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for proposing this FLIP Xintong. All in all I think it already looks quite good. Concerning the first open question about allocating memory segments, I was wondering whether this is strictly necessary to do in the context of this FLIP or whether this could be done as a follow up? Without kn

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

Hello all, I noticed the PubSub example jar is not included in the examples/ dir of flink-dist. I've created https://issues.apache.org/jira/browse/FLINK-13700 + https://github.com/apache/flink/pull/9424/files to fix this. I will leave it up to you to decide if we want to add this to 1.9.0. Rega

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

Hi Jark, thanks for reporting this issue. Could this be a documented limitation of Blink's preview version? I think we have agreed that the Blink SQL planner will be rather a preview feature than production ready. Hence it could still contain some bugs. My concern is that there might be still othe