Re:Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang

[Haibo Sun]

Congratulations!


Best,
Haibo

在 2020-09-18 01:46:59，"Piotr Nowojski"  写道：
>Congratulations :)
>
>czw., 17 wrz 2020 o 15:20 godfrey he  napisał(a):
>
>> Congratulations!
>>
>> Regards,
>> Godfrey
>>
>> Yun Tang  于2020年9月17日周四 下午2:22写道：
>>
>> >  Thanks for all your kind welcome and very glad to be one of the
>> > committers of Flink community.
>> >
>> > Best
>> > Yun Tang
>> >
>> > 
>> > From: Congxian Qiu 
>> > Sent: Wednesday, September 16, 2020 13:10
>> > To: dev@flink.apache.org 
>> > Cc: Zhijiang ; tangyun ;
>> > Yun Tang 
>> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang
>> >
>> > Congratulations!
>> > Best,
>> > Congxian
>> >
>> >
>> > Guowei Ma mailto:guowei@gmail.com>>
>> > 于2020年9月16日周三 下午12:37写道：
>> > Congratulations :)
>> >
>> > Best,
>> > Guowei
>> >
>> >
>> > On Wed, Sep 16, 2020 at 11:54 AM Zhijiang
>> > mailto:wangzhijiang...@aliyun.com>.invalid>
>> > wrote:
>> >
>> > > Congratulations and welcome, Yun!
>> > >
>> > >
>> > > --
>> > > From:Jark Wu mailto:imj...@gmail.com>>
>> > > Send Time:2020年9月16日(星期三) 11:35
>> > > To:dev mailto:dev@flink.apache.org>>
>> > > Cc:tangyun mailto:tang...@apache.org>>; Yun Tang <
>> > myas...@live.com>
>> > > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang
>> > >
>> > > Congratulations Yun!
>> > >
>> > > On Wed, 16 Sep 2020 at 10:40, Rui Li > > lirui.fu...@gmail.com>> wrote:
>> > >
>> > > > Congratulations Yun!
>> > > >
>> > > > On Wed, Sep 16, 2020 at 10:20 AM Paul Lam > > > wrote:
>> > > >
>> > > > > Congrats, Yun! Well deserved!
>> > > > >
>> > > > > Best,
>> > > > > Paul Lam
>> > > > >
>> > > > > > 2020年9月15日 19:14，Yang Wang > > danrtsey...@gmail.com>> 写道：
>> > > > > >
>> > > > > > Congratulations, Yun!
>> > > > > >
>> > > > > > Best,
>> > > > > > Yang
>> > > > > >
>> > > > > > Leonard Xu mailto:xbjt...@gmail.com>>
>> > 于2020年9月15日周二 下午7:11写道：
>> > > > > >
>> > > > > >> Congrats, Yun!
>> > > > > >>
>> > > > > >> Best,
>> > > > > >> Leonard
>> > > > > >>> 在 2020年9月15日，19:01，Yangze Guo > > karma...@gmail.com>> 写道：
>> > > > > >>>
>> > > > > >>> Congrats, Yun!
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > Best regards!
>> > > > Rui Li
>> > > >
>> > >
>> > >
>> >
>>

Re:Re: Re: [ANNOUNCE] New PMC member: Dian Fu

[Haibo Sun]

Congratulations Dian !



Best,
Haibo

At 2020-08-27 18:03:38, "Zhijiang"  wrote:
>Congrats, Dian!
>
>
>--
>From:Yun Gao 
>Send Time:2020年8月27日(星期四) 17:44
>To:dev ; Dian Fu ; user 
>; user-zh 
>Subject:Re: Re: [ANNOUNCE] New PMC member: Dian Fu
>
>Congratulations Dian !
>
> Best
> Yun
>
>
>--
>Sender:Marta Paes Moreira
>Date:2020/08/27 17:42:34
>Recipient:Yuan Mei
>Cc:Xingbo Huang; jincheng sun; 
>dev; Dian Fu; 
>user; user-zh
>Theme:Re: [ANNOUNCE] New PMC member: Dian Fu
>
>Congrats, Dian!
>On Thu, Aug 27, 2020 at 11:39 AM Yuan Mei  wrote:
>
>Congrats!
>On Thu, Aug 27, 2020 at 5:38 PM Xingbo Huang  wrote:
>
>Congratulations Dian!
>
>Best,
>Xingbo
>jincheng sun  于2020年8月27日周四 下午5:24写道：
>
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Dian Fu is now part of 
>the Apache Flink Project Management Committee (PMC).
>
>Dian Fu has been very active on PyFlink component, working on various 
>important features, such as the Python UDF and Pandas integration, and keeps 
>checking and voting for our releases, and also has successfully produced two 
>releases(1.9.3&1.11.1) as RM, currently working as RM to push forward the 
>release of Flink 1.12.
>
>Please join me in congratulating Dian Fu for becoming a Flink PMC Member!
>
>Best,
>Jincheng(on behalf of the Flink PMC)
>

Re:[ANNOUNCE] Yu Li is now part of the Flink PMC

[Haibo Sun]

Congratulations Yu!


Best,
Haibo




At 2020-06-17 09:15:02, "jincheng sun"  wrote:
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Yu Li is now
>part of the Apache Flink Project Management Committee (PMC).
>
>Yu Li has been very active on Flink's Statebackend component, working on
>various improvements, for example the RocksDB memory management for 1.10.
>and keeps checking and voting for our releases, and also has successfully
>produced two releases(1.10.0&1.10.1) as RM.
>
>Congratulations & Welcome Yu Li!
>
>Best,
>Jincheng (on behalf of the Flink PMC)

[jira] [Created] (FLINK-16174) Add a better tryYield() method to MailboxExecutor to return the lowest priority of the remaining mails

[Haibo Sun (Jira)]

Haibo Sun created FLINK-16174:
-

 Summary: Add a better tryYield() method to MailboxExecutor to 
return the lowest priority of the remaining mails
 Key: FLINK-16174
 URL: https://issues.apache.org/jira/browse/FLINK-16174
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Task
Affects Versions: 1.10.0
Reporter: Haibo Sun


Currently, we use chainIndex as the priority to create MailboxExecutor to 
process its mails. When MailboxExecutor#tryYield  is called to process mails, 
it will take the mails of this operator and all downstream operators in the 
chain. But sometimes, after calling MailboxExecutor#tryYield, we need to know 
whether there is any mail of the current operator in the mailbox, which can 
simplify some operations. 

For example, when we close a operator in runtime, after quiescing the 
processing time service and waiting for its running timers to finish, if there 
is no mail of the current operator in the mailbox,  we call 
StreamOperator#close to close the operator. Then the runtime code of closing a 
operator can be simplified as follows.
{code:java}
quiesceProcessingTimeService().get();
while (mailboxExecuto.betterTryYield() <= self.priority) {}
closeOperator(actionExecutor);
{code}

With the existing #tryYield method, if the following simplified code is used to 
close a operator, then when a downstream operator is implemented like 
MailboxOperatorTest.ReplicatingMail, the tryyield() loop will 
 be prevented from exiting, which results deadlock.

{code:java}
quiesceProcessingTimeService().get();
while (mailboxExecuto.tryYield()) {}
closeOperator(actionExecutor);
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re:[ANNOUNCE] Apache Flink 1.10.0 released

[Haibo Sun]

Thanks Gary & Yu. Great work!


Best,
Haibo


At 2020-02-12 21:31:00, "Yu Li"  wrote:
>The Apache Flink community is very happy to announce the release of Apache
>Flink 1.10.0, which is the latest major release.
>
>Apache Flink® is an open-source stream processing framework for
>distributed, high-performing, always-available, and accurate data streaming
>applications.
>
>The release is available for download at:
>https://flink.apache.org/downloads.html
>
>Please check out the release blog post for an overview of the improvements
>for this new major release:
>https://flink.apache.org/news/2020/02/11/release-1.10.0.html
>
>The full release notes are available in Jira:
>https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345845
>
>We would like to thank all contributors of the Apache Flink community who
>made this release possible!
>
>Cheers,
>Gary & Yu

Re:Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink

[Haibo Sun]

+1 (non-binding)


Best,
Haibo

At 2020-01-13 11:36:12, "Yun Gao"  wrote:
>+1 (non-binding).
>
>Very thanks for introducing this topic back, and it should be able to bring 
>improvements in the discussed scenarios. 
>
>Best,
>Yun
>
>
>--
>From:Arvid Heise 
>Send Time:2020 Jan. 10 (Fri.) 16:48
>To:dev ; Zhijiang 
>Subject:Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink
>
>non-binding +1
>
>On Fri, Jan 10, 2020 at 9:11 AM Zhijiang 
>wrote:
>
>> +1, it is really nice to have the N-Ary stream operator which is
>> meaningful in some scenarios.
>>
>> best,
>> Zhijiang
>>
>>
>> --
>> From:Jingsong Li 
>> Send Time:2020 Jan. 10 (Fri.) 11:00
>> To:dev 
>> Subject:Re: [VOTE] FLIP-92: Add N-Ary Stream Operator in Flink
>>
>> +1 non-binding to the N-Ary Stream Operator. Thanks Piotr for driving.
>> Looks like the previous FLIP-92 did not change the "Next FLIP Number" in
>> FLIP page.
>>
>> Best,
>> Jingsong Lee
>>
>> On Fri, Jan 10, 2020 at 8:40 AM Benchao Li  wrote:
>>
>> > Hi Piotr,
>> >
>> > It seems that we have the 'FLIP-92' already.
>> > see:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+JDBC+catalog+and+Postgres+catalog
>> >
>> >
>> > Piotr Nowojski  于2020年1月9日周四 下午11:25写道：
>> >
>> > > Hi,
>> > >
>> > > I would like to start a vote for adding the N-Ary Stream Operator in
>> > Flink
>> > > as discussed in the discussion thread [1].
>> > >
>> > > This vote will be opened at least until Wednesday, January 15th 8:00
>> UTC.
>> > >
>> > > Piotrek
>> > >
>> > > [1]
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-N-Ary-Stream-Operator-td11341.html
>> > > <
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-N-Ary-Stream-Operator-td11341.html
>> > > >
>> >
>> >
>> >
>> > --
>> >
>> > Benchao Li
>> > School of Electronics Engineering and Computer Science, Peking University
>> > Tel:+86-15650713730
>> > Email: libenc...@gmail.com; libenc...@pku.edu.cn
>> >
>>
>>
>> --
>> Best, Jingsong Lee
>>
>>

Re:[DISCUSS] Deal with the timers better before closing operators

[Haibo Sun]

I found that the format of the mail was out of order. Please look at the 
document: 
https://docs.google.com/document/d/1jCKan5LGlkBZr2seUdwTsy-biu8Sz0xyQwqizCUZ2uk/edit?usp=sharing


Best,
Haibo
At 2019-12-16 21:29:15, "Haibo Sun"  wrote:
>Hi, all
>
>
>I want to bring up a discussion about how to deal with the pending(registered 
>but not scheduled for execution) 
>timers better before closing operator.
>
>
>Introduction: From my understanding, there are two types of timers in Flink: 
>1) One is one-shot or periodic
>timers registered with ProcessingTimeService. In terms of the underlying 
>implementation, each timer corresponds
>to a Runnable and is executed through a thread pool. 2) The other is 
>event-time or processing-time timers
>registered with InternalTimerService, which can be stateful. Event-time timers 
>are triggered by watermark,
>and processing-time timers registered with the same InternalTimerService 
>instance are triggered by a real
>timer registered with ProcessingTimeService. For the convenience of later 
>expression, here we define the
>first type as "physical timer", and the second type (including timers 
>registered with the interfaces built on
>InternalTimerService such as api.windowing.triggers.TriggerContext) as 
>"logical timer".
>
>
>Why and how to deal with the physical timers better before closing operators?
>Currently, after the operator is closed, it is still allowed to fire the 
>physical timers registered by it, which may
>output data again, making the close semantics of operators on the operator 
>chain not strict (the strict semantics
>should no longer output data after closing). So we need to explicitly let all 
>physical timers done before we close
>the operator. Because physical timers are registered by the operator, runtime 
>cannot, at its own discretion, cancels
>or triggers them before closing the operator. For example, If runtime cancels 
>one of the physical timers registered
>but not scheduled for execution before closing the operator, a deadlock may 
>occur when the operator waits in
>the "close()" method for the physical timer to be fired. Overall, we need to 
>expose some capabilities to the operators
>so that they can decide whether to cancel or trigger the physical timers 
>registered but not scheduled for execution
>before closing. 
>
>
>About how to expose such capabilities, we may have the following options. For 
>a periodic physical timer, it should
>only need the "cancel" action and the operator can cancel it by calling the 
>"ScheduledFuture#cancel()" method ,
>so here we should not need to consider it.
>
>
>Option 1:  The physical timer is of the ScheduledFuture type, and there is 
>already the "#cancel()" method in
>ScheduledFuture. We just need to add the "#triggerIfNotExecuted()" method, and 
>the changes mainly include:
>1) Adds the following TimerScheduledFuture interface that extended 
> ScheduledFuture, and changes the return
>   type of "ProcessingTimerService#registerTimer" to TimerScheduledFuture. 
> If the operator wants to trigger a
>   pending physical timer before closing, 
> "TimerScheduledFuture#triggerIfNotExecuted()"
>   need to be called explicitly in its "endInput" method.
>public interface TimerScheduledFuture extends TimerScheduledFuture {
>void triggerIfNotExecuted(long timestamp);
>}
>2) For the pending physical timers that are not cancelled or triggered by 
> the operator before closing, runtime will
>cancel them but wait for those in executing to finish to maintain 
> backward compatibility.
>
>
>Option 2:  When the operator registers a physical timer, it passes a callback 
>parameter of Runnable type
>(like ProcessingTimeCallback), which is called back by runtime before closing 
>the operator to let the operator
>deal with those pending physical timers. The changes mainly include:
>1) Applies all changes from option 1
>2) Adds the following overloaded method to the ProcessingTimerService 
> interface.
>ScheduledFuture registerTimer(long timestamp, ProcessingTimeCallback 
>target, Runnable actionBeforeClosingOperator);
>
>
>How to deal with the logical timers better before closing operators?
>For logic timers, that capabilities also should be exposed to operators. 
>According to my understanding, the logical
>timer can be registered through the three interfaces: api.TimerService, 
>api.windowing.triggers.TriggerContext,
>api.operators.InternalTimerService. And the first two are implemented using 
>api.operators.InternalTimerService.
>As can be known from the declaration of t

[DISCUSS] Deal with the timers better before closing operators

[Haibo Sun]

Hi, all


I want to bring up a discussion about how to deal with the pending(registered 
but not scheduled for execution) 
timers better before closing operator.


Introduction: From my understanding, there are two types of timers in Flink: 1) 
One is one-shot or periodic
timers registered with ProcessingTimeService. In terms of the underlying 
implementation, each timer corresponds
to a Runnable and is executed through a thread pool. 2) The other is event-time 
or processing-time timers
registered with InternalTimerService, which can be stateful. Event-time timers 
are triggered by watermark,
and processing-time timers registered with the same InternalTimerService 
instance are triggered by a real
timer registered with ProcessingTimeService. For the convenience of later 
expression, here we define the
first type as "physical timer", and the second type (including timers 
registered with the interfaces built on
InternalTimerService such as api.windowing.triggers.TriggerContext) as "logical 
timer".


Why and how to deal with the physical timers better before closing operators?
Currently, after the operator is closed, it is still allowed to fire the 
physical timers registered by it, which may
output data again, making the close semantics of operators on the operator 
chain not strict (the strict semantics
should no longer output data after closing). So we need to explicitly let all 
physical timers done before we close
the operator. Because physical timers are registered by the operator, runtime 
cannot, at its own discretion, cancels
or triggers them before closing the operator. For example, If runtime cancels 
one of the physical timers registered
but not scheduled for execution before closing the operator, a deadlock may 
occur when the operator waits in
the "close()" method for the physical timer to be fired. Overall, we need to 
expose some capabilities to the operators
so that they can decide whether to cancel or trigger the physical timers 
registered but not scheduled for execution
before closing. 


About how to expose such capabilities, we may have the following options. For a 
periodic physical timer, it should
only need the "cancel" action and the operator can cancel it by calling the 
"ScheduledFuture#cancel()" method ,
so here we should not need to consider it.


Option 1:  The physical timer is of the ScheduledFuture type, and there is 
already the "#cancel()" method in
ScheduledFuture. We just need to add the "#triggerIfNotExecuted()" method, and 
the changes mainly include:
1) Adds the following TimerScheduledFuture interface that extended 
ScheduledFuture, and changes the return
   type of "ProcessingTimerService#registerTimer" to TimerScheduledFuture. 
If the operator wants to trigger a
   pending physical timer before closing, 
"TimerScheduledFuture#triggerIfNotExecuted()"
   need to be called explicitly in its "endInput" method.
public interface TimerScheduledFuture extends TimerScheduledFuture {
void triggerIfNotExecuted(long timestamp);
}
2) For the pending physical timers that are not cancelled or triggered by 
the operator before closing, runtime will
cancel them but wait for those in executing to finish to maintain 
backward compatibility.


Option 2:  When the operator registers a physical timer, it passes a callback 
parameter of Runnable type
(like ProcessingTimeCallback), which is called back by runtime before closing 
the operator to let the operator
deal with those pending physical timers. The changes mainly include:
1) Applies all changes from option 1
2) Adds the following overloaded method to the ProcessingTimerService 
interface.
ScheduledFuture registerTimer(long timestamp, ProcessingTimeCallback target, 
Runnable actionBeforeClosingOperator);


How to deal with the logical timers better before closing operators?
For logic timers, that capabilities also should be exposed to operators. 
According to my understanding, the logical
timer can be registered through the three interfaces: api.TimerService, 
api.windowing.triggers.TriggerContext,
api.operators.InternalTimerService. And the first two are implemented using 
api.operators.InternalTimerService.
As can be known from the declaration of the registration method, it does not 
return the logical timer object, which
is managed uniformly by the implementation of InternalTimerService.


1. For the api.operators.InternalTimerService interface, we may make the 
following changes:
1) Adds the following overloaded method to the InternalTimerService 
interface. 
void registerProcessingTimeTimer(N namespace, long time, 
TimerActionOnOperatorClose timerActionOnOperatorClose);
/**
 * It defines how to handle the timers not scheduled for execution before the 
operator is closed.
 *
 */
enum TimerActionOnOperatorClose { 
CANCEL, 
TRIGGER_WITH_ORIGINAL_TIMESTAMP,
TRIGGER_WITH_CURRENT_TIMESTAMP,
TRIGGER_WITH_MAX_TIMESTAMP
}
2) Changes the serialization of stateful timers to

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

[Haibo Sun]

Congratulations, Zhu Zhu!


Best,
Haibo
在 2019-12-16 11:16:55，"Yun Tang"  写道：
>Congratulations ZZ
>
>Best
>Yun Tang
>
>From: Guowei Ma 
>Sent: Monday, December 16, 2019 11:15
>To: dev 
>Subject: Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer
>
>Congrats Zhuzhu!
>Best,
>Guowei
>
>
>Zhenghua Gao  于2019年12月16日周一 上午10:47写道：
>
>> Congrats!
>>
>> *Best Regards,*
>> *Zhenghua Gao*
>>
>>
>> On Mon, Dec 16, 2019 at 10:36 AM Biao Liu  wrote:
>>
>> > Congrats Zhu Zhu!
>> >
>> > Thanks,
>> > Biao /'bɪ.aʊ/
>> >
>> >
>> >
>> > On Mon, 16 Dec 2019 at 10:23, Congxian Qiu 
>> wrote:
>> >
>> > > Congrats, Zhu Zhu!
>> > >
>> > > Best,
>> > > Congxian
>> > >
>> > >
>> > > aihua li  于2019年12月16日周一 上午10:16写道：
>> > >
>> > > > Congratulations， zhuzhu！
>> > > >
>> > > > > 在 2019年12月16日，上午10:04，Jingsong Li  写道：
>> > > > >
>> > > > > Congratulations Zhu Zhu!
>> > > > >
>> > > > > Best,
>> > > > > Jingsong Lee
>> > > > >
>> > > > > On Mon, Dec 16, 2019 at 10:01 AM Yang Wang 
>> > > > wrote:
>> > > > >
>> > > > >> Congratulations, Zhu Zhu!
>> > > > >>
>> > > > >> wenlong.lwl  于2019年12月16日周一 上午9:56写道：
>> > > > >>
>> > > > >>> Congratulations, Zhu Zhu!
>> > > > >>>
>> > > > >>> On Mon, 16 Dec 2019 at 09:14, Leonard Xu 
>> > wrote:
>> > > > >>>
>> > > >  Congratulations, Zhu Zhu ! !
>> > > > 
>> > > >  Best,
>> > > >  Leonard Xu
>> > > > 
>> > > > > On Dec 16, 2019, at 07:53, Becket Qin 
>> > > wrote:
>> > > > >
>> > > > > Congrats, Zhu Zhu!
>> > > > >
>> > > > > On Sun, Dec 15, 2019 at 10:26 PM Dian Fu <
>> dian0511...@gmail.com>
>> > > > >>> wrote:
>> > > > >
>> > > > >> Congrats Zhu Zhu!
>> > > > >>
>> > > > >>> 在 2019年12月15日，下午6:23，Zhu Zhu  写道：
>> > > > >>>
>> > > > >>> Thanks everyone for the warm welcome!
>> > > > >>> It's my honor and pleasure to improve Flink with all of you
>> in
>> > > the
>> > > > >>> community!
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Zhu Zhu
>> > > > >>>
>> > > > >>> Benchao Li  于2019年12月15日周日 下午3:54写道：
>> > > > >>>
>> > > >  Congratulations!:)
>> > > > 
>> > > >  Hequn Cheng  于2019年12月15日周日
>> 上午11:47写道：
>> > > > 
>> > > > > Congrats, Zhu Zhu!
>> > > > >
>> > > > > Best, Hequn
>> > > > >
>> > > > > On Sun, Dec 15, 2019 at 6:11 AM Shuyi Chen <
>> > suez1...@gmail.com
>> > > >
>> > > >  wrote:
>> > > > >
>> > > > >> Congratulations!
>> > > > >>
>> > > > >> On Sat, Dec 14, 2019 at 7:59 AM Rong Rong <
>> > > walter...@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Congrats Zhu Zhu :-)
>> > > > >>>
>> > > > >>> --
>> > > > >>> Rong
>> > > > >>>
>> > > > >>> On Sat, Dec 14, 2019 at 4:47 AM tison <
>> > wander4...@gmail.com>
>> > > >  wrote:
>> > > > >>>
>> > > >  Congratulations!:)
>> > > > 
>> > > >  Best,
>> > > >  tison.
>> > > > 
>> > > > 
>> > > >  OpenInx  于2019年12月14日周六 下午7:34写道：
>> > > > 
>> > > > > Congrats Zhu Zhu!
>> > > > >
>> > > > > On Sat, Dec 14, 2019 at 2:38 PM Jeff Zhang <
>> > > zjf...@gmail.com
>> > > > >>>
>> > > > > wrote:
>> > > > >
>> > > > >> Congrats, Zhu Zhu!
>> > > > >>
>> > > > >> Paul Lam  于2019年12月14日周六
>> > > 上午10:29写道：
>> > > > >>
>> > > > >>> Congrats Zhu Zhu!
>> > > > >>>
>> > > > >>> Best,
>> > > > >>> Paul Lam
>> > > > >>>
>> > > > >>> Kurt Young  于2019年12月14日周六
>> > 上午10:22写道：
>> > > > >>>
>> > > >  Congratulations Zhu Zhu!
>> > > > 
>> > > >  Best,
>> > > >  Kurt
>> > > > 
>> > > > 
>> > > >  On Sat, Dec 14, 2019 at 10:04 AM jincheng sun <
>> > > > >> sunjincheng...@gmail.com>
>> > > >  wrote:
>> > > > 
>> > > > > Congrats ZhuZhu and welcome on board!
>> > > > >
>> > > > > Best,
>> > > > > Jincheng
>> > > > >
>> > > > >
>> > > > > Jark Wu  于2019年12月14日周六
>> 上午9:55写道：
>> > > > >
>> > > > >> Congratulations, Zhu Zhu!
>> > > > >>
>> > > > >> Best,
>> > > > >> Jark
>> > > > >>
>> > > > >> On Sat, 14 Dec 2019 at 08:20, Yangze Guo <
>> > > > >> karma...@gmail.com
>> > > > 
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Congrats, ZhuZhu!
>> > > > >>>
>> > > > >>> Bowen Li  于 2019年12月14日周六
>> > > > > 上午5:37写道：
>> >

Re:Re: [ANNOUNCE] Becket Qin joins the Flink PMC

[Haibo Sun]

Congrats Becket!


Best,
Haibo
在 2019-10-29 09:52:50，"Congxian Qiu"  写道：
>Congratulations Becket!
>
>Best,
>Congxian
>
>
>Wei Zhong  于2019年10月29日周二 上午9:42写道：
>
>> Congratulations Becket!
>>
>> Best,
>> Wei
>>
>> > 在 2019年10月29日，09:36，Paul Lam  写道：
>> >
>> > Congrats Becket!
>> >
>> > Best,
>> > Paul Lam
>> >
>> >> 在 2019年10月29日，02:18，Xingcan Cui  写道：
>> >>
>> >> Congratulations, Becket!
>> >>
>> >> Best,
>> >> Xingcan
>> >>
>> >>> On Oct 28, 2019, at 1:23 PM, Xuefu Z  wrote:
>> >>>
>> >>> Congratulations, Becket!
>> >>>
>> >>> On Mon, Oct 28, 2019 at 10:08 AM Zhu Zhu  wrote:
>> >>>
>>  Congratulations Becket!
>> 
>>  Thanks,
>>  Zhu Zhu
>> 
>>  Peter Huang  于2019年10月29日周二 上午1:01写道：
>> 
>> > Congratulations Becket Qin!
>> >
>> >
>> > Best Regards
>> > Peter Huang
>> >
>> > On Mon, Oct 28, 2019 at 9:19 AM Rong Rong 
>> wrote:
>> >
>> >> Congratulations Becket!!
>> >>
>> >> --
>> >> Rong
>> >>
>> >> On Mon, Oct 28, 2019, 7:53 AM Jark Wu  wrote:
>> >>
>> >>> Congratulations Becket!
>> >>>
>> >>> Best,
>> >>> Jark
>> >>>
>> >>> On Mon, 28 Oct 2019 at 20:26, Benchao Li 
>>  wrote:
>> >>>
>>  Congratulations Becket.
>> 
>>  Dian Fu  于2019年10月28日周一 下午7:22写道：
>> 
>> > Congrats, Becket.
>> >
>> >> 在 2019年10月28日，下午6:07，Fabian Hueske  写道：
>> >>
>> >> Hi everyone,
>> >>
>> >> I'm happy to announce that Becket Qin has joined the Flink PMC.
>> >> Let's congratulate and welcome Becket as a new member of the
>> > Flink
>> >>> PMC!
>> >>
>> >> Cheers,
>> >> Fabian
>> >
>> >
>> 
>>  --
>> 
>>  Benchao Li
>>  School of Electronics Engineering and Computer Science, Peking
>> >> University
>>  Tel:+86-15650713730
>>  Email: libenc...@gmail.com; libenc...@pku.edu.cn
>> 
>> >>>
>> >>
>> >
>> 
>> >>>
>> >>>
>> >>> --
>> >>> Xuefu Zhang
>> >>>
>> >>> "In Honey We Trust!"
>> >>
>> >
>>
>>

Re:Re: [VOTE] Accept Stateful Functions into Apache Flink

[Haibo Sun]

+1 (non-binding)Best,
Haibo


At 2019-10-23 09:07:41, "Becket Qin"  wrote:
>+1 (binding)
>
>Thanks,
>
>Jiangjie (Becket) Qin
>
>On Tue, Oct 22, 2019 at 11:44 PM Tzu-Li (Gordon) Tai 
>wrote:
>
>> +1 (binding)
>>
>> Gordon
>>
>> On Tue, Oct 22, 2019, 10:58 PM Zhijiang > .invalid>
>> wrote:
>>
>> > +1 (non-binding)
>> >
>> > Best,
>> > Zhijiang
>> >
>> >
>> > --
>> > From:Zhu Zhu 
>> > Send Time:2019 Oct. 22 (Tue.) 16:33
>> > To:dev 
>> > Subject:Re: [VOTE] Accept Stateful Functions into Apache Flink
>> >
>> > +1 (non-binding)
>> >
>> > Thanks,
>> > Zhu Zhu
>> >
>> > Biao Liu  于2019年10月22日周二 上午11:06写道：
>> >
>> > > +1 (non-binding)
>> > >
>> > > Thanks,
>> > > Biao /'bɪ.aʊ/
>> > >
>> > >
>> > >
>> > > On Tue, 22 Oct 2019 at 10:26, Jark Wu  wrote:
>> > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Best,
>> > > > Jark
>> > > >
>> > > > On Tue, 22 Oct 2019 at 09:38, Hequn Cheng 
>> > wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > Best, Hequn
>> > > > >
>> > > > > On Tue, Oct 22, 2019 at 9:21 AM Dian Fu 
>> > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > Regards,
>> > > > > > Dian
>> > > > > >
>> > > > > > > 在 2019年10月22日，上午9:10，Kurt Young  写道：
>> > > > > > >
>> > > > > > > +1 (binding)
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Kurt
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, Oct 22, 2019 at 12:56 AM Fabian Hueske <
>> > fhue...@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > >> +1 (binding)
>> > > > > > >>
>> > > > > > >> Am Mo., 21. Okt. 2019 um 16:18 Uhr schrieb Thomas Weise <
>> > > > > t...@apache.org
>> > > > > > >:
>> > > > > > >>
>> > > > > > >>> +1 (binding)
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> On Mon, Oct 21, 2019 at 7:10 AM Timo Walther <
>> > twal...@apache.org
>> > > >
>> > > > > > wrote:
>> > > > > > >>>
>> > > > > >  +1 (binding)
>> > > > > > 
>> > > > > >  Thanks,
>> > > > > >  Timo
>> > > > > > 
>> > > > > > 
>> > > > > >  On 21.10.19 15:59, Till Rohrmann wrote:
>> > > > > > > +1 (binding)
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > > Till
>> > > > > > >
>> > > > > > > On Mon, Oct 21, 2019 at 12:13 PM Robert Metzger <
>> > > > > rmetz...@apache.org
>> > > > > > >>>
>> > > > > >  wrote:
>> > > > > > >
>> > > > > > >> +1 (binding)
>> > > > > > >>
>> > > > > > >> On Mon, Oct 21, 2019 at 12:06 PM Stephan Ewen <
>> > > se...@apache.org
>> > > > >
>> > > > > > >>> wrote:
>> > > > > > >>
>> > > > > > >>> This is the official vote whether to accept the Stateful
>> > > > > Functions
>> > > > > > >>> code
>> > > > > > >>> contribution to Apache Flink.
>> > > > > > >>>
>> > > > > > >>> The current Stateful Functions code, documentation, and
>> > > website
>> > > > > can
>> > > > > > >>> be
>> > > > > > >>> found here:
>> > > > > > >>> https://statefun.io/
>> > > > > > >>> https://github.com/ververica/stateful-functions
>> > > > > > >>>
>> > > > > > >>> This vote should capture whether the Apache Flink
>> community
>> > > is
>> > > > > >  interested
>> > > > > > >>> in accepting, maintaining, and evolving Stateful
>> Functions.
>> > > > > > >>>
>> > > > > > >>> Reiterating my original motivation, I believe that this
>> > > project
>> > > > > is
>> > > > > > >> a
>> > > > > > >> great
>> > > > > > >>> match for Apache Flink, because it helps Flink to grow
>> the
>> > > > > > >> community
>> > > > > > >> into a
>> > > > > > >>> new set of use cases. We see current users interested in
>> > such
>> > > > use
>> > > > > >  cases,
>> > > > > > >>> but they are not well supported by Flink as it currently
>> > is.
>> > > > > > >>>
>> > > > > > >>> I also personally commit to put time into making sure
>> this
>> > > > > > >> integrates
>> > > > > > >> well
>> > > > > > >>> with Flink and that we grow contributors and committers
>> to
>> > > > > maintain
>> > > > > >  this
>> > > > > > >>> new component well.
>> > > > > > >>>
>> > > > > > >>> This is a "Adoption of a new Codebase" vote as per the
>> > Flink
>> > > > > bylaws
>> > > > > >  [1].
>> > > > > > >>> Only PMC votes are binding. The vote will be open at
>> least
>> > 6
>> > > > days
>> > > > > > >>> (excluding weekends), meaning until Tuesday Oct.29th
>> 12:00
>> > > UTC,
>> > > > > or
>> > > > > >  until
>> > > > > > >> we
>> > > > > > >>> achieve the 2/3rd majority.
>> > > > > > >>>
>> > > > > > >>> Happy voting!
>> > > > > > >>>
>> > > > > > >>> Best,
>> > > > > > >>> Stephan
>> > > > > > >>>
>> > > > > > >>> [1]
>> > > > > > >>>
>> > > > > > >>
>> > > > > > 
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
>> > > > > > 
>> > > > > > 
>> > > >

[jira] [Created] (FLINK-14239) Emitting the max watermark in StreamSource#run may cause it to arrive the downstream early

[Haibo Sun (Jira)]

Haibo Sun created FLINK-14239:
-

 Summary: Emitting the max watermark in StreamSource#run may cause 
it to arrive the downstream early
 Key: FLINK-14239
 URL: https://issues.apache.org/jira/browse/FLINK-14239
 Project: Flink
  Issue Type: Bug
Reporter: Haibo Sun
 Fix For: 1.10.0


For {{Source}}, the max watermark is emitted in {{StreamSource#run}} currently. 
If some records are also output in {{close}} of {{RichSourceFunction}}, then 
the max watermark will reach the downstream operator before these records.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14231) Support the timers of the upstream operator with endInput properly

[Haibo Sun (Jira)]

Haibo Sun created FLINK-14231:
-

 Summary: Support the timers of the upstream operator with endInput 
properly
 Key: FLINK-14231
 URL: https://issues.apache.org/jira/browse/FLINK-14231
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Haibo Sun






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14230) Change the endinput call of the downstream operator to after the upstream operator closes

[Haibo Sun (Jira)]

Haibo Sun created FLINK-14230:
-

 Summary:  Change the endinput call of the downstream operator to 
after the upstream operator closes
 Key: FLINK-14230
 URL: https://issues.apache.org/jira/browse/FLINK-14230
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Haibo Sun






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14229) Change the endinput call of the downstream operator to after the upstream operator closes

[Haibo Sun (Jira)]

Haibo Sun created FLINK-14229:
-

 Summary:  Change the endinput call of the downstream operator to 
after the upstream operator closes
 Key: FLINK-14229
 URL: https://issues.apache.org/jira/browse/FLINK-14229
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Haibo Sun






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14228) The runtime support for Bounded[One|Multi]Input#endInput does not properly implement their semantics

[Haibo Sun (Jira)]

Haibo Sun created FLINK-14228:
-

 Summary: The runtime support for Bounded[One|Multi]Input#endInput 
does not properly implement their semantics
 Key: FLINK-14228
 URL: https://issues.apache.org/jira/browse/FLINK-14228
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.9.0
Reporter: Haibo Sun
 Fix For: 1.10.0


Currently, the runtime support implementation of 
{{Bounded[One|Multi]Input#endInput}} has the following problems:1.The runtime 
are propagating endInput immediately on the operator chain when input of the 
head operator is finished.Because some operators flush the buffered data in 
close, the downstream operators still receive records after executing 
endInput.This need the operator to flush the buffered data in "endInput" 
instead of "close", like the PRs for fixing issue#13491 and 
issue#13376.2.Timers are not taken into account.
 close tells the operator to finish all its processing and flush output (all 
remaining buffered data), while endInput indicates that no more data will 
arrive on some input of the operator. That is to say, for the non-tail 
operators on the operator chain, when the upstream operator is closed, the 
input of its downstream operator arrives at the end. So for an operator chain 
OP1 -> OP2 -> ... ,  the logic should be:



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re:[ANNOUNCE] Apache Flink 1.9.0 released

[Haibo Sun]

Great news! Thanks Gordon and Kurt!Best,
Haibo

At 2019-08-22 20:03:26, "Tzu-Li (Gordon) Tai"  wrote:
>The Apache Flink community is very happy to announce the release of Apache
>Flink 1.9.0, which is the latest major release.
>
>Apache Flink® is an open-source stream processing framework for
>distributed, high-performing, always-available, and accurate data streaming
>applications.
>
>The release is available for download at:
>https://flink.apache.org/downloads.html
>
>Please check out the release blog post for an overview of the improvements
>for this new major release:
>https://flink.apache.org/news/2019/08/22/release-1.9.0.html
>
>The full release notes are available in Jira:
>https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344601
>
>We would like to thank all contributors of the Apache Flink community who
>made this release possible!
>
>Cheers,
>Gordon

Re:Re: [ANNOUNCE] Hequn becomes a Flink committer

[Haibo Sun]

Congratulations!


Best,
Haibo
At 2019-08-08 02:08:21, "Yun Tang"  wrote:
>Congratulations Hequn.
>
>Best
>Yun Tang
>
>From: Rong Rong 
>Sent: Thursday, August 8, 2019 0:41
>Cc: dev ; user 
>Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>Congratulations Hequn, well deserved!
>
>--
>Rong
>
>On Wed, Aug 7, 2019 at 8:30 AM mailto:xingc...@gmail.com>> 
>wrote:
>
>Congratulations, Hequn!
>
>
>
>From: Xintong Song mailto:tonysong...@gmail.com>>
>Sent: Wednesday, August 07, 2019 10:41 AM
>To: dev@flink.apache.org
>Cc: user mailto:u...@flink.apache.org>>
>Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>
>
>Congratulations~!
>
>
>Thank you~
>
>Xintong Song
>
>
>
>
>
>On Wed, Aug 7, 2019 at 4:00 PM vino yang 
>mailto:yanghua1...@gmail.com>> wrote:
>
>Congratulations!
>
>highfei2...@126.com 
>mailto:highfei2...@126.com>> 于2019年8月7日周三 下午7:09写道：
>
>> Congrats Hequn!
>>
>> Best,
>> Jeff Yang
>>
>>
>>  Original Message 
>> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>> From: Piotr Nowojski
>> To: JingsongLee
>> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
>> ,user
>>
>>
>> Congratulations :)
>>
>> On 7 Aug 2019, at 12:09, JingsongLee 
>> mailto:lzljs3620...@aliyun.com>> wrote:
>>
>> Congrats Hequn!
>>
>> Best,
>> Jingsong Lee
>>
>> --
>> From:Biao Liu mailto:mmyy1...@gmail.com>>
>> Send Time:2019年8月7日(星期三) 12:05
>> To:Zhu Zhu mailto:reed...@gmail.com>>
>> Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff Zhang 
>> mailto:zjf...@gmail.com>>; Paul
>> Lam mailto:paullin3...@gmail.com>>; jincheng sun 
>> mailto:sunjincheng...@gmail.com>>; dev
>> mailto:dev@flink.apache.org>>; user 
>> mailto:u...@flink.apache.org>>
>> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>>
>> Congrats Hequn!
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu 
>> mailto:reed...@gmail.com>> wrote:
>> Congratulations to Hequn!
>>
>> Thanks,
>> Zhu Zhu
>>
>> Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 
>> 下午5:16写道：
>> Congrats Hequn!
>>
>> Best,
>> tison.
>>
>>
>> Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道：
>> Congrats Hequn!
>>
>> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
>> 下午5:08写道：
>> Congrats Hequn! Well deserved!
>>
>> Best,
>> Paul Lam
>>
>> 在 2019年8月7日，16:28，jincheng sun 
>> mailto:sunjincheng...@gmail.com>> 写道：
>>
>> Hi everyone,
>>
>> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
>> to become a committer of the Flink project.
>>
>> Hequn has been contributing to Flink for many years, mainly working on
>> SQL/Table API features. He's also frequently helping out on the user
>> mailing lists and helping check/vote the release.
>>
>> Congratulations Hequn!
>>
>> Best, Jincheng
>> (on behalf of the Flink PMC)
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>>

Re:[ANNOUNCE] Kete Young is now part of the Flink PMC

[Haibo Sun]

Congrats Kurt!


Best,
Haibo
At 2019-07-23 17:24:18, "Robert Metzger"  wrote:
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
>part of the Apache Flink Project Management Committee (PMC).
>
>Kete has been a committer since February 2017, working a lot on Table API /
>SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
>for Flink!
>
>Congratulations & Welcome Kurt!
>
>Best,
>Robert

Re:Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Haibo Sun]

Congrats, Zhejiang!


Best,
Haibo
在 2019-07-23 10:26:20，"Yun Tang"  写道：
>Congratulations Zhijiang, well deserved.
>
>Best
>
>From: Yingjie Cao 
>Sent: Tuesday, July 23, 2019 10:23
>To: dev@flink.apache.org 
>Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the 
>Flink project
>
>Congratulations Zhijiang!
>
>yangtao.yt  于2019年7月23日周二 上午10:17写道：
>
>> Congrats, Zhejiang!
>>
>> Best,
>> Tao Yang
>>
>> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
>> >
>> > Congratulations Zhijiang
>> >
>> > 发自我的 iPhone
>> >
>> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
>> >>
>> >> Congratulations, Zhijiang!
>> >>
>> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG 
>> wrote:
>> >>>
>> >>> Congratulations Zhijiang!
>> >>>
>> >>>
>> >>> Best,
>> >>>
>> >>> Bo WANG
>> >>>
>> >>>
>> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger 
>> >>> wrote:
>> >>>
>>  Hey all,
>> 
>>  We've added another committer to the Flink project: Zhijiang Wang.
>> 
>>  Congratulations Zhijiang!
>> 
>>  Best,
>>  Robert
>>  (on behalf of the Flink PMC)
>> 
>> >>>
>> >>
>> >>
>> >> --
>> >> Xuefu Zhang
>> >>
>> >> "In Honey We Trust!"
>>
>>

Re:Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Haibo Sun]

+1. Sounds good.Letting the PR creators know the build results of the master 
branch can help to determine more quickly whether Travis failures of their PR 
are an exact failure or because of the instability of test case. By the way, if 
the PR creator can abort their own Travis build, I think it can improve the 
efficient use of Travis resources (of course, this discussion does not deal 
with this issue). 


Best,
Haibo
At 2019-07-22 12:36:31, "Yun Tang"  wrote:
>+1 sounds good, more people could be encouraged to involve in paying attention 
>to failure commit.
>
>Best
>Yun Tang
>
>From: Becket Qin 
>Sent: Monday, July 22, 2019 9:44
>To: dev 
>Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis 
>builds
>
>+1. Sounds a good idea to me.
>
>On Sat, Jul 20, 2019 at 7:07 PM Dian Fu  wrote:
>
>> Thanks Jark for the proposal, sounds reasonable for me. +1. This ML could
>> be used for all the build notifications including master and CRON jobs.
>>
>> > 在 2019年7月20日，下午2:55，Xu Forward  写道：
>> >
>> > +1 ,Thanks jark for the proposal.
>> > Best
>> > Forward
>> >
>> > Jark Wu  于2019年7月20日周六 下午12:14写道：
>> >
>> >> Hi all,
>> >>
>> >> As far as I know, currently, email notifications of Travis builds for
>> >> master branch are sent to the commit author when a build was just
>> broken or
>> >> still is broken. And there is no email notifications for CRON builds.
>> >>
>> >> Recently, we are suffering from compile errors for scala-2.12 and java-9
>> >> which are only ran in CRON jobs. So I'm figuring out a way to get
>> >> notifications of CRON builds (or all builds) to quick fix compile errors
>> >> and failed cron tests.
>> >>
>> >> After reaching out to @Chesnay Schepler  (thanks
>> for
>> >> the helping), I know that we are using a Slack channel to receive all
>> >> failed build notifications. However, IMO, email notification might be a
>> >> better way than Slack channel to encourage more people to pay attention
>> on
>> >> the builds.
>> >>
>> >> So I'm here to propose to setup a bui...@flink.apache.org mailing list
>> for
>> >> receiving build notifications. I also find that Beam has such mailing
>> list
>> >> too[1]. After we have such a mailing list, we can integrate it to travis
>> >> according to the travis doc[2].
>> >>
>> >> What do you think? Do we need a formal vote for this?
>> >>
>> >> Best and thanks,
>> >> Jark
>> >>
>> >> [1]: https://beam.apache.org/community/contact-us/
>> >> [2]:
>> >>
>> >>
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
>> >>
>> >> <
>> >>
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
>> >>>
>> >>
>> >> <
>> >>
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
>> >>>
>> >>
>>
>>

Re:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project

[Haibo Sun]

Congratulations Becket!Best,
Haibo
在 2019-07-18 17:51:06，"Hequn Cheng"  写道：
>Congratulations Becket!
>
>Best, Hequn
>
>On Thu, Jul 18, 2019 at 5:34 PM vino yang  wrote:
>
>> Congratulations!
>>
>> Best,
>> Vino
>>
>> Yun Gao  于2019年7月18日周四 下午5:31写道：
>>
>> > Congratulations!
>> >
>> > Best,
>> > Yun
>> >
>> >
>> > --
>> > From:Kostas Kloudas 
>> > Send Time:2019 Jul. 18 (Thu.) 17:30
>> > To:dev 
>> > Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a
>> committer
>> > to the Flink project
>> >
>> > Congratulations Becket!
>> >
>> > Kostas
>> >
>> > On Thu, Jul 18, 2019 at 11:21 AM Guowei Ma  wrote:
>> >
>> > > Congrats Becket!
>> > >
>> > > Best,
>> > > Guowei
>> > >
>> > >
>> > > Terry Wang  于2019年7月18日周四 下午5:17写道：
>> > >
>> > > > Congratulations Becket!
>> > > >
>> > > > > 在 2019年7月18日，下午5:09，Dawid Wysakowicz  写道：
>> > > > >
>> > > > > Congratulations Becket! Good to have you onboard!
>> > > > >
>> > > > > On 18/07/2019 10:56, Till Rohrmann wrote:
>> > > > >> Congrats Becket!
>> > > > >>
>> > > > >> On Thu, Jul 18, 2019 at 10:52 AM Jeff Zhang 
>> > wrote:
>> > > > >>
>> > > > >>> Congratulations Becket!
>> > > > >>>
>> > > > >>> Xu Forward  于2019年7月18日周四 下午4:39写道：
>> > > > >>>
>> > > >  Congratulations Becket! Well deserved.
>> > > > 
>> > > > 
>> > > >  Cheers,
>> > > > 
>> > > >  forward
>> > > > 
>> > > >  Kurt Young  于2019年7月18日周四 下午4:20写道：
>> > > > 
>> > > > > Congrats Becket!
>> > > > >
>> > > > > Best,
>> > > > > Kurt
>> > > > >
>> > > > >
>> > > > > On Thu, Jul 18, 2019 at 4:12 PM JingsongLee <
>> > > lzljs3620...@aliyun.com
>> > > > > .invalid>
>> > > > > wrote:
>> > > > >
>> > > > >> Congratulations Becket!
>> > > > >>
>> > > > >> Best, Jingsong Lee
>> > > > >>
>> > > > >>
>> > > > >>
>> > --
>> > > > >> From:Congxian Qiu 
>> > > > >> Send Time:2019年7月18日(星期四) 16:09
>> > > > >> To:dev@flink.apache.org 
>> > > > >> Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added
>> as a
>> > > > > committer
>> > > > >> to the Flink project
>> > > > >>
>> > > > >> Congratulations Becket! Well deserved.
>> > > > >>
>> > > > >> Best,
>> > > > >> Congxian
>> > > > >>
>> > > > >>
>> > > > >> Jark Wu  于2019年7月18日周四 下午4:03写道：
>> > > > >>
>> > > > >>> Congratulations Becket! Well deserved.
>> > > > >>>
>> > > > >>> Cheers,
>> > > > >>> Jark
>> > > > >>>
>> > > > >>> On Thu, 18 Jul 2019 at 15:56, Paul Lam <
>> paullin3...@gmail.com>
>> > > >  wrote:
>> > > >  Congrats Becket!
>> > > > 
>> > > >  Best,
>> > > >  Paul Lam
>> > > > 
>> > > > > 在 2019年7月18日，15:41，Robert Metzger 
>> 写道：
>> > > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I'm excited to announce that Jiangjie (Becket) Qin just
>> > became
>> > > > >>> a
>> > > > >> Flink
>> > > > > committer!
>> > > > >
>> > > > > Congratulations Becket!
>> > > > >
>> > > > > Best,
>> > > > > Robert (on behalf of the Flink PMC)
>> > > > 
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards
>> > > > >>>
>> > > > >>> Jeff Zhang
>> > > > >>>
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>>

flink-python failed on Travis

[Haibo Sun]

Hi, folks


I noticed that all of the Travis tests reported the following failure. Is 
anyone working on this issue? 


___ summary 
ERROR:   py27: InvocationError for command 
/home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m 
virtualenv --no-download --python 
/home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/2.7/bin/python2.7
 py27 (exited with code 1)
  py33: commands succeeded
ERROR:   py34: InvocationError for command 
/home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m 
virtualenv --no-download --python 
/home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/3.4/bin/python3.4
 py34 (exited with code 100)
  py35: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
tox checks... [FAILED]
PYTHON exited with EXIT CODE: 1.
Trying to KILL watchdog (12990).


Best,
Haibo

[jira] [Created] (FLINK-13299) flink-python failed on Travis

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-13299:
-

 Summary: flink-python failed on Travis
 Key: FLINK-13299
 URL: https://issues.apache.org/jira/browse/FLINK-13299
 Project: Flink
  Issue Type: Bug
Reporter: Haibo Sun


Log: [https://api.travis-ci.com/v3/job/216620643/log.txt]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re:Re: Flink benchmark Jenkins is broken due to missing 1.10 snapshots

[Haibo Sun]

 `flink-tests_2.11:jar:tests:1.10-SNAPSHOT` was not deployed because the JIRA 
(https://issues.apache.org/jira/browse/FLINK-12602) changed the `artifactId` of 
`flink-tests` from `flink-tests_${scala.binary.version}` to `flink-tests`.


I have created a PR (https://github.com/dataArtisans/flink-benchmarks/pull/29) 
to revise pom.xml of `flink-benchmarks` accordingly, and need someone to help 
merge it.  @Chesnay Schepler 


Best,
Haobo


At 2019-07-15 19:01:57, "Yu Li"  wrote:
>Thanks for the note Chesnay, will wait and report back then.
>
>Best Regards,
>Yu
>
>
>On Mon, 15 Jul 2019 at 18:51, Chesnay Schepler  wrote:
>
>> It will likely take about a day for the 1.10 snapshots to be released.
>>
>> On 15/07/2019 12:41, Yu Li wrote:
>> > Thanks for the follow up Chesnay, Gordon and Kurt. It seems the
>> > flink-tests_2.11 snapshot [1] is not deployed yet thus flink-benchmark
>> > build [2] hasn't recovered, will watch for the next round and report back
>> > if fixed.
>> >
>> > [1]
>> >
>> https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-tests_2.11/
>> > [2] http://codespeed.dak8s.net:8080/job/flink-master-benchmarks/
>> >
>> > Best Regards,
>> > Yu
>> >
>> >
>> > On Mon, 15 Jul 2019 at 18:25, Kurt Young  wrote:
>> >
>> >> Sorry about that and thanks Gordon for fixing this!
>> >>
>> >> Best,
>> >> Kurt
>> >>
>> >>
>> >> On Mon, Jul 15, 2019 at 5:43 PM Tzu-Li (Gordon) Tai <
>> tzuli...@apache.org>
>> >> wrote:
>> >>
>> >>> Done.
>> >>>
>> >>> Thanks for the reminder and help with the Jenkins deployment setup!
>> >>>
>> >>> Cheers,
>> >>> Gordon
>> >>>
>> >>> On Mon, Jul 15, 2019 at 3:54 PM Chesnay Schepler 
>> >>> wrote:
>> >>>
>>  Please also setup the 1.9 travis cron branch.
>> 
>>  On 15/07/2019 09:46, Chesnay Schepler wrote:
>> > It is documented in the release guide that a new jenkins deployment
>> > must be setup when creating a new release branch, and even contains
>> > step-by-step instructions for doing so.
>> >
>> > @Kurt @Gordon please fix this
>> >
>> > On 12/07/2019 20:10, Yu Li wrote:
>> >> Hi All,
>> >>
>> >> I just found our flink benchmark Jenkins build [1] is broken with
>>  below
>> >> error:
>> >>
>> >> *[ERROR] Failed to execute goal on project
>> flink-hackathon-benchmarks:
>> >> Could not resolve dependencies for project
>> >> org.apache.flink.benchmark:flink-hackathon-benchmarks:jar:0.1:
>> >> Failure to
>> >> find org.apache.flink:flink-tests_2.11:jar:tests:1.10-SNAPSHOT in
>> >> https://repository.apache.org/content/repositories/snapshots/
>> >> *
>> >>
>> >> Which is due to the branching of 1.9 has updated our flink project
>> >> version
>> >> to 1.10 while still no 1.10 snapshot deployed yet. I tried to help
>> >> deploy
>> >> but it turned out with no access. Could anyone with the privilege
>> help
>> >> deploy the snapshot for the new version? Thanks.
>> >>
>> >> What's more, no blame but to prevent such issue happen again, should
>>  we
>> >> document it somewhere that a deploy of snapshot is necessary when
>> >> branching
>> >> new releases and update the snapshot version?
>> >>
>> >> I've also opened an issue in our flink-benchmarks project [2].
>> >>
>> >> Thanks.
>> >>
>> >> [1] http://codespeed.dak8s.net:8080/job/flink-master-benchmarks/
>> >> [2] https://github.com/dataArtisans/flink-benchmarks/issues/28
>> >>
>> >> Best Regards,
>> >> Yu
>> >>
>> >
>> 
>>
>>

Re:Re: [ANNOUNCE] Rong Rong becomes a Flink committer

[Haibo Sun]

Congrats Rong!Best,
Haibo

At 2019-07-12 09:40:26, "JingsongLee"  wrote:

Congratulations Rong. 
Rong Rong has done a lot of nice work in the past time to the flink community.


Best, JingsongLee


--
From:Rong Rong 
Send Time:2019年7月12日(星期五) 08:09
To:Hao Sun 
Cc:Xuefu Z ; dev ; Flink ML 

Subject:Re: [ANNOUNCE] Rong Rong becomes a Flink committer


Thank you all for the warm welcome!


It's my honor to become an Apache Flink committer. 
I will continue to work on this great project and contribute more to the 
community.



Cheers,
Rong


On Thu, Jul 11, 2019 at 1:05 PM Hao Sun  wrote:

Congratulations Rong. 


On Thu, Jul 11, 2019, 11:39 Xuefu Z  wrote:

Congratulations, Rong!



On Thu, Jul 11, 2019 at 10:59 AM Bowen Li  wrote:

Congrats, Rong!


On Thu, Jul 11, 2019 at 10:48 AM Oytun Tez  wrote:

> Congratulations Rong!
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oy...@motaword.com — www.motaword.com
>
>
> On Thu, Jul 11, 2019 at 1:44 PM Peter Huang 
> wrote:
>
>> Congrats Rong!
>>
>> On Thu, Jul 11, 2019 at 10:40 AM Becket Qin  wrote:
>>
>>> Congrats, Rong!
>>>
>>> On Fri, Jul 12, 2019 at 1:13 AM Xingcan Cui  wrote:
>>>
 Congrats Rong!

 Best,
 Xingcan

 On Jul 11, 2019, at 1:08 PM, Shuyi Chen  wrote:

 Congratulations, Rong!

 On Thu, Jul 11, 2019 at 8:26 AM Yu Li  wrote:

> Congratulations Rong!
>
> Best Regards,
> Yu
>
>
> On Thu, 11 Jul 2019 at 22:54, zhijiang 
> wrote:
>
>> Congratulations Rong!
>>
>> Best,
>> Zhijiang
>>
>> --
>> From:Kurt Young 
>> Send Time:2019年7月11日(星期四) 22:54
>> To:Kostas Kloudas 
>> Cc:Jark Wu ; Fabian Hueske ;
>> dev ; user 
>> Subject:Re: [ANNOUNCE] Rong Rong becomes a Flink committer
>>
>> Congratulations Rong!
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Jul 11, 2019 at 10:53 PM Kostas Kloudas 
>> wrote:
>> Congratulations Rong!
>>
>> On Thu, Jul 11, 2019 at 4:40 PM Jark Wu  wrote:
>> Congratulations Rong Rong!
>> Welcome on board!
>>
>> On Thu, 11 Jul 2019 at 22:25, Fabian Hueske 
>> wrote:
>> Hi everyone,
>>
>> I'm very happy to announce that Rong Rong accepted the offer of the
>> Flink PMC to become a committer of the Flink project.
>>
>> Rong has been contributing to Flink for many years, mainly working on
>> SQL and Yarn security features. He's also frequently helping out on the
>> user@f.a.o mailing lists.
>>
>> Congratulations Rong!
>>
>> Best, Fabian
>> (on behalf of the Flink PMC)
>>
>>
>>




--

Xuefu Zhang

"In Honey We Trust!"

Re:Re: [VOTE] Migrate to sponsored Travis account

[Haibo Sun]

+1. Thank Chesnay for pushing this forward.

Best,
Haibo


At 2019-07-04 17:58:28, "Kurt Young"  wrote:
>+1 and great thanks Chesnay for pushing this.
>
>Best,
>Kurt
>
>
>On Thu, Jul 4, 2019 at 5:44 PM Aljoscha Krettek  wrote:
>
>> +1
>>
>> Aljoscha
>>
>> > On 4. Jul 2019, at 11:09, Stephan Ewen  wrote:
>> >
>> > +1 to move to a private Travis account.
>> >
>> > I can confirm that Ververica will sponsor a Travis CI plan that is
>> > equivalent or a bit higher than the previous ASF quota (10 concurrent
>> build
>> > queues)
>> >
>> > Best,
>> > Stephan
>> >
>> > On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler 
>> wrote:
>> >
>> >> I've raised a JIRA
>> >> with INFRA to
>> inquire
>> >> whether it would be possible to switch to a different Travis account,
>> >> and if so what steps would need to be taken.
>> >> We need a proper confirmation from INFRA since we are not in full
>> >> control of the flink repository (for example, we cannot access the
>> >> settings page).
>> >>
>> >> If this is indeed possible, Ververica is willing sponsor a Travis
>> >> account for the Flink project.
>> >> This would provide us with more than enough resources than we need.
>> >>
>> >> Since this makes the project more reliant on resources provided by
>> >> external companies I would like to vote on this.
>> >>
>> >> Please vote on this proposal, as follows:
>> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
>> >> provided that INFRA approves
>> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
>> >> account
>> >>
>> >> The vote will be open for at least 24h, and until we have confirmation
>> >> from INFRA. The voting period may be shorter than the usual 3 days since
>> >> our current is effectively not working.
>> >>
>> >> On 04/07/2019 06:51, Bowen Li wrote:
>> >>> Re: > Are they using their own Travis CI pool, or did the switch to an
>> >>> entirely different CI service?
>> >>>
>> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>> >>> currently moving away from ASF's Travis to their own in-house metal
>> >>> machines at [1] with custom CI application at [2]. They've seen
>> >>> significant improvement w.r.t both much higher performance and
>> >>> basically no resource waiting time, "night-and-day" difference quoting
>> >>> Wes.
>> >>>
>> >>> Re: > If we can just switch to our own Travis pool, just for our
>> >>> project, then this might be something we can do fairly quickly?
>> >>>
>> >>> I believe so, according to [3] and [4]
>> >>>
>> >>>
>> >>> [1] https://ci.ursalabs.org/ 
>> >>> [2] https://github.com/ursa-labs/ursabot
>> >>> [3]
>> >>>
>> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
>> >>> [4]
>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler > >>> > wrote:
>> >>>
>> >>>Are they using their own Travis CI pool, or did the switch to an
>> >>>entirely different CI service?
>> >>>
>> >>>If we can just switch to our own Travis pool, just for our
>> >>>project, then
>> >>>this might be something we can do fairly quickly?
>> >>>
>> >>>On 03/07/2019 05:55, Bowen Li wrote:
>>  I responded in the INFRA ticket [1] that I believe they are
>> >>>using a wrong
>>  metric against Flink and the total build time is a completely
>> >>>different
>>  thing than guaranteed build capacity.
>> 
>>  My response:
>> 
>>  "As mentioned above, since I started to pay attention to Flink's
>> >>>build
>>  queue a few tens of days ago, I'm in Seattle and I saw no build
>> >>>was kicking
>>  off in PST daytime in weekdays for Flink. Our teammates in China
>> >>>and Europe
>>  have also reported similar observations. So we need to evaluate
>> >>>how the
>>  large total build time came from - if 1) your number and 2) our
>>  observations from three locations that cover pretty much a full
>> >>>day, are
>>  all true, I **guess** one reason can be that - highly likely the
>> >>>extra
>>  build time came from weekends when other Apache projects may be
>> >>>idle and
>>  Flink just drains hard its congested queue.
>> 
>>  Please be aware of that we're not complaining about the lack of
>> >>>resources
>>  in general, I'm complaining about the lack of **stable, dedicated**
>>  resources. An example for the latter one is, currently even if
>> >>>no build is
>>  in Flink's queue and I submit a request to be the queue head in PST
>>  morning, my build won't even start in 6-8+h. That is an absurd
>> >>>amount of
>>  waiting time.
>> 
>>  That's saying, if ASF INFRA decides to adopt a quota system and
>> >>>grants
>>  Flink five DEDICATED servers that runs all the time only for
>> >>>Fl

[jira] [Created] (FLINK-13051) Drop the non-selectable two-input StreamTask and Processor

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-13051:
-

 Summary: Drop the non-selectable two-input StreamTask and Processor
 Key: FLINK-13051
 URL: https://issues.apache.org/jira/browse/FLINK-13051
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Haibo Sun
Assignee: Haibo Sun


After `StreamTwoInputSelectableProcessor` supports `CheckpointBarrierHandler`, 
we should  drop the non-selectable  two-input StreamTask and Processor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13014) ChainBreakTest failed on Travis

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-13014:
-

 Summary: ChainBreakTest failed on Travis
 Key: FLINK-13014
 URL: https://issues.apache.org/jira/browse/FLINK-13014
 Project: Flink
  Issue Type: Bug
Reporter: Haibo Sun


07:12:52.246 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 16.872 s <<< FAILURE! - in 
org.apache.flink.test.state.operator.restore.unkeyed.ChainBreakTest
07:12:52.247 [ERROR] testMigrationAndRestore[Migrate Savepoint: 
1.3](org.apache.flink.test.state.operator.restore.unkeyed.ChainBreakTest) Time 
elapsed: 1.545 s <<< ERROR!
java.util.concurrent.ExecutionException: 
java.util.concurrent.CompletionException: 
org.apache.flink.runtime.checkpoint.CheckpointException: Task received 
cancellation from one of its inputs
Caused by: java.util.concurrent.CompletionException: 
org.apache.flink.runtime.checkpoint.CheckpointException: Task received 
cancellation from one of its inputs
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Task 
received cancellation from one of its inputs
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Task 
received cancellation from one of its inputs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12967) Change the input selection switching in StreamTwoInputSelectableProcessor#checkFinished to invoking the nextSelection method of the stream operator

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12967:
-

 Summary: Change the input selection switching in 
StreamTwoInputSelectableProcessor#checkFinished to invoking the nextSelection 
method of the stream operator
 Key: FLINK-12967
 URL: https://issues.apache.org/jira/browse/FLINK-12967
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun


The runtime (`StreamTwoInputSelectableProcessor#checkFinished()`) switches the 
input selection when one input is finished, because `BoundedxInput.endInput()` 
was not supported before the PR#8731 
(https://github.com/apache/flink/pull/8731) is merged. Now we should change the 
logic of `StreamTwoInputSelectableProcessor#checkFinished()` to invoke 
`InputSelectable#nextSelection()`, and the input selection should been switched 
in `endInput()` by the operator.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re:[ANNOUNCE] Jincheng Sun is now part of the Flink PMC

[Haibo Sun]

Congratulations!


Best,
Haibo

At 2019-06-24 23:08:54, "Robert Metzger"  wrote:
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Jincheng Sun is now
>part of the Apache Flink Project Management Committee (PMC).
>
>Jincheng has been a committer since July 2017. He has been very active on
>Flink's Table API / SQL component, as well as helping with releases.
>
>Congratulations & Welcome Jincheng!
>
>Best,
>Robert

Re:Re: CatalogTestBase.* failed on travis

[Haibo Sun]

Thanks for the quick fix. 


Best,
Haibo

在 2019-06-20 21:24:47，"jincheng sun"  写道：
>Hi Haibo,
>
>The issue is fixed, great thanks for your timely report on this issue！
>
>Best，
>Jincheng
>
>
>jincheng sun  于2019年6月20日周四 下午9:04写道：
>
>> Hi flink devs:
>>
>> Normally, in order not to affect the merge of the followup PR, we need to
>> revert the problematic commit, but since Dian quickly opens the hotfix PR,
>> this time we take the hotfix.
>>
>> So great thanks to Dian！
>>
>> Hi Dian, you are right，the test fail caused by the PR #8786， and I
>> completely agree all the committers should check the Travis success before
>> merging.
>>
>> Your PR looks good, I'll merge it after the test passed! (waiting test
>> pass)
>>
>> Best,
>> Jincheng
>>
>> jincheng sun  于2019年6月20日周四 下午8:30写道：
>>
>>> Great thanks for the help to fix it Dian!  I'll merge it.
>>>
>>>
>>>
>>> Dian Fu  于2019年6月20日周四 下午7:50写道：
>>>
>>>> Hi Haibo,
>>>>
>>>> Thanks a lot for report this bug. I guess it's caused by this PR:
>>>> https://github.com/apache/flink/pull/8786 <
>>>> https://github.com/apache/flink/pull/8786> @Bowen. I think we'd better
>>>> merge the code ONLY after the travis passed, especially when the changes
>>>> are not just hotfix/documentation. Anyway, I'll try to provide a fix ASAP.
>>>>
>>>> Regards,
>>>> Dian
>>>>
>>>> > 在 2019年6月20日，下午6:15，Haibo Sun  写道：
>>>> >
>>>> > Hi, guys
>>>> >
>>>> >
>>>> > I noticed that all of the Travis tests reported a number of failures
>>>> as following. Is anyone working on this problem?
>>>> >
>>>> >
>>>> > __ CatalogTestBase.test_table_exists
>>>> ___
>>>> >
>>>> > self = >>> testMethod=test_table_exists>
>>>> >
>>>> >def test_table_exists(self):
>>>> >>  self.catalog.create_database(self.db1, self.create_db(), False)
>>>> >
>>>> > pyflink/table/tests/test_catalog.py:491:
>>>> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>> _ _ _ _ _
>>>> >
>>>> >@staticmethod
>>>> >def create_db():
>>>> >gateway = get_gateway()
>>>> >>  j_database = gateway.jvm.GenericCatalogDatabase({"k1": "v1"},
>>>> CatalogTestBase.test_comment)
>>>> > E   TypeError: 'JavaPackage' object is not callable
>>>> >
>>>> >
>>>> > pyflink/table/tests/test_catalog.py:78: TypeError
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Best,
>>>> > Haibo
>>>> >
>>>> >
>>>> >
>>>>
>>>>

CatalogTestBase.* failed on travis

[Haibo Sun]

Hi, guys


I noticed that all of the Travis tests reported a number of failures as 
following. Is anyone working on this problem? 


__ CatalogTestBase.test_table_exists ___

self = 

def test_table_exists(self):
>   self.catalog.create_database(self.db1, self.create_db(), False)

pyflink/table/tests/test_catalog.py:491: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

@staticmethod
def create_db():
gateway = get_gateway()
>   j_database = gateway.jvm.GenericCatalogDatabase({"k1": "v1"}, 
> CatalogTestBase.test_comment)
E   TypeError: 'JavaPackage' object is not callable


pyflink/table/tests/test_catalog.py:78: TypeError




Best,
Haibo

[jira] [Created] (FLINK-12895) TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12895:
-

 Summary: 
TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure 
failed on travis 
 Key: FLINK-12895
 URL: https://issues.apache.org/jira/browse/FLINK-12895
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Haibo Sun


Logs:  [https://api.travis-ci.org/v3/job/547509708/log.txt]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12547) Deadlock when the task thread downloads jars using BlobClient

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12547:
-

 Summary: Deadlock when the task thread downloads jars using 
BlobClient
 Key: FLINK-12547
 URL: https://issues.apache.org/jira/browse/FLINK-12547
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Operators
Affects Versions: 1.8.0
Reporter: Haibo Sun
Assignee: Haibo Sun


The jstack is as follows (this jstack is from an old Flink version, but the 
master branch has the same problem).
{code:java}
"Source: Custom Source (76/400)" #68 prio=5 os_prio=0 tid=0x7f8139cd3000 
nid=0xe2 runnable [0x7f80da5fd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.flink.runtime.blob.BlobInputStream.read(BlobInputStream.java:152)
at org.apache.flink.runtime.blob.BlobInputStream.read(BlobInputStream.java:140)
at 
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:164)
at 
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:181)
at 
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:206)
at 
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120)
- locked <0x00062cf2a188> (a java.lang.Object)
at 
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:968)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:604)
at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:
- None
{code}
The reason is that SO_TIMEOUT is not set in the socket connection of the blob 
client. When the network packet loss seriously due to the high CPU load of the 
machine, the blob client connection fails to perceive that the server has been 
disconnected, which results in blocking in the native method. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12529) Release buffers of the record deserializer timely to improve the efficiency of heap memory usage on taskmanager

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12529:
-

 Summary: Release buffers of the record deserializer timely to 
improve the efficiency of heap memory usage on taskmanager
 Key: FLINK-12529
 URL: https://issues.apache.org/jira/browse/FLINK-12529
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Haibo Sun
Assignee: Haibo Sun


In input processors (`StreamInputProcessor` and `StreamTwoInputProcessor`), 
each input channel has a corresponding record deserializer. Currently, these 
record deserializers are cleaned up at the end of the task (look at 
`StreamInputProcessor#cleanup()` and `StreamTwoInputProcessor#cleanup()`). This 
is not a problem for unbounded streams, but it may reduce the efficiency of 
heap memory usage on taskmanger when input is bounded stream.

For example, in case that all inputs are bounded streams, some of them end very 
early because of the small amount of data, and the other end very late because 
of the large amount of data, then the buffers of the record deserializers 
corresponding to the input channels finished early is idle for a long time and 
no longer used.

In another case, when both unbounded and bounded streams exist in the inputs, 
the buffers of the record deserializers corresponding to the bounded stream are 
idle for ever (no longer used) after the bounded streams are finished. 
Especially when the record and the parallelism of upstream are large, the total 
size of `SpanningWrapper#buffer` are very large. The size of 
`SpanningWrapper#buffer` is allowed to reach up to 5 MB, and if the parallelism 
of upstream is 100, the maximum total size will reach 500 MB (in our 
production, there are jobs with the record size up to hundreds of KB and the 
parallelism of upstream up to 1000).

Overall, after receiving `EndOfPartitionEvent` from the input channel, the 
corresponding record deserializer should be cleared immediately to improve the 
efficiency of heap memory usage on taskmanager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12292) Add benchmarks for the input processors

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12292:
-

 Summary: Add benchmarks for the input processors
 Key: FLINK-12292
 URL: https://issues.apache.org/jira/browse/FLINK-12292
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Operators
Reporter: Haibo Sun
Assignee: Haibo Sun


Adds benchmarks for `StreamTwoInputProcessor` and 
`StreamTwoInputSelectableProcessor` to ensure that 
StreamTwoInputSelectableProcessor's throughput is the same or the regression is 
acceptable in the case of constant `InputSelection.ALL`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12291) Add benchmarks for the input processors

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-12291:
-

 Summary: Add benchmarks for the input processors
 Key: FLINK-12291
 URL: https://issues.apache.org/jira/browse/FLINK-12291
 Project: Flink
  Issue Type: Task
  Components: Runtime / Operators
Reporter: Haibo Sun
Assignee: Haibo Sun


Adds benchmarks for `StreamTwoInputProcessor` and 
`StreamTwoInputSelectableProcessor` to ensure that 
StreamTwoInputSelectableProcessor's throughput is the same or the regression is 
acceptable in the case of constant `InputSelection.ALL`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11878) Implement the runtime handling of BoundedOneInput and BoundedTwoInput

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11878:
-

 Summary: Implement the runtime handling of BoundedOneInput and 
BoundedTwoInput
 Key: FLINK-11878
 URL: https://issues.apache.org/jira/browse/FLINK-11878
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11877) Implement the runtime handling of TwoInputSelectable

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11877:
-

 Summary: Implement the runtime handling of TwoInputSelectable
 Key: FLINK-11877
 URL: https://issues.apache.org/jira/browse/FLINK-11877
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun


- Introduces a new class `Input` to represent the logical input of operators.
 - Introduces a new class `StreamTwoInputSelectableProcessor` to implement 
selectively reading.
 - Adds benchmarks for `StreamTwoInputProcessor` and 
`StreamTwoInputSelectableProcessor` to ensure good performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11880) Add JobGraph validators for the uses of TwoInputSelectable, BoundedOneInput and BoundedTwoInput

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11880:
-

 Summary: Add JobGraph validators for the uses of 
TwoInputSelectable, BoundedOneInput and BoundedTwoInput
 Key: FLINK-11880
 URL: https://issues.apache.org/jira/browse/FLINK-11880
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun


- Rejects the jobs containing operators which were implemented 
`TwoInputSelectable` in case of enabled checkpointing.
 - Rejects the jobs containing operators which were implemented `BoundedInput` 
or `BoundedTwoInput` in case of enabled checkpointing.
 - Rejects the jobs containing operators which were implemented 
`TwoInputSelectable` in case that credit-based flow control is disabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11879) Add JobGraph validators for the uses of TwoInputSelectable, BoundedOneInput and BoundedTwoInput

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11879:
-

 Summary: Add JobGraph validators for the uses of 
TwoInputSelectable, BoundedOneInput and BoundedTwoInput
 Key: FLINK-11879
 URL: https://issues.apache.org/jira/browse/FLINK-11879
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun


- Rejects the jobs containing operators which were implemented 
`TwoInputSelectable` in case of enabled checkpointing.
 - Rejects the jobs containing operators which were implemented `BoundedInput` 
or `BoundedTwoInput` in case of enabled checkpointing.
 - Rejects the jobs containing operators which were implemented 
`TwoInputSelectable` in case that credit-based flow control is disabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11876) Introduce the new interfaces TwoInputSelectable, BoundedOneInput and BoundedTwoInput

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11876:
-

 Summary: Introduce the new interfaces TwoInputSelectable, 
BoundedOneInput and BoundedTwoInput
 Key: FLINK-11876
 URL: https://issues.apache.org/jira/browse/FLINK-11876
 Project: Flink
  Issue Type: Sub-task
Reporter: Haibo Sun
Assignee: Haibo Sun






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11875) Enhance stream operator API to support selective reading and EndOfInput event

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-11875:
-

 Summary: Enhance stream operator API to support selective reading 
and EndOfInput event
 Key: FLINK-11875
 URL: https://issues.apache.org/jira/browse/FLINK-11875
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Operators
Reporter: Haibo Sun
Assignee: Haibo Sun


Towards the goal that unify Streaming and Batch, this jira proposes enhanced 
stream operator api to support selective reading and EndOfInput event.

Design Doc:

[https://docs.google.com/document/d/10k5pQm3SkMiK5Zn1iFDqhQnzjQTLF0Vtcbc8poB4_c8/edit?usp=sharing|http://example.com/]

Discussion Mail:

[http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enhance-Operator-API-to-Support-Dynamically-Selective-Reading-and-EndOfInput-Event-td26753.html|http://example.com/]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Enhance Operator API to Support Dynamically Selective Reading and EndOfInput Event

[Haibo Sun]

>> >
>> > *early-out*
>> >
>> > Letting nextSelection() return “NONE” or “FINISHED" may be relevant for
>> > early-out cases, but I would remove this from the scope of this proposal.
>> > There are most likely other big changes involved, like communicating this
>> > to the upstream operators.
>> >
>> > *distributed stream deadlocks*
>> >
>> > We had this issue in the DataSet API. Earlier versions of the DataSet API
>> > made an analysis of the flow detecting dams and whether the pipeline
>> > breaking behavior in the flow would cause deadlocks, and introduce
>> > artificial pipeline breakers in response.
>> >
>> > The logic was really complicated and it took a while to become stable. We
>> > had several issues that certain user functions (like mapPartition) could
>> > either be pipelined or have a full dam (not possible to know for the
>> > system), so we had to insert artificial pipeline breakers in all paths.
>> >
>> > In the end we simply decided that in the case of a diamond-style flow, we
>> > make the point where the flow first forks as blocking shuffle. That was
>> > super simple, solved all issues, and has the additional nice property
>> that
>> > it great point to materialize data for recovery, because it helps both
>> > paths of the diamond upon failure.
>> >
>> > My suggestion:
>> > ==> For streaming, no problem so far, nothing to do
>> > ==> For batch, would suggest to go with the simple solution described
>> above
>> > first, and improve when we see cases where this impacts performance
>> > significantly
>> >
>> > *empty input / selection timeout*
>> >
>> > I can see that being relevant in future streaming cases, for example with
>> > side inputs. You want to wait for the side input data, but with a
>> timeout,
>> > so the program can still proceed with non-perfect context data in case
>> that
>> > context data is very late.
>> >
>> > Because we do not support side inputs at the moment, we may want to defer
>> > this for now. Let's not over-design for problems that are not well
>> > understood at this point.
>> >
>> > *timers*
>> >
>> > I don't understand the problem with timers. Timers are bound to the
>> > operator, not the input, so they should still work if an input ends.
>> > There are cases where some state in the operator that is only relevant as
>> > long as an input still has data (like in symmetric joins) and the timers
>> > are relevant to that state.
>> > When the state is dropped, the timers should also be dropped, but that is
>> > the operator's logic on "endInput()". So there is no inherent issue
>> between
>> > input and timers.
>> >
>> > Best,
>> > Stephan
>> >
>> >
>> > On Sat, Feb 2, 2019 at 3:55 AM Guowei Ma  wrote:
>> >
>> > > Hi, guys:
>> > > I propose a design to enhance Stream Operator API for Batch’s
>> > requirements.
>> > > This is also the Flink’s goal that Batch is a special case of
>> Streaming.
>> > > This
>> > > proposal mainly contains two changes to operator api:
>> > >
>> > > 1. Allow "StreamOperator" can choose which input to read;
>> > > 2. Notify "StreamOperator" that an input has ended.
>> > >
>> > >
>> > > This proposal was discussed with Piotr Nowojski, Kostas Kloudas, Haibo
>> > Sun
>> > > offlline.
>> > > It will be great to hear the feed backs and suggestions from the
>> > community.
>> > > Please kindly share your comments and suggestions.
>> > >
>> > > Best
>> > > GuoWei Ma.
>> > >
>> > >  Enhance Operator API to Support Dynamically Sel...
>> > > <
>> > >
>> >
>> https://docs.google.com/document/d/10k5pQm3SkMiK5Zn1iFDqhQnzjQTLF0Vtcbc8poB4_c8/edit?usp=drive_web
>> > > >
>> > >
>> >
>>

Apply for flink contributor permission

[Haibo Sun]

Hi guys,
Could anyone kindly give me contributor permission? My JIRA username is
sunhaibotb.

Thanks,
Haibo

Re: [DISCUSS] Unified Core API for Streaming and Batch

[Haibo Sun]

Hi All,

Thank Aljoscha for further spitting up topics.

I will start separate threads on each topic which you propose.

Best,
Haibo



Aljoscha Krettek-2 wrote
> Hi All,
> 
> this is a great discussion! (I have some thoughts on most of the topics
> but I'll wait for the separate discussion threads)
> 
> @Haibo Will you start a separate threads? I think the separate discussion
> topics would be (based on Stephans mail but further split up):
> 
> 1. What should the API stack look like?
> 2. What should the interface for a single operator look like, i.e. what
> will StreamOperator look like?
> 3. What does a job look like, i.e. the graph of operations. Maybe a proper
> serialized format for DAGs.
> 4. Modules and dependency structure. This is currently a bit messed up for
> flink-streaming, which depends on flink-runtime
> 5. What's special for batch.
> 
> There's some interdependencies, i.e. 2 depends on 5. and maybe 1.
> 
> Best,
> Aljoscha
> 
>> On 7. Dec 2018, at 10:00, Shuai Xu <

> chiggics@

> > wrote:
>> 
>> Hi all
>> Glad to see the discussion, we are now designing to enhance the
>> scheduling
>> of batch job, a unified api will help a lot.
>> 
>> Haibo Sun <

> sunhaibotb@

> > 于2018年12月5日周三 下午4:45写道：
>> 
>>> Hi all,
>>> 
>>> Thank Kurt, you see more benefits of the unification than I do.
>>> 
>>> I quite agree Kurt's views. DataStream, DataSet and Table are remained
>>> independent for now, and subsumed DataSet in data stream in the future.
>>> The
>>> collection execution mode is replaced by mini cluster. The high-level
>>> semantic APIs  have their own optimizations, but StreamTransformation
>>> does
>>> not.
>>> 
>>> About iterations, I have not more ideas at the moment.
>>> 
>>> 
>>> Best,
>>> Haibo
>>> 
>>> 
>>> 
>>> --
>>> Sent from:
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>>>





--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re: [DISCUSS] Unified Core API for Streaming and Batch

[Haibo Sun]

Hi all,

Thank Kurt, you see more benefits of the unification than I do.

I quite agree Kurt's views. DataStream, DataSet and Table are remained
independent for now, and subsumed DataSet in data stream in the future. The
collection execution mode is replaced by mini cluster. The high-level
semantic APIs  have their own optimizations, but StreamTransformation does
not.

About iterations, I have not more ideas at the moment.


Best, 
Haibo



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re:[DISCUSS] Unified Core API for Streaming and Batch

[Haibo Sun]

Thanks, zhijiang. 


For the optimization, such as cost-based estimation, we still want to keep it 
in the data set layer, 
but your suggestion is also a thought that can be considered.


As I know, currently these batch scenarios have been contained in DataSet, such 
as
the sort-merge join algorithm. So I think that the unification should consider 
such features
as input selection at reading.


Best,
Haibo


At 2018-12-03 16:38:13, "zhijiang"  wrote:
>Hi haibo,
>
>Thanks for bringing this discussion!
>
> I reviewd the google doc and really like the idea of unifying the stream and 
> batch in all stacks. Currently only network runtime stack is unified for both 
> stream and batch jobs, but the compilation, operator and runtime task stacks 
> are all separate. The stream stack developed frequently and behaved 
> dominantly these years, but the batch stack was touched less. If they are 
> unified into one stack, the batch jobs can also get benefits from all the 
> improvements. I think it is a very big work but worth doing, left some 
> concerns:
>
>1. The current job graph generation for batch covers complicated optimization 
>such as cost-based estimate, plan etc. Would this part also be considered 
>retaining during integrating with stream graph generation?
>
>2. I saw some other special improvements for batch scenarios in the doc, such 
>as input selection while reading. I acknowledge these roles for special batch 
>scenarios, but they seem not the blocker for unification motivation, because 
>current batch jobs can also work without these improvements. So the further 
>improvments can be separated into individual topics after we reaching the 
>unification of stream and batch firstly.
>
>Best,
>Zhijiang
>
>
>--
>发件人：孙海波 
>发送时间：2018年12月3日(星期一) 10:52
>收件人：dev 
>主　题：[DISCUSS] Unified Core API for Streaming and Batch
>
>Hi all,
>This post proposes unified core API for Streaming and Batch. 
>Currently DataStream and DataSet adopt separated compilation processes, 
>execution tasks
>and basic programming models in the runtime layer, which complicates the 
>system implementation. 
>We think that batch jobs can be processed in the same way as streaming jobs, 
>thus we can unify
>the execution stack of DataSet into that of DataStream.  After the unification 
>the DataSet API will
>also be built on top of StreamTransformation, and its basic programming model 
>will be changed
>from "UDF on Driver" to "UDF on StreamOperator". Although the DataSet 
>operators will need to
>implement the interface StreamOperator instead after the unification, user 
>jobs do not need to change
>since DataSet uses the same UDF interfaces as DataStream.
>
>The unification has at least three benefits:
>1. The system will be greatly simplified with the same execution stack for 
>both streaming and batch jobs.
>2. It is no longer necessary to implement two sets of Driver(s) (operator 
>strategies) for batch, namely chained and non-chained.
>3. The unified programming model enables streaming and batch jobs to share the 
>same operator implementation.
>

>The following is the design draft. Any feedback is highly appreciated.
>https://docs.google.com/document/d/1G0NUIaaNJvT6CMrNCP6dRXGv88xNhDQqZFrQEuJ0rVU/edit?usp=sharing
>
>Best, 
>Haibo