[jira] [Created] (FLINK-13395) Add source and sink connector for Aliyun Log Service

[Ke Li (JIRA)]

Ke Li created FLINK-13395:
-

 Summary: Add source and sink connector for Aliyun Log Service
 Key: FLINK-13395
 URL: https://issues.apache.org/jira/browse/FLINK-13395
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Common
Reporter: Ke Li


 Aliyun Log Service is a storage service which has been widely used in Alibaba 
Group and a lot of customers on Alibaba Cloud. The core storage engine is call 
Loghub which is a large scale distributed storage system and provides 
producer/consumer API as Kafka/Kinesis does. 

There are a lot of users are using Log Service to collect data from on premise 
and cloud and consuming from Flink or Blink for streaming compute. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13394) secure MapR repo URL is not work in E2E crontab builds

[Zhenghua Gao (JIRA)]

Zhenghua Gao created FLINK-13394:


 Summary: secure MapR repo URL is not work in E2E crontab builds
 Key: FLINK-13394
 URL: https://issues.apache.org/jira/browse/FLINK-13394
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhenghua Gao
 Fix For: 1.9.0, 1.10.0


[FLINK-12578|https://issues.apache.org/jira/browse/FLINK-12578] 
[FLINK-12578|http://example.com/] intros https URL for MapR, but this causes 
fails on Travis for some reason. travis_watchdog.sh and travis_controller.sh 
are fixed by unsafe-mapr-repo profile, but nightly.sh is not fixed.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Publish the PyFlink into PyPI

[Dian Fu]

Hi Stephan & Jeff,

Thanks a lot for sharing your thoughts!

Regarding the bundled jars, currently only the jars in the flink binary 
distribution is packaged in the pyflink package. That maybe a good idea to also 
bundle the other jars such as flink-hadoop-compatibility. We may need also 
consider whether to bundle the format jars such as flink-avro, flink-json, 
flink-csv and the connector jars such as flink-connector-kafka, etc.

If FLINK_HOME is set, the binary distribution specified by FLINK_HOME will be 
used instead.

Regards,
Dian

> 在 2019年7月24日，上午9:47，Jeff Zhang  写道：
> 
> +1 for publishing pyflink to pypi.
> 
> Regarding including jar, I just want to make sure which flink binary
> distribution we would ship with pyflink since we have multiple flink binary
> distributions (w/o hadoop).
> Personally, I prefer to use the hadoop-included binary distribution.
> 
> And I just want to confirm whether it is possible for users to use a
> different flink binary distribution as long as he set env FLINK_HOME.
> 
> Besides that, I hope that there will be bi-direction link reference between
> flink doc and pypi doc.
> 
> 
> 
> Stephan Ewen  于2019年7月24日周三 上午12:07写道：
> 
>> Hi!
>> 
>> Sorry for the late involvement. Here are some thoughts from my side:
>> 
>> Definitely +1 to publishing to PyPy, even if it is a binary release.
>> Community growth into other communities is great, and if this is the
>> natural way to reach developers in the Python community, let's do it. This
>> is not about our convenience, but reaching users.
>> 
>> I think the way to look at this is that this is a convenience distribution
>> channel, courtesy of the Flink community. It is not an Apache release, we
>> make this clear in the Readme.
>> Of course, this doesn't mean we don't try to uphold similar standards as
>> for our official release (like proper license information).
>> 
>> Concerning credentials sharing, I would be fine with whatever option. The
>> PMC doesn't own it (it is an initiative by some community members), but the
>> PMC needs to ensure trademark compliance, so slight preference for option
>> #1 (PMC would have means to correct problems).
>> 
>> I believe there is no need to differentiate between Scala versions, because
>> this is merely a convenience thing for pure Python users. Users that mix
>> python and scala (and thus depend on specific scala versions) can still
>> download from Apache or build themselves.
>> 
>> Best,
>> Stephan
>> 
>> 
>> 
>> On Thu, Jul 4, 2019 at 9:51 AM jincheng sun 
>> wrote:
>> 
>>> Hi All,
>>> 
>>> Thanks for the feedback @Chesnay Schepler  @Dian!
>>> 
>>> I think using `apache-flink` for the project name also makes sense to me.
>>> due to we should always keep in mind that Flink is owned by Apache. (And
>>> beam also using this pattern `apache-beam` for Python API)
>>> 
>>> Regarding the Python API release with the JAVA JARs, I think the
>> principle
>>> of consideration is the convenience of the user. So, Thanks for the
>>> explanation @Dian!
>>> 
>>> And your right @Chesnay Schepler   we can't make a
>>> hasty decision and we need more people's opinions！
>>> 
>>> So, I appreciate it if anyone can give us feedback and suggestions!
>>> 
>>> Best,
>>> Jincheng
>>> 
>>> 
>>> 
>>> 
>>> Chesnay Schepler  于2019年7月3日周三 下午8:46写道：
>>> 
 So this would not be a source release then, but a full-blown binary
 release.
 
 Maybe it is just me, but I find it a bit suspect to ship an entire java
 application via PyPI, just because there's a Python API for it.
 
 We definitely need input from more people here.
 
 On 03/07/2019 14:09, Dian Fu wrote:
> Hi Chesnay,
> 
> Thanks a lot for the suggestions.
> 
> Regarding “distributing java/scala code to PyPI”:
> The Python Table API is just a wrapper of the Java Table API and
>>> without
 the java/scala code, two steps will be needed to set up an environment
>> to
 execute a Python Table API program:
> 1) Install pyflink using "pip install apache-flink"
> 2) Download the flink distribution and set the FLINK_HOME to it.
> Besides, users have to make sure that the manually installed Flink is
 compatible with the pip installed pyflink.
> 
> Bundle the java/scala code inside the Python package will eliminate
>>> step
 2) and makes it more simple for users to install pyflink. There was a
>>> short
 discussion  on this
>> in
 Spark community and they finally decide to package the java/scala code
>> in
 the python package. (BTW, PySpark only bundle the jars of scala 2.11).
> 
> Regards,
> Dian
> 
>> 在 2019年7月3日，下午7:13，Chesnay Schepler  写道：
>> 
>> The existing artifact in the pyflink project was neither released by
 the Flink project / anyone affiliated with it nor approved by the Flink
>>> PMC.
>> 
>> As such, if we were to use this account I believe we should delete
>> it

[jira] [Created] (FLINK-13393) Blink-planner not support generic TableSource

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-13393:


 Summary: Blink-planner not support generic TableSource
 Key: FLINK-13393
 URL: https://issues.apache.org/jira/browse/FLINK-13393
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Jingsong Lee
 Fix For: 1.9.0, 1.10.0


Now there is a exception when use:

class MyTableSource[T] extend StreamTableSource[T].



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: flink-mapr-fs failed in travis

[JingsongLee]

Sorry, 
"It looks like it's been compiled all the time." 
should be: "It looks like it can not be compiled all the time in local."

Best, Jingsong Lee


--
From:JingsongLee 
Send Time:2019年7月24日(星期三) 10:54
To:Jark Wu ; dev ; chesnay 

Subject:Re: flink-mapr-fs failed in travis

Hi @chesnay :
Thanks for fix on travis, Do you have any idea about why we build failed in 
local? 
It looks like it's been compiled all the time.
 (It should have nothing to do with your previous changes)

Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not 
transfer artifact com.mapr.hadoop:maprfs:pom:5.2.1-mapr from/to mapr-releases 
(https://repository.mapr.com/maven/): 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
[1] 
https://stackoverflow.com/questions/57106180/unable-to-build-flink-from-sources-due-to-mapr-artifacts-problems

Best, Jingsong Lee


--
From:Jark Wu 
Send Time:2019年7月19日(星期五) 18:47
To:dev 
Cc:JingsongLee 
Subject:Re: flink-mapr-fs failed in travis

Great! Thanks Chesnay for the quick fixing.


On Fri, 19 Jul 2019 at 16:40, Chesnay Schepler  wrote:
I think I found the issue; I forgot to update travis_controller.sh .

 On 19/07/2019 10:02, Chesnay Schepler wrote:
 > Ah, I added it to the common options in the travis_manv_watchdog.sh .
 >
 > On 19/07/2019 09:58, Chesnay Schepler wrote:
 >> I did modify the .travis.yml do activate the unsafe-mapr-repo 
 >> profile; did I modified the wrong profile?...
 >>
 >>
 >> On 19/07/2019 07:57, Jark Wu wrote:
 >>> It seems that it is introduced by this commit:
 >>> https://github.com/apache/flink/commit/5c36c650e6520d92191ce2da33f7dcae774319f6
 >>>  
 >>>
 >>> Hi @Chesnay Schepler  , do we need to add
 >>> "-Punsafe-mapr-repo" to the ".travis.yml"?
 >>>
 >>> Best,
 >>> Jark
 >>>
 >>> On Fri, 19 Jul 2019 at 10:58, JingsongLee 
 >>> 
 >>> wrote:
 >>>
  Hi everyone:
 
  flink-mapr-fs failed in travis, and I retried many times, and also 
  failed.
  Anyone has idea about this?
 
  01:32:54.755 [ERROR] Failed to execute goal on project flink-mapr-fs:
  Could not resolve dependencies for project
  org.apache.flink:flink-mapr-fs:jar:1.10-SNAPSHOT: Failed to collect
  dependencies at com.mapr.hadoop:maprfs:jar:5.2.1-mapr: Failed to read
  artifact descriptor for com.mapr.hadoop:maprfs:jar:5.2.1-mapr: 
  Could not
  transfer artifact com.mapr.hadoop:maprfs:pom:5.2.1-mapr from/to
  mapr-releases (https://repository.mapr.com/maven/):
  sun.security.validator.ValidatorException: PKIX path building failed:
  sun.security.provider.certpath.SunCertPathBuilderException: unable 
  to find
  valid certification path to requested target -> [Help 1]
 
  https://api.travis-ci.org/v3/job/560790299/log.txt
 
  Best, Jingsong Lee
 >>
 >>
 >>
 >
 >

Re: flink-mapr-fs failed in travis

[JingsongLee]

Hi @chesnay :
Thanks for fix on travis, Do you have any idea about why we build failed in 
local? 
It looks like it's been compiled all the time.
 (It should have nothing to do with your previous changes)

Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not 
transfer artifact com.mapr.hadoop:maprfs:pom:5.2.1-mapr from/to mapr-releases 
(https://repository.mapr.com/maven/): 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
[1] 
https://stackoverflow.com/questions/57106180/unable-to-build-flink-from-sources-due-to-mapr-artifacts-problems

Best, Jingsong Lee


--
From:Jark Wu 
Send Time:2019年7月19日(星期五) 18:47
To:dev 
Cc:JingsongLee 
Subject:Re: flink-mapr-fs failed in travis

Great! Thanks Chesnay for the quick fixing.


On Fri, 19 Jul 2019 at 16:40, Chesnay Schepler  wrote:
I think I found the issue; I forgot to update travis_controller.sh .

 On 19/07/2019 10:02, Chesnay Schepler wrote:
 > Ah, I added it to the common options in the travis_manv_watchdog.sh .
 >
 > On 19/07/2019 09:58, Chesnay Schepler wrote:
 >> I did modify the .travis.yml do activate the unsafe-mapr-repo 
 >> profile; did I modified the wrong profile?...
 >>
 >>
 >> On 19/07/2019 07:57, Jark Wu wrote:
 >>> It seems that it is introduced by this commit:
 >>> https://github.com/apache/flink/commit/5c36c650e6520d92191ce2da33f7dcae774319f6
 >>>  
 >>>
 >>> Hi @Chesnay Schepler  , do we need to add
 >>> "-Punsafe-mapr-repo" to the ".travis.yml"?
 >>>
 >>> Best,
 >>> Jark
 >>>
 >>> On Fri, 19 Jul 2019 at 10:58, JingsongLee 
 >>> 
 >>> wrote:
 >>>
  Hi everyone:
 
  flink-mapr-fs failed in travis, and I retried many times, and also 
  failed.
  Anyone has idea about this?
 
  01:32:54.755 [ERROR] Failed to execute goal on project flink-mapr-fs:
  Could not resolve dependencies for project
  org.apache.flink:flink-mapr-fs:jar:1.10-SNAPSHOT: Failed to collect
  dependencies at com.mapr.hadoop:maprfs:jar:5.2.1-mapr: Failed to read
  artifact descriptor for com.mapr.hadoop:maprfs:jar:5.2.1-mapr: 
  Could not
  transfer artifact com.mapr.hadoop:maprfs:pom:5.2.1-mapr from/to
  mapr-releases (https://repository.mapr.com/maven/):
  sun.security.validator.ValidatorException: PKIX path building failed:
  sun.security.provider.certpath.SunCertPathBuilderException: unable 
  to find
  valid certification path to requested target -> [Help 1]
 
  https://api.travis-ci.org/v3/job/560790299/log.txt
 
  Best, Jingsong Lee
 >>
 >>
 >>
 >
 >

Re: [DISCUSS] Allow at-most-once delivery in case of failures

[Biao Liu]

Hi Stephan & Xiaogang,

It's great to see this discussion active again!

It makes sense to me that doing some private optimization and trial through
plugin. I understand that the community could not satisfy every one and
every requirement due to limited resources. The pluggable strategy is a
good way to compromise. In that way, it might be also helpful for improving
the pluggable strategy itself since there might be some reasonable
requirements from the plugin.

Regarding to the "at-most-once" or "best-effort" semantics, I think it
worths going further since we heard these requirements several times.
However I think we need more investigations of implementing based on
pluggable shuffle service and scheduler (or some more components?). There
might be a public discussion when we are ready. I hope it would happen soon.


On Wed, Jul 24, 2019 at 9:43 AM SHI Xiaogang  wrote:

> Hi Stephan,
>
> I agree with you that  the implementation of "at-most-once" or
> "best-effort" recovery will benefit from pluggable shuffle service and
> pluggable scheduler.  Actually we made some attempts in our private
> repository and it turns out that it requires quite a lot of work to
> implement this with exsiting network stack. We can start the work on this
> when pluggable shuffle service and pluggable scheduler are ready.
>
> The suggestion of external implementation is a very good idea. That way, we
> can implement both "at-most-once" and "best-effort" guarantees as different
> checkpoint/failover strategies. If so, i think we should focus on the
> components that are changed in different strategies. These components may
> include a pluggable checkpoint barrier handler and a pluggable failover
> strategy. We can list these components and discuss implementation details
> then.
>
> What do you think, Biao Liu and Zhu Zhu?
>
> Regards,
> Xiaogang
>
>
> Stephan Ewen  于2019年7月24日周三 上午1:31写道：
>
> > Hi all!
> >
> > This is an interesting discussion for sure.
> >
> > Concerning user requests for changes modes, I also hear the following
> quite
> > often:
> >   - reduce the expensiveness of checkpoint alignment (unaligned
> > checkpoints) to make checkpoints fast/stable under high backpressure
> >   - more fine-grained failover while maintaining exactly-once (even if
> > costly)
> >
> > Having also "at most once" to the mix is quite a long list of big changes
> > to the system.
> >
> > My feeling is that on such a core system, the community can not push all
> > these efforts at the same time, especially because they touch overlapping
> > areas of the system and need the same committers involved.
> >
> > On the other hand, the pluggable shuffle service and pluggable scheduler
> > could make it possible to have an external implementation of that.
> >   - of a network stack that supports "reconnects" of failed tasks with
> > continuing tasks
> >   - a scheduling strategy that restarts tasks individually even in
> > pipelined regions
> >
> > I think contributors/committers could implements this separate from the
> > Flink core. The feature would be trial-run it through the community
> > packages. If it gains a lot of traction, the community could decide to
> put
> > in the effort to merge this into the core.
> >
> > Best,
> > Stephan
> >
> >
> > On Tue, Jun 11, 2019 at 2:10 PM SHI Xiaogang 
> > wrote:
> >
> > > Hi All,
> > >
> > > It definitely requires a massive effort to allow at-most-once delivery
> in
> > > Flink. But as the feature is urgently demanded by many Flink users, i
> > think
> > > every effort we made is worthy. Actually, the inability to support
> > > at-most-once delivery has become a major obstacle for Storm users to
> turn
> > > to Flink. It's undesirable for us to run different stream processing
> > > systems for different scenarios.
> > >
> > > I agree with Zhu Zhu that the guarantee we provide is the very first
> > thing
> > > to be discussed. Recovering with checkpoints will lead to duplicated
> > > records, thus break the guarantee on at-most-once delivery.
> > >
> > > A method to achieve at-most-once guarantee is to completely disable
> > > checkpointing and let sources only read those records posted after they
> > > start. The method requires sources to allow the configuration to read
> > > latest records, which luckily is supported by many message queues
> > including
> > > Kafka. As Flink relies sources' ability to rollback to achieve
> exact-only
> > > and at-least-once delivery, i think it's acceptable for Flink to rely
> > > sources' ability to read latest records to achieve at-most once
> delivery.
> > > This method does not require any modification to existing checkpointing
> > > mechanism. Besides, as there is no need to restoring from checkpoints,
> > > failed tasks can recover themselves at the fastest speed.
> > >
> > > Concerning the implementation efforts, i think we can benefit from some
> > > ongoing work including shuffle services and fine-grained recovery. For
> > > example, currently the 

Re: [DISCUSS] Flink project bylaws

[Becket Qin]

Hi folks,

Thanks for all the feedback.

It seems that there are a few concerns over the emeritus status after 6
months of inactivity. Given that the main purpose is just to make sure 2/3
majority can pass and we sort of have a solution for that, I just updated
the draft with the following changes:

1. Removed the inactivity term for emeritus committers / PMCs. A committer
/ PMC will only be considered emeritus by their own claim.
2. Removed the approval process for reinstatement of the emeritus
committers / PMCs. An emeritus committer / PMC will be reinstated when they
send an email to the priv...@flink.apache.org.
3. Adde the term to ensure 2/3 majority voting is still doable when there
are non-emeritus committers / PMCs who do not cast the vote.

Please let me know if you have any further thoughts.

Thanks,

Jiangjie (Becket) Qin

On Tue, Jul 23, 2019 at 10:18 AM Becket Qin  wrote:

> Hi Fabian,
>
> Thanks for the feedback.
>
> I agree that if we don't like emeritus committers / PMCs, we don't have to
> do it. That said, emeritus status simply means whether an individual is
> active / inactive in the community. It does not make the merits earned to
> go away. For our purpose, emeritus status is mostly just a way to make 2/3
> majority possible. As you noticed, since reaching out to binding voters who
> have not voted can achieve the same goal, the emeritus status is more of an
> optimization so we don't have to ping the inactive binding voters every
> time and wait for long. However, given that 2/3 majority votings are rare,
> such communication cost is probably OK. So I think we can remove that
> emeritus part from the bylaws.
>
> 1) We should add to the requirements of the PMC that they need to make
>> sure the project complies with brand issues and reports misuse of ASF
>> brands.
>
> Good point. Added.
>
> 2) Do we want to restrict voting days to working days, i.e., a 3 day vote
>> that starts on Friday 11:00am ends on Wednesday 11:00am?
>
> This might be a little tricky because people are from countries in
> different time zones and with different holidays, and so on. If we are
> worrying about 3 days minimum length is not enough for those who want to
> give feedback, we can make it 5 days.
>
> 3) Do we need a process do decide about removal of features (like the
>> DataSet API for instance or the legacy DataSet/DataStream Python APIs)?
>
> I assume such action should be covered by FLIP as it is a change to the
> API and probably needs a migration plan. It would be useful to have a
> formal deprecation procedure. But that might be better to be put into
> somewhere else because the bylaws are primarily focusing on the
> non-technical rules, whereas the deprecation seems more on the technical
> side.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Jul 22, 2019 at 11:42 PM Fabian Hueske  wrote:
>
>> Hi all,
>>
>> First of all thank you very much Becket for starting this discussion!
>> I think it is a very good idea and overdue to formally define some of our
>> community processes.
>>
>> Similar to Chesnay, I have concerns about the distinction between active
>> and non-active / emeritus committers and PMC members.
>> My foremost concern is that this is not in the spirit of the Apache Way
>> [1]
>> which is (among other things) based on the idea of merit which once
>> earned,
>> does not go away over time.
>> I know other projects like Hadoop and Kafka have similar clauses in the
>> bylaws but IMO we don't need to adopt them if we don't like them.
>> For example, I don't like the idea that committers or PMC members who are
>> temporarily away from the project (for whatever reason: parental leave,
>> sabbatical, health issues, etc.) need the PMC approval to be "active"
>> again.
>> As far as I am aware, we have not seen any issues with inactive members in
>> the past.
>> Moreover, it would be hard to track whether somebody became inactive at
>> some point in time (which we would need to do, if I understand the
>> proposal
>> correctly).
>> With the approach that Becket suggested in his last email (reaching out to
>> binding voters who haven't voted yet), we could drop the distinction
>> between active and non-active committers and PMC members.
>>
>> I also have a few minor comments:
>>
>> 1) We should add to the requirements of the PMC [2] that they need to make
>> sure the project complies with brand issues and reports misuse of ASF
>> brands.
>> 2) Do we want to restrict voting days to working days, i.e., a 3 day vote
>> that starts on Friday 11:00am ends on Wednesday 11:00am?
>> 3) Do we need a process do decide about removal of features (like the
>> DataSet API for instance or the legacy DataSet/DataStream Python APIs)?
>>
>> Thank you,
>> Fabian
>>
>> [1] https://www.apache.org/theapacheway/
>> [2] https://www.apache.org/dev/pmc.html
>>
>> Am So., 21. Juli 2019 um 13:22 Uhr schrieb Becket Qin <
>> becket@gmail.com
>> >:
>>
>> > Hi Hequn,
>> >
>> > Thanks for sharing your thoughts. Please see 

Re: Probing of simple repartition hash join

[Caizhi Weng]

Hi Benjamin,

As you mentioned hash join I assume that you are referring to
`HashJoinOperator` in blink planner.

The input is selected by `nextSelection` method. As you can see, it will
first read all records in the build side then read all records in the probe
side. So the probing will only start after the build side has been fully
received.

Apart from this specific question, I'm interested in how you're going to
implement a hash join where any of the two sides can be read. Could you
share your ideas or give some hints about this? Thanks a lot.

Benjamin Burkhardt  于2019年7月24日周三 上午1:24写道：

> Hi all,
>
> Let’s imagine a simple repartition hash Join oft two tables.
>
> As soon as the first table is hashed completely (all EndOfPartition Events
> sent) the shipping and probing of the second table starts.
>
> What I can’t find:
>
> 1. What triggers to start the probing exactly?
> 2. Where can I find it in the code?
>
>
> My final goal is to change the 2-phase join mechanism to a mixed
> implementation where probing for finished subpartitions begins earlier.
>
> I appreciate any help.
>
> Thanks.
>
> Benjamin
>

Re: [DISCUSS] Publish the PyFlink into PyPI

[Jeff Zhang]

+1 for publishing pyflink to pypi.

Regarding including jar, I just want to make sure which flink binary
distribution we would ship with pyflink since we have multiple flink binary
distributions (w/o hadoop).
Personally, I prefer to use the hadoop-included binary distribution.

And I just want to confirm whether it is possible for users to use a
different flink binary distribution as long as he set env FLINK_HOME.

Besides that, I hope that there will be bi-direction link reference between
flink doc and pypi doc.



Stephan Ewen  于2019年7月24日周三 上午12:07写道：

> Hi!
>
> Sorry for the late involvement. Here are some thoughts from my side:
>
> Definitely +1 to publishing to PyPy, even if it is a binary release.
> Community growth into other communities is great, and if this is the
> natural way to reach developers in the Python community, let's do it. This
> is not about our convenience, but reaching users.
>
> I think the way to look at this is that this is a convenience distribution
> channel, courtesy of the Flink community. It is not an Apache release, we
> make this clear in the Readme.
> Of course, this doesn't mean we don't try to uphold similar standards as
> for our official release (like proper license information).
>
> Concerning credentials sharing, I would be fine with whatever option. The
> PMC doesn't own it (it is an initiative by some community members), but the
> PMC needs to ensure trademark compliance, so slight preference for option
> #1 (PMC would have means to correct problems).
>
> I believe there is no need to differentiate between Scala versions, because
> this is merely a convenience thing for pure Python users. Users that mix
> python and scala (and thus depend on specific scala versions) can still
> download from Apache or build themselves.
>
> Best,
> Stephan
>
>
>
> On Thu, Jul 4, 2019 at 9:51 AM jincheng sun 
> wrote:
>
> > Hi All,
> >
> > Thanks for the feedback @Chesnay Schepler  @Dian!
> >
> > I think using `apache-flink` for the project name also makes sense to me.
> > due to we should always keep in mind that Flink is owned by Apache. (And
> > beam also using this pattern `apache-beam` for Python API)
> >
> > Regarding the Python API release with the JAVA JARs, I think the
> principle
> > of consideration is the convenience of the user. So, Thanks for the
> > explanation @Dian!
> >
> > And your right @Chesnay Schepler   we can't make a
> > hasty decision and we need more people's opinions！
> >
> > So, I appreciate it if anyone can give us feedback and suggestions!
> >
> > Best,
> > Jincheng
> >
> >
> >
> >
> > Chesnay Schepler  于2019年7月3日周三 下午8:46写道：
> >
> > > So this would not be a source release then, but a full-blown binary
> > > release.
> > >
> > > Maybe it is just me, but I find it a bit suspect to ship an entire java
> > > application via PyPI, just because there's a Python API for it.
> > >
> > > We definitely need input from more people here.
> > >
> > > On 03/07/2019 14:09, Dian Fu wrote:
> > > > Hi Chesnay,
> > > >
> > > > Thanks a lot for the suggestions.
> > > >
> > > > Regarding “distributing java/scala code to PyPI”:
> > > > The Python Table API is just a wrapper of the Java Table API and
> > without
> > > the java/scala code, two steps will be needed to set up an environment
> to
> > > execute a Python Table API program:
> > > > 1) Install pyflink using "pip install apache-flink"
> > > > 2) Download the flink distribution and set the FLINK_HOME to it.
> > > > Besides, users have to make sure that the manually installed Flink is
> > > compatible with the pip installed pyflink.
> > > >
> > > > Bundle the java/scala code inside the Python package will eliminate
> > step
> > > 2) and makes it more simple for users to install pyflink. There was a
> > short
> > > discussion  on this
> in
> > > Spark community and they finally decide to package the java/scala code
> in
> > > the python package. (BTW, PySpark only bundle the jars of scala 2.11).
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > >> 在 2019年7月3日，下午7:13，Chesnay Schepler  写道：
> > > >>
> > > >> The existing artifact in the pyflink project was neither released by
> > > the Flink project / anyone affiliated with it nor approved by the Flink
> > PMC.
> > > >>
> > > >> As such, if we were to use this account I believe we should delete
> it
> > > to not mislead users that this is in any way an apache-provided
> > > distribution. Since this goes against the users wishes, I would be in
> > favor
> > > of creating a separate account, and giving back control over the
> pyflink
> > > account.
> > > >>
> > > >> My take on the raised points:
> > > >> 1.1) "apache-flink"
> > > >> 1.2)  option 2
> > > >> 2) Given that we only distribute python code there should be no
> reason
> > > to differentiate between scala versions. We should not be distributing
> > any
> > > java/scala code and/or modules to PyPi. Currently, I'm a bit confused
> > about
> > > this question and 

[jira] [Created] (FLINK-13392) WindowAggregate inherited from Aggregate incorrectly

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-13392:
---

 Summary: WindowAggregate inherited from Aggregate incorrectly
 Key: FLINK-13392
 URL: https://issues.apache.org/jira/browse/FLINK-13392
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Hequn Cheng


As discussed in FLINK-12249, the WindowAggregate inherited from Aggregate 
incorrectly.

For WindowAggregate, the group keys are window group and normal fields (may be 
empty), while Aggregate only has normal group keys part, and know nothing about 
window group key. Currently, many planner rules match and apply transformations 
on Aggregate, however some of them does not applicable to WindowAggregate, e.g. 
AggregateJoinTransposeRule, AggregateProjectMergeRule, etc.

Although FLINK-12249 fixes the type equivalence check problem, we should do a 
step further to correct the WindowAggregate behavior. There are three options 
now:
 # make Aggregate's group key supports expressions(such as RexCall), not field 
reference only. and then the window group expression could be as a part of 
Aggregate's group key. the disadvantage is we must update all existing 
aggregate rules, metadata handlers, etc.
 # make WindowAggregate extends from SingleRel, not from Aggregate. the 
disadvantage is we must implement related planner rules about WindowAggregate.
 # in logical phase, we does not merge Aggregate and Project (with window 
group) into WindowAggregate, and convert the Project to a new kind of node 
named WindowAssigner, which could prevent Project from being pushed 
down/merged. and in physical phase, we merge them into WindowAggregate. the 
advantage is we could reuse current aggregate rules, and the disadvantage is we 
should add new rules about WindowAssigner.

We could have some further discussions in the jira ticket.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Allow at-most-once delivery in case of failures

[SHI Xiaogang]

Hi Stephan,

I agree with you that  the implementation of "at-most-once" or
"best-effort" recovery will benefit from pluggable shuffle service and
pluggable scheduler.  Actually we made some attempts in our private
repository and it turns out that it requires quite a lot of work to
implement this with exsiting network stack. We can start the work on this
when pluggable shuffle service and pluggable scheduler are ready.

The suggestion of external implementation is a very good idea. That way, we
can implement both "at-most-once" and "best-effort" guarantees as different
checkpoint/failover strategies. If so, i think we should focus on the
components that are changed in different strategies. These components may
include a pluggable checkpoint barrier handler and a pluggable failover
strategy. We can list these components and discuss implementation details
then.

What do you think, Biao Liu and Zhu Zhu?

Regards,
Xiaogang


Stephan Ewen  于2019年7月24日周三 上午1:31写道：

> Hi all!
>
> This is an interesting discussion for sure.
>
> Concerning user requests for changes modes, I also hear the following quite
> often:
>   - reduce the expensiveness of checkpoint alignment (unaligned
> checkpoints) to make checkpoints fast/stable under high backpressure
>   - more fine-grained failover while maintaining exactly-once (even if
> costly)
>
> Having also "at most once" to the mix is quite a long list of big changes
> to the system.
>
> My feeling is that on such a core system, the community can not push all
> these efforts at the same time, especially because they touch overlapping
> areas of the system and need the same committers involved.
>
> On the other hand, the pluggable shuffle service and pluggable scheduler
> could make it possible to have an external implementation of that.
>   - of a network stack that supports "reconnects" of failed tasks with
> continuing tasks
>   - a scheduling strategy that restarts tasks individually even in
> pipelined regions
>
> I think contributors/committers could implements this separate from the
> Flink core. The feature would be trial-run it through the community
> packages. If it gains a lot of traction, the community could decide to put
> in the effort to merge this into the core.
>
> Best,
> Stephan
>
>
> On Tue, Jun 11, 2019 at 2:10 PM SHI Xiaogang 
> wrote:
>
> > Hi All,
> >
> > It definitely requires a massive effort to allow at-most-once delivery in
> > Flink. But as the feature is urgently demanded by many Flink users, i
> think
> > every effort we made is worthy. Actually, the inability to support
> > at-most-once delivery has become a major obstacle for Storm users to turn
> > to Flink. It's undesirable for us to run different stream processing
> > systems for different scenarios.
> >
> > I agree with Zhu Zhu that the guarantee we provide is the very first
> thing
> > to be discussed. Recovering with checkpoints will lead to duplicated
> > records, thus break the guarantee on at-most-once delivery.
> >
> > A method to achieve at-most-once guarantee is to completely disable
> > checkpointing and let sources only read those records posted after they
> > start. The method requires sources to allow the configuration to read
> > latest records, which luckily is supported by many message queues
> including
> > Kafka. As Flink relies sources' ability to rollback to achieve exact-only
> > and at-least-once delivery, i think it's acceptable for Flink to rely
> > sources' ability to read latest records to achieve at-most once delivery.
> > This method does not require any modification to existing checkpointing
> > mechanism. Besides, as there is no need to restoring from checkpoints,
> > failed tasks can recover themselves at the fastest speed.
> >
> > Concerning the implementation efforts, i think we can benefit from some
> > ongoing work including shuffle services and fine-grained recovery. For
> > example, currently the exceptions in network connections will lead to
> > failures of downstream and upstream tasks. To achieve at-most-once
> > delivery, we should decouple intermediate results from tasks, reporting
> the
> > exceptions of intermediate results to job master and letting the failover
> > strategy to determine the actions taken. Some work is already done in the
> > efforts to achieve fine-grained recovery, which can be extended to allow
> > at-most-once delivery in Flink.
> >
> > But before starting the discussion on implementation details, as said at
> > prior, we need to determine the guarantee we provide in the scenarios
> where
> > timely recovery is needed.
> > * What do you think of the at-most-once guarantee achieved by the
> proposed
> > method?
> > * Do we need checkpointing to reduce the amount of lost data?
> > * Do we need deduplication to guarantee at-most-once delivery or just
> > provide best-effort delivery?
> >
> > Regards,
> > Xiaogang Shi
> >
> >
> > Piotr Nowojski  于2019年6月11日周二 下午5:31写道：
> >
> > > Hi Xiaogang,
> > >
> > > It sounds interesting and 

[jira] [Created] (FLINK-13391) Blink-planner should not invoke deprecated getReturnType

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-13391:


 Summary: Blink-planner should not invoke deprecated getReturnType
 Key: FLINK-13391
 URL: https://issues.apache.org/jira/browse/FLINK-13391
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Jingsong Lee
 Fix For: 1.9.0, 1.10.0


Now, blink-planner will invoke getDataStream of InputFormatTableSource, this 
will invoke deprecated getReturnType method.

We should invoke getInputFormat of InputFormatTableSource to be same as 
flink-planner.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Flink client api enhancement for downstream project

[Jeff Zhang]

Thanks Stephan, I will follow up this issue in next few weeks, and will
refine the design doc. We could discuss more details after 1.9 release.

Stephan Ewen  于2019年7月24日周三 上午12:58写道：

> Hi all!
>
> This thread has stalled for a bit, which I assume ist mostly due to the
> Flink 1.9 feature freeze and release testing effort.
>
> I personally still recognize this issue as one important to be solved. I'd
> be happy to help resume this discussion soon (after the 1.9 release) and
> see if we can do some step towards this in Flink 1.10.
>
> Best,
> Stephan
>
>
>
> On Mon, Jun 24, 2019 at 10:41 AM Flavio Pompermaier 
> wrote:
>
> > That's exactly what I suggested a long time ago: the Flink REST client
> > should not require any Flink dependency, only http library to call the
> REST
> > services to submit and monitor a job.
> > What I suggested also in [1] was to have a way to automatically suggest
> the
> > user (via a UI) the available main classes and their required
> > parameters[2].
> > Another problem we have with Flink is that the Rest client and the CLI
> one
> > behaves differently and we use the CLI client (via ssh) because it allows
> > to call some other method after env.execute() [3] (we have to call
> another
> > REST service to signal the end of the job).
> > Int his regard, a dedicated interface, like the JobListener suggested in
> > the previous emails, would be very helpful (IMHO).
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-10864
> > [2] https://issues.apache.org/jira/browse/FLINK-10862
> > [3] https://issues.apache.org/jira/browse/FLINK-10879
> >
> > Best,
> > Flavio
> >
> > On Mon, Jun 24, 2019 at 9:54 AM Jeff Zhang  wrote:
> >
> > > Hi, Tison,
> > >
> > > Thanks for your comments. Overall I agree with you that it is difficult
> > for
> > > down stream project to integrate with flink and we need to refactor the
> > > current flink client api.
> > > And I agree that CliFrontend should only parsing command line arguments
> > and
> > > then pass them to ExecutionEnvironment. It is ExecutionEnvironment's
> > > responsibility to compile job, create cluster, and submit job. Besides
> > > that, Currently flink has many ExecutionEnvironment implementations,
> and
> > > flink will use the specific one based on the context. IMHO, it is not
> > > necessary, ExecutionEnvironment should be able to do the right thing
> > based
> > > on the FlinkConf it is received. Too many ExecutionEnvironment
> > > implementation is another burden for downstream project integration.
> > >
> > > One thing I'd like to mention is flink's scala shell and sql client,
> > > although they are sub-modules of flink, they could be treated as
> > downstream
> > > project which use flink's client api. Currently you will find it is not
> > > easy for them to integrate with flink, they share many duplicated code.
> > It
> > > is another sign that we should refactor flink client api.
> > >
> > > I believe it is a large and hard change, and I am afraid we can not
> keep
> > > compatibility since many of changes are user facing.
> > >
> > >
> > >
> > > Zili Chen  于2019年6月24日周一 下午2:53写道：
> > >
> > > > Hi all,
> > > >
> > > > After a closer look on our client apis, I can see there are two major
> > > > issues to consistency and integration, namely different deployment of
> > > > job cluster which couples job graph creation and cluster deployment,
> > > > and submission via CliFrontend confusing control flow of job graph
> > > > compilation and job submission. I'd like to follow the discuss above,
> > > > mainly the process described by Jeff and Stephan, and share my
> > > > ideas on these issues.
> > > >
> > > > 1) CliFrontend confuses the control flow of job compilation and
> > > submission.
> > > > Following the process of job submission Stephan and Jeff described,
> > > > execution environment knows all configs of the cluster and
> > topos/settings
> > > > of the job. Ideally, in the main method of user program, it calls
> > > #execute
> > > > (or named #submit) and Flink deploys the cluster, compile the job
> graph
> > > > and submit it to the cluster. However, current CliFrontend does all
> > these
> > > > things inside its #runProgram method, which introduces a lot of
> > > subclasses
> > > > of (stream) execution environment.
> > > >
> > > > Actually, it sets up an exec env that hijacks the
> #execute/executePlan
> > > > method, initializes the job graph and abort execution. And then
> > > > control flow back to CliFrontend, it deploys the cluster(or retrieve
> > > > the client) and submits the job graph. This is quite a specific
> > internal
> > > > process inside Flink and none of consistency to anything.
> > > >
> > > > 2) Deployment of job cluster couples job graph creation and cluster
> > > > deployment. Abstractly, from user job to a concrete submission, it
> > > requires
> > > >
> > > >  create JobGraph --\
> > > >
> > > > create ClusterClient -->  submit JobGraph
> > > >
> > > > such a dependency. 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Becket Qin]

Congrats, Kurt. Well deserved!

Jiangjie (Becket) Qin

On Wed, Jul 24, 2019 at 1:11 AM Xuefu Z  wrote:

> Congratulations, Kurt!
>
> On Tue, Jul 23, 2019 at 7:48 AM Fabian Hueske  wrote:
>
> > Congrats Kurt!
> >
> > Cheers, Fabian
> >
> > Am Di., 23. Juli 2019 um 16:42 Uhr schrieb Rong Rong <
> walter...@gmail.com
> > >:
> >
> > > Congratulations Kurt!!
> > >
> > > --
> > > Rong
> > >
> > > On Tue, Jul 23, 2019 at 7:31 AM zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > Congrats Kurt!
> > > >
> > > > Best,
> > > > Zhijiang
> > > > --
> > > > From:Till Rohrmann 
> > > > Send Time:2019年7月23日(星期二) 21:08
> > > > To:dev 
> > > > Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC
> > > >
> > > > Congrats Kurt!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz <
> > dwysakow...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 23/07/2019 13:39, Hequn Cheng wrote:
> > > > > > Congratulations Kurt!
> > > > > >
> > > > > > Best, Hequn
> > > > > >
> > > > > > On Tue, Jul 23, 2019 at 7:27 PM vino yang  >
> > > > wrote:
> > > > > >
> > > > > >> Congratulations Kurt!
> > > > > >>
> > > > > >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> > > > > >>
> > > > > >>> Congratulations Kurt!
> > > > > >>>
> > > > > >>>
> > > > > >>> Best,
> > > > > >>>
> > > > > >>> Bo WANG
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger <
> > > rmetz...@apache.org>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Hi all,
> > > > > 
> > > > >  On behalf of the Flink PMC, I'm happy to announce that Kete
> > Young
> > > is
> > > > > >> now
> > > > >  part of the Apache Flink Project Management Committee (PMC).
> > > > > 
> > > > >  Kete has been a committer since February 2017, working a lot
> on
> > > > Table
> > > > > >>> API /
> > > > >  SQL. He's currently co-managing the 1.9 release! Thanks a lot
> > for
> > > > your
> > > > > >>> work
> > > > >  for Flink!
> > > > > 
> > > > >  Congratulations & Welcome Kurt!
> > > > > 
> > > > >  Best,
> > > > >  Robert
> > > > > 
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
>
> --
> Xuefu Zhang
>
> "In Honey We Trust!"
>

Re: [DISCUSS] Flink client api enhancement for downstream project

[Zili Chen]

Hi Stephan,

Thanks for waking up this thread.

Jeff and I had a discussion yesterday sharing our respective
observations and ideas of client api enhancements. We are
glad to make some progress in Flink 1.10.

It's really nice to hear that you're gonna participate this
thread soon. In ML threads we can see a consensus to client
api enhancement and with several offline discussions we have
rough ideas on how to introduce these enhancements. It's
hopeful that we draft a FLIP based on Jeff's doc and the
discussions so far. And it is appreciated that some of our
committers/PMCs could shepherd the FLIP.

By the way, there is separated thread concurrently happening[1],
which, besides of the major enhancements, helps make progress in
refactoring client api.

Best,
tison.

[1]
https://lists.apache.org/thread.html/7ffc9936a384b891dbcf0a481d26c6d13b2125607c200577780d1e18@%3Cdev.flink.apache.org%3E


Stephan Ewen  于2019年7月24日周三 上午12:58写道：

> Hi all!
>
> This thread has stalled for a bit, which I assume ist mostly due to the
> Flink 1.9 feature freeze and release testing effort.
>
> I personally still recognize this issue as one important to be solved. I'd
> be happy to help resume this discussion soon (after the 1.9 release) and
> see if we can do some step towards this in Flink 1.10.
>
> Best,
> Stephan
>
>
>
> On Mon, Jun 24, 2019 at 10:41 AM Flavio Pompermaier 
> wrote:
>
> > That's exactly what I suggested a long time ago: the Flink REST client
> > should not require any Flink dependency, only http library to call the
> REST
> > services to submit and monitor a job.
> > What I suggested also in [1] was to have a way to automatically suggest
> the
> > user (via a UI) the available main classes and their required
> > parameters[2].
> > Another problem we have with Flink is that the Rest client and the CLI
> one
> > behaves differently and we use the CLI client (via ssh) because it allows
> > to call some other method after env.execute() [3] (we have to call
> another
> > REST service to signal the end of the job).
> > Int his regard, a dedicated interface, like the JobListener suggested in
> > the previous emails, would be very helpful (IMHO).
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-10864
> > [2] https://issues.apache.org/jira/browse/FLINK-10862
> > [3] https://issues.apache.org/jira/browse/FLINK-10879
> >
> > Best,
> > Flavio
> >
> > On Mon, Jun 24, 2019 at 9:54 AM Jeff Zhang  wrote:
> >
> > > Hi, Tison,
> > >
> > > Thanks for your comments. Overall I agree with you that it is difficult
> > for
> > > down stream project to integrate with flink and we need to refactor the
> > > current flink client api.
> > > And I agree that CliFrontend should only parsing command line arguments
> > and
> > > then pass them to ExecutionEnvironment. It is ExecutionEnvironment's
> > > responsibility to compile job, create cluster, and submit job. Besides
> > > that, Currently flink has many ExecutionEnvironment implementations,
> and
> > > flink will use the specific one based on the context. IMHO, it is not
> > > necessary, ExecutionEnvironment should be able to do the right thing
> > based
> > > on the FlinkConf it is received. Too many ExecutionEnvironment
> > > implementation is another burden for downstream project integration.
> > >
> > > One thing I'd like to mention is flink's scala shell and sql client,
> > > although they are sub-modules of flink, they could be treated as
> > downstream
> > > project which use flink's client api. Currently you will find it is not
> > > easy for them to integrate with flink, they share many duplicated code.
> > It
> > > is another sign that we should refactor flink client api.
> > >
> > > I believe it is a large and hard change, and I am afraid we can not
> keep
> > > compatibility since many of changes are user facing.
> > >
> > >
> > >
> > > Zili Chen  于2019年6月24日周一 下午2:53写道：
> > >
> > > > Hi all,
> > > >
> > > > After a closer look on our client apis, I can see there are two major
> > > > issues to consistency and integration, namely different deployment of
> > > > job cluster which couples job graph creation and cluster deployment,
> > > > and submission via CliFrontend confusing control flow of job graph
> > > > compilation and job submission. I'd like to follow the discuss above,
> > > > mainly the process described by Jeff and Stephan, and share my
> > > > ideas on these issues.
> > > >
> > > > 1) CliFrontend confuses the control flow of job compilation and
> > > submission.
> > > > Following the process of job submission Stephan and Jeff described,
> > > > execution environment knows all configs of the cluster and
> > topos/settings
> > > > of the job. Ideally, in the main method of user program, it calls
> > > #execute
> > > > (or named #submit) and Flink deploys the cluster, compile the job
> graph
> > > > and submit it to the cluster. However, current CliFrontend does all
> > these
> > > > things inside its #runProgram method, which introduces a lot of
> > > 

Re: [DISCUSS] Support temporary tables in SQL API

[Xuefu Z]

I favored #3 if that wasn't obvious.

Usability issue with #2 makes Hive  too hard to use. #3 keeps the old
behavior for existing users who don't have Hive and thus there is only one,
in-memory catalog. If a user does register Hive, he/she understands that
there are multiple catalogs and that fully qualified table name is
necessary. Thus, #3 has no impact (and no surprises) for existing users,
and new requirement of fully qualified names is for only for users of the
new feature (multiple catalogs), which seems very natural.

Thanks,
Xuefu

On Tue, Jul 23, 2019 at 5:47 AM Dawid Wysakowicz 
wrote:

> I think we all agree so far that we should implement one of the short term
> solutions for 1.9 release (#2 or #3) and continue the discussion on option
> #1 for the next release. Personally I prefer option #2, because it is
> closest to the current behavior and as Kurt said it is the most intuitive
> one, but I am also fine with option #3
>
> To sum up the opinions so far:
>
> *option #2: 3 votes(Kurt, Aljoscha, me)*
>
> *option #3: 2 votes(Timo, Jingsong)*
>
> I wasn't sure which option out of the two Xuefu prefers.
>
> I would like to conclude the discussion by the end of tomorrow, so that we
> can prepare a proper fix as soon as possible. Therefore I would suggest to
> proceed with the option that gets the most votes until tomorrow (*July
> 24th 12:00 CET*), unless there are some hard objections.
>
>
> Comment on option #1 concerns:
>
> I agree with Jingsong on that. I think there are some benefits of the
> approach, as it makes Flink in control of the temporary tables.
>
> 1. We have a unified behavior across all catalogs. Also for the catalogs
> that do not support temporary tables natively.
>
> 2. As Flink is in control of the temporary tables it makes it easier to
> control their lifecycle.
>
> Best,
>
> Dawid
> On 23/07/2019 11:40, JingsongLee wrote:
>
> And I think we should recommend user to use catalog api to
>  createTable and createFunction,(I guess most scenarios do
>  not use temporary objects) in this way, it is good to option #3
>
> Best, JingsongLee
>
>
> --
> From:JingsongLee  
> 
> Send Time:2019年7月23日(星期二) 17:35
> To:dev  
> Subject:Re: [DISCUSS] Support temporary tables in SQL API
>
> Thanks Dawid and other people.
> +1 for using option #3 for 1.9.0 and go with option #1
>  in 1.10.0.
>
> Regarding Xuefu's concern, I don't know how necessary it is for each catalog 
> to
>  deal with tmpView. I think Catalog is different from DB, we can have single 
> concept for tmpView, that make user easier to understand.
>
> Regarding option #2, It is hard to use if we let user to use fully qualified 
> name for hive catalog. Would this experience be too bad to use?
>
> Best, Jingsong Lee
>
>
> --
> From:Kurt Young  
> Send Time:2019年7月23日(星期二) 17:03
> To:dev  
> Subject:Re: [DISCUSS] Support temporary tables in SQL API
>
> Thanks Dawid for driving this discussion.
> Personally, I would +1 for using option #2 for 1.9.0 and go with option #1
> in 1.10.0.
>
> Regarding Xuefu's concern about option #1, I think we could also try to
> reuse the in-memory catalog
> for the builtin temporary table storage.
>
> Regarding to option #2 and option #3, from user's perspective, IIUC option
> #2 allows user to have
> simple name to reference temporary table and should use fully qualified
> name for external catalogs.
> But option #3 provide the opposite behavior, user can use simple name for
> external tables after he
> changed current catalog and current database, but have to use fully
> qualified name for temporary
> tables. IMO, option #2 will be more straightforward.
>
> Best,
> Kurt
>
>
> On Tue, Jul 23, 2019 at 4:01 PM Aljoscha Krettek  
> 
> wrote:
>
>
> I would be fine with option 3) but I think option 2) is the more implicit
> solution that has less surprising behaviour.
>
> Aljoscha
>
>
> On 22. Jul 2019, at 23:59, Xuefu Zhang   
> wrote:
>
> Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
> that for 1.9 we should have some quick and simple solution, leaving time
> for more thorough discussions for 1.10.
>
> In particular, I'm not fully with solution #1. For one thing, it seems
> proposing storing all temporary objects in a memory map in
>
> CatalogManager,
>
> and the memory map duplicates the functionality of the in-memory catalog,
> which also store temporary objects. For another, as pointed out by the
> google doc, different db may handle the temporary tables differently, and
> accordingly it may make more sense to let each catalog to handle its
> temporary objects.
>
> Therefore, postponing the fix buys us time to flush out all the details.
>
> Thanks,
> Xuefu
>
> On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  
>  wrote:
>
>
> Thanks for summarizing our offline discussion Dawid! Even though I would
> prefer solution 1 instead of releasing half-baked 

Re: [DISCUSS] Allow at-most-once delivery in case of failures

[Stephan Ewen]

Hi all!

This is an interesting discussion for sure.

Concerning user requests for changes modes, I also hear the following quite
often:
  - reduce the expensiveness of checkpoint alignment (unaligned
checkpoints) to make checkpoints fast/stable under high backpressure
  - more fine-grained failover while maintaining exactly-once (even if
costly)

Having also "at most once" to the mix is quite a long list of big changes
to the system.

My feeling is that on such a core system, the community can not push all
these efforts at the same time, especially because they touch overlapping
areas of the system and need the same committers involved.

On the other hand, the pluggable shuffle service and pluggable scheduler
could make it possible to have an external implementation of that.
  - of a network stack that supports "reconnects" of failed tasks with
continuing tasks
  - a scheduling strategy that restarts tasks individually even in
pipelined regions

I think contributors/committers could implements this separate from the
Flink core. The feature would be trial-run it through the community
packages. If it gains a lot of traction, the community could decide to put
in the effort to merge this into the core.

Best,
Stephan


On Tue, Jun 11, 2019 at 2:10 PM SHI Xiaogang  wrote:

> Hi All,
>
> It definitely requires a massive effort to allow at-most-once delivery in
> Flink. But as the feature is urgently demanded by many Flink users, i think
> every effort we made is worthy. Actually, the inability to support
> at-most-once delivery has become a major obstacle for Storm users to turn
> to Flink. It's undesirable for us to run different stream processing
> systems for different scenarios.
>
> I agree with Zhu Zhu that the guarantee we provide is the very first thing
> to be discussed. Recovering with checkpoints will lead to duplicated
> records, thus break the guarantee on at-most-once delivery.
>
> A method to achieve at-most-once guarantee is to completely disable
> checkpointing and let sources only read those records posted after they
> start. The method requires sources to allow the configuration to read
> latest records, which luckily is supported by many message queues including
> Kafka. As Flink relies sources' ability to rollback to achieve exact-only
> and at-least-once delivery, i think it's acceptable for Flink to rely
> sources' ability to read latest records to achieve at-most once delivery.
> This method does not require any modification to existing checkpointing
> mechanism. Besides, as there is no need to restoring from checkpoints,
> failed tasks can recover themselves at the fastest speed.
>
> Concerning the implementation efforts, i think we can benefit from some
> ongoing work including shuffle services and fine-grained recovery. For
> example, currently the exceptions in network connections will lead to
> failures of downstream and upstream tasks. To achieve at-most-once
> delivery, we should decouple intermediate results from tasks, reporting the
> exceptions of intermediate results to job master and letting the failover
> strategy to determine the actions taken. Some work is already done in the
> efforts to achieve fine-grained recovery, which can be extended to allow
> at-most-once delivery in Flink.
>
> But before starting the discussion on implementation details, as said at
> prior, we need to determine the guarantee we provide in the scenarios where
> timely recovery is needed.
> * What do you think of the at-most-once guarantee achieved by the proposed
> method?
> * Do we need checkpointing to reduce the amount of lost data?
> * Do we need deduplication to guarantee at-most-once delivery or just
> provide best-effort delivery?
>
> Regards,
> Xiaogang Shi
>
>
> Piotr Nowojski  于2019年6月11日周二 下午5:31写道：
>
> > Hi Xiaogang,
> >
> > It sounds interesting and definitely a useful feature, however the
> > questions for me would be how useful, how much effort would it require
> and
> > is it worth it? We simply can not do all things at once, and currently
> > people that could review/drive/mentor this effort are pretty much
> strained
> > :( For me one would have to investigate answers to those questions and
> > prioritise it compared to other ongoing efforts, before I could vote +1
> for
> > this.
> >
> > Couple of things to consider:
> > - would it be only a job manager/failure region recovery feature?
> > - would it require changes in CheckpointBarrierHandler,
> > CheckpointCoordinator classes?
> > - with `at-most-once` semantic theoretically speaking we could just drop
> > the current `CheckpointBarrier` handling/injecting code and avoid all of
> > the checkpoint alignment issues - we could just checkpoint all of the
> tasks
> > independently of one another. However maybe that could be a follow up
> > optimisation step?
> >
> > Piotrek
> >
> > > On 11 Jun 2019, at 10:53, Zili Chen  wrote:
> > >
> > > Hi Xiaogang,
> > >
> > > It is an interesting topic.
> > >
> > > Notice that there is some 

Probing of simple repartition hash join

[Benjamin Burkhardt]

Hi all,

Let’s imagine a simple repartition hash Join oft two tables.

As soon as the first table is hashed completely (all EndOfPartition Events 
sent) the shipping and probing of the second table starts.

What I can’t find:

1. What triggers to start the probing exactly?
2. Where can I find it in the code?


My final goal is to change the 2-phase join mechanism to a mixed implementation 
where probing for finished subpartitions begins earlier.

I appreciate any help.

Thanks.

Benjamin

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Hugo Louro]

Sounds good @stephan! Thanks.

On Tue, Jul 23, 2019 at 10:15 AM Stephan Ewen  wrote:

> @hugo For your suggestions, I would ask to start a separate discussion
> thread.
> I think this mail thread has converged towards merging the initial
> suggestion as a starting point and refining it later based on new
> discussions.
>
> Best,
> Stephan
>
>
> On Thu, Jun 27, 2019 at 10:48 PM Hugo Louro  wrote:
>
> > +1. Thanks for working on the guide. It's very thorough and a good
> resource
> > to learn good practices from.
> >
> > I would like use this thread as a placeholder for a couple of topics that
> > may be deserving of further discussion on different threads:
> >   - What's the best way to keep track of checkstyle version updates. For
> > instance, currently there is a PR
> >  proposing checkstyle to be
> > updated because version 8.12 is no longer supported
> >  - When classes import shaded dependencies, it is not trivial for
> IntelliJ
> > to download and associate sources and javadocs, which makes debugging and
> > navigate the code base harder. I tried installing the version of the
> > library using maven from the CLI, and associate the sources "manually" on
> > IntelliJ, but it seems it does not work (perhaps IntelliJ issue). Does
> > anyone know of a good solution? If so, we should added here
> > <
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/ide_setup.html#intellij-idea
> > >.
> > I can volunteer for that if you tell me how to do it :)
> > - did the community evaluate naming test methods according to these
> > conventions  ?
> >
> > Thanks
> >
> > On Mon, Jun 24, 2019 at 11:38 AM Stephan Ewen  wrote:
> >
> > > I think it makes sense to also start individual [DISCUSS] threads about
> > > extensions and follow-ups.
> > > Various suggestions came up, partly as comments in the doc, some as
> > > questions in other threads.
> > >
> > > Examples:
> > >   - use of null in return types versus Optional versus
> @Nullable/@Nonnull
> > >   - initialization of collection sizes
> > >   - logging
> > >
> > > I think these would be best discussed in separate threads each.
> > > So, for contributors to whom these issues are dear, feel free to spawn
> > > these additional threads.
> > > (Bear in mind it is close to 1.9 feature freeze time, so please leave
> > this
> > > discussions a bit of time so that all community members have a chance
> to
> > > participate)
> > >
> > >
> > >
> > > On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen  wrote:
> > >
> > > > Thanks for the pointer David.
> > > >
> > > > I was not aware of this tool and I have no experience with its
> > practical
> > > > impact. For example I would be curious what the effect of this is for
> > > > existing PRs, merge conflicts, etc.
> > > >
> > > > If you want, feel free to start another discuss thread on the
> > > introduction
> > > > of such a tool.
> > > >
> > > > On Sun, Jun 23, 2019 at 6:32 PM David Morávek 
> wrote:
> > > >
> > > >> I love this kind of unification being pushed forward, the benefit it
> > has
> > > >> on
> > > >> code reviews is enormous. Just my two cents, did you guys think
> about
> > > >> adopting an automatic code formatter for java, the same way as
> Apache
> > > Beam
> > > >> did?
> > > >>
> > > >> It is super easy to use for contributors as they don't need to keep
> > any
> > > >> particular coding style in mind and they can only focus on
> > functionality
> > > >> they want to fix, and it's also great for reviewers, because they
> only
> > > see
> > > >> the important changes. This also eliminates need for any special
> > editor
> > > /
> > > >> checkstyle configs as the code formatting is part of the build
> itself.
> > > >>
> > > >> The one Beam uses is https://github.com/diffplug/spotless with
> > > >> GoogleJavaFormat, it may be worth to look into.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen 
> > wrote:
> > > >>
> > > >> > Thanks, everyone, for the positive feedback :-)
> > > >> >
> > > >> > @Robert - It probably makes sense to break this down into various
> > > pages,
> > > >> > like PR, general code style guide, Java, component specific
> guides,
> > > >> > formats, etc.
> > > >> >
> > > >> > Best,
> > > >> > Stephan
> > > >> >
> > > >> >
> > > >> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger <
> rmetz...@apache.org
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > It seems that the discussion around this topic has settled.
> > > >> > >
> > > >> > > I'm going to turn the Google Doc into a markdown file (maybe
> also
> > > >> > multiple,
> > > >> > > I'll try out different things) and then open a pull request for
> > the
> > > >> Flink
> > > >> > > website.
> > > >> > > I'll post a link to the PR here once I'm done.
> > > >> > >
> > > >> > > On Fri, Jun 14, 2019 at 9:36 AM zhijiang <
> > > wangzhijiang...@aliyun.com
> > > >> > > .invalid>
> > > >> > > wrote:

[jira] [Created] (FLINK-13390) Clarify the exact meaning of state size when executing incremental checkpoint

[Yun Tang (JIRA)]

Yun Tang created FLINK-13390:


 Summary: Clarify the exact meaning of state size when executing 
incremental checkpoint
 Key: FLINK-13390
 URL: https://issues.apache.org/jira/browse/FLINK-13390
 Project: Flink
  Issue Type: Improvement
Reporter: Yun Tang
 Fix For: 1.10.0


This issue is inspired from [a user 
mail|https://lists.apache.org/thread.html/56069ce869afbfca66179e89788c05d3b092e3fe363f3540dcdeb7a1@%3Cuser.flink.apache.org%3E]
 which confused about the state size meaning.
I think changing the description of state size and add some notices on 
documentation could help this. Moreover, change the log when complete 
checkpoint should be also taken into account.





--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Stephan Ewen]

@hugo For your suggestions, I would ask to start a separate discussion
thread.
I think this mail thread has converged towards merging the initial
suggestion as a starting point and refining it later based on new
discussions.

Best,
Stephan


On Thu, Jun 27, 2019 at 10:48 PM Hugo Louro  wrote:

> +1. Thanks for working on the guide. It's very thorough and a good resource
> to learn good practices from.
>
> I would like use this thread as a placeholder for a couple of topics that
> may be deserving of further discussion on different threads:
>   - What's the best way to keep track of checkstyle version updates. For
> instance, currently there is a PR
>  proposing checkstyle to be
> updated because version 8.12 is no longer supported
>  - When classes import shaded dependencies, it is not trivial for IntelliJ
> to download and associate sources and javadocs, which makes debugging and
> navigate the code base harder. I tried installing the version of the
> library using maven from the CLI, and associate the sources "manually" on
> IntelliJ, but it seems it does not work (perhaps IntelliJ issue). Does
> anyone know of a good solution? If so, we should added here
> <
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/ide_setup.html#intellij-idea
> >.
> I can volunteer for that if you tell me how to do it :)
> - did the community evaluate naming test methods according to these
> conventions  ?
>
> Thanks
>
> On Mon, Jun 24, 2019 at 11:38 AM Stephan Ewen  wrote:
>
> > I think it makes sense to also start individual [DISCUSS] threads about
> > extensions and follow-ups.
> > Various suggestions came up, partly as comments in the doc, some as
> > questions in other threads.
> >
> > Examples:
> >   - use of null in return types versus Optional versus @Nullable/@Nonnull
> >   - initialization of collection sizes
> >   - logging
> >
> > I think these would be best discussed in separate threads each.
> > So, for contributors to whom these issues are dear, feel free to spawn
> > these additional threads.
> > (Bear in mind it is close to 1.9 feature freeze time, so please leave
> this
> > discussions a bit of time so that all community members have a chance to
> > participate)
> >
> >
> >
> > On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen  wrote:
> >
> > > Thanks for the pointer David.
> > >
> > > I was not aware of this tool and I have no experience with its
> practical
> > > impact. For example I would be curious what the effect of this is for
> > > existing PRs, merge conflicts, etc.
> > >
> > > If you want, feel free to start another discuss thread on the
> > introduction
> > > of such a tool.
> > >
> > > On Sun, Jun 23, 2019 at 6:32 PM David Morávek  wrote:
> > >
> > >> I love this kind of unification being pushed forward, the benefit it
> has
> > >> on
> > >> code reviews is enormous. Just my two cents, did you guys think about
> > >> adopting an automatic code formatter for java, the same way as Apache
> > Beam
> > >> did?
> > >>
> > >> It is super easy to use for contributors as they don't need to keep
> any
> > >> particular coding style in mind and they can only focus on
> functionality
> > >> they want to fix, and it's also great for reviewers, because they only
> > see
> > >> the important changes. This also eliminates need for any special
> editor
> > /
> > >> checkstyle configs as the code formatting is part of the build itself.
> > >>
> > >> The one Beam uses is https://github.com/diffplug/spotless with
> > >> GoogleJavaFormat, it may be worth to look into.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen 
> wrote:
> > >>
> > >> > Thanks, everyone, for the positive feedback :-)
> > >> >
> > >> > @Robert - It probably makes sense to break this down into various
> > pages,
> > >> > like PR, general code style guide, Java, component specific guides,
> > >> > formats, etc.
> > >> >
> > >> > Best,
> > >> > Stephan
> > >> >
> > >> >
> > >> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger  >
> > >> > wrote:
> > >> >
> > >> > > It seems that the discussion around this topic has settled.
> > >> > >
> > >> > > I'm going to turn the Google Doc into a markdown file (maybe also
> > >> > multiple,
> > >> > > I'll try out different things) and then open a pull request for
> the
> > >> Flink
> > >> > > website.
> > >> > > I'll post a link to the PR here once I'm done.
> > >> > >
> > >> > > On Fri, Jun 14, 2019 at 9:36 AM zhijiang <
> > wangzhijiang...@aliyun.com
> > >> > > .invalid>
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks for providing this useful guide for benefiting both
> > >> contributors
> > >> > > > and committers in consistency.
> > >> > > >
> > >> > > > I just reviewed and learned the whole doc which covers a lot of
> > >> > > > information. Wish it further categoried and put onto somewhere
> for
> > >> > easily
> > >> > > > traced future.
> > >> > > >
> > >> > > > 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Xuefu Z]

Congratulations, Kurt!

On Tue, Jul 23, 2019 at 7:48 AM Fabian Hueske  wrote:

> Congrats Kurt!
>
> Cheers, Fabian
>
> Am Di., 23. Juli 2019 um 16:42 Uhr schrieb Rong Rong  >:
>
> > Congratulations Kurt!!
> >
> > --
> > Rong
> >
> > On Tue, Jul 23, 2019 at 7:31 AM zhijiang  > .invalid>
> > wrote:
> >
> > > Congrats Kurt!
> > >
> > > Best,
> > > Zhijiang
> > > --
> > > From:Till Rohrmann 
> > > Send Time:2019年7月23日(星期二) 21:08
> > > To:dev 
> > > Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC
> > >
> > > Congrats Kurt!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz <
> dwysakow...@apache.org
> > >
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 23/07/2019 13:39, Hequn Cheng wrote:
> > > > > Congratulations Kurt!
> > > > >
> > > > > Best, Hequn
> > > > >
> > > > > On Tue, Jul 23, 2019 at 7:27 PM vino yang 
> > > wrote:
> > > > >
> > > > >> Congratulations Kurt!
> > > > >>
> > > > >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> > > > >>
> > > > >>> Congratulations Kurt!
> > > > >>>
> > > > >>>
> > > > >>> Best,
> > > > >>>
> > > > >>> Bo WANG
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger <
> > rmetz...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi all,
> > > > 
> > > >  On behalf of the Flink PMC, I'm happy to announce that Kete
> Young
> > is
> > > > >> now
> > > >  part of the Apache Flink Project Management Committee (PMC).
> > > > 
> > > >  Kete has been a committer since February 2017, working a lot on
> > > Table
> > > > >>> API /
> > > >  SQL. He's currently co-managing the 1.9 release! Thanks a lot
> for
> > > your
> > > > >>> work
> > > >  for Flink!
> > > > 
> > > >  Congratulations & Welcome Kurt!
> > > > 
> > > >  Best,
> > > >  Robert
> > > > 
> > > >
> > > >
> > >
> > >
> >
>


-- 
Xuefu Zhang

"In Honey We Trust!"

Re: [DISCUSS] Flink client api enhancement for downstream project

[Stephan Ewen]

Hi all!

This thread has stalled for a bit, which I assume ist mostly due to the
Flink 1.9 feature freeze and release testing effort.

I personally still recognize this issue as one important to be solved. I'd
be happy to help resume this discussion soon (after the 1.9 release) and
see if we can do some step towards this in Flink 1.10.

Best,
Stephan



On Mon, Jun 24, 2019 at 10:41 AM Flavio Pompermaier 
wrote:

> That's exactly what I suggested a long time ago: the Flink REST client
> should not require any Flink dependency, only http library to call the REST
> services to submit and monitor a job.
> What I suggested also in [1] was to have a way to automatically suggest the
> user (via a UI) the available main classes and their required
> parameters[2].
> Another problem we have with Flink is that the Rest client and the CLI one
> behaves differently and we use the CLI client (via ssh) because it allows
> to call some other method after env.execute() [3] (we have to call another
> REST service to signal the end of the job).
> Int his regard, a dedicated interface, like the JobListener suggested in
> the previous emails, would be very helpful (IMHO).
>
> [1] https://issues.apache.org/jira/browse/FLINK-10864
> [2] https://issues.apache.org/jira/browse/FLINK-10862
> [3] https://issues.apache.org/jira/browse/FLINK-10879
>
> Best,
> Flavio
>
> On Mon, Jun 24, 2019 at 9:54 AM Jeff Zhang  wrote:
>
> > Hi, Tison,
> >
> > Thanks for your comments. Overall I agree with you that it is difficult
> for
> > down stream project to integrate with flink and we need to refactor the
> > current flink client api.
> > And I agree that CliFrontend should only parsing command line arguments
> and
> > then pass them to ExecutionEnvironment. It is ExecutionEnvironment's
> > responsibility to compile job, create cluster, and submit job. Besides
> > that, Currently flink has many ExecutionEnvironment implementations, and
> > flink will use the specific one based on the context. IMHO, it is not
> > necessary, ExecutionEnvironment should be able to do the right thing
> based
> > on the FlinkConf it is received. Too many ExecutionEnvironment
> > implementation is another burden for downstream project integration.
> >
> > One thing I'd like to mention is flink's scala shell and sql client,
> > although they are sub-modules of flink, they could be treated as
> downstream
> > project which use flink's client api. Currently you will find it is not
> > easy for them to integrate with flink, they share many duplicated code.
> It
> > is another sign that we should refactor flink client api.
> >
> > I believe it is a large and hard change, and I am afraid we can not keep
> > compatibility since many of changes are user facing.
> >
> >
> >
> > Zili Chen  于2019年6月24日周一 下午2:53写道：
> >
> > > Hi all,
> > >
> > > After a closer look on our client apis, I can see there are two major
> > > issues to consistency and integration, namely different deployment of
> > > job cluster which couples job graph creation and cluster deployment,
> > > and submission via CliFrontend confusing control flow of job graph
> > > compilation and job submission. I'd like to follow the discuss above,
> > > mainly the process described by Jeff and Stephan, and share my
> > > ideas on these issues.
> > >
> > > 1) CliFrontend confuses the control flow of job compilation and
> > submission.
> > > Following the process of job submission Stephan and Jeff described,
> > > execution environment knows all configs of the cluster and
> topos/settings
> > > of the job. Ideally, in the main method of user program, it calls
> > #execute
> > > (or named #submit) and Flink deploys the cluster, compile the job graph
> > > and submit it to the cluster. However, current CliFrontend does all
> these
> > > things inside its #runProgram method, which introduces a lot of
> > subclasses
> > > of (stream) execution environment.
> > >
> > > Actually, it sets up an exec env that hijacks the #execute/executePlan
> > > method, initializes the job graph and abort execution. And then
> > > control flow back to CliFrontend, it deploys the cluster(or retrieve
> > > the client) and submits the job graph. This is quite a specific
> internal
> > > process inside Flink and none of consistency to anything.
> > >
> > > 2) Deployment of job cluster couples job graph creation and cluster
> > > deployment. Abstractly, from user job to a concrete submission, it
> > requires
> > >
> > >  create JobGraph --\
> > >
> > > create ClusterClient -->  submit JobGraph
> > >
> > > such a dependency. ClusterClient was created by deploying or
> retrieving.
> > > JobGraph submission requires a compiled JobGraph and valid
> ClusterClient,
> > > but the creation of ClusterClient is abstractly independent of that of
> > > JobGraph. However, in job cluster mode, we deploy job cluster with a
> job
> > > graph, which means we use another process:
> > >
> > > create JobGraph --> deploy cluster with the JobGraph
> > >
> > > Here 

Re: [DISCUSS] Publish the PyFlink into PyPI

[Stephan Ewen]

Hi!

Sorry for the late involvement. Here are some thoughts from my side:

Definitely +1 to publishing to PyPy, even if it is a binary release.
Community growth into other communities is great, and if this is the
natural way to reach developers in the Python community, let's do it. This
is not about our convenience, but reaching users.

I think the way to look at this is that this is a convenience distribution
channel, courtesy of the Flink community. It is not an Apache release, we
make this clear in the Readme.
Of course, this doesn't mean we don't try to uphold similar standards as
for our official release (like proper license information).

Concerning credentials sharing, I would be fine with whatever option. The
PMC doesn't own it (it is an initiative by some community members), but the
PMC needs to ensure trademark compliance, so slight preference for option
#1 (PMC would have means to correct problems).

I believe there is no need to differentiate between Scala versions, because
this is merely a convenience thing for pure Python users. Users that mix
python and scala (and thus depend on specific scala versions) can still
download from Apache or build themselves.

Best,
Stephan



On Thu, Jul 4, 2019 at 9:51 AM jincheng sun 
wrote:

> Hi All,
>
> Thanks for the feedback @Chesnay Schepler  @Dian!
>
> I think using `apache-flink` for the project name also makes sense to me.
> due to we should always keep in mind that Flink is owned by Apache. (And
> beam also using this pattern `apache-beam` for Python API)
>
> Regarding the Python API release with the JAVA JARs, I think the principle
> of consideration is the convenience of the user. So, Thanks for the
> explanation @Dian!
>
> And your right @Chesnay Schepler   we can't make a
> hasty decision and we need more people's opinions！
>
> So, I appreciate it if anyone can give us feedback and suggestions!
>
> Best,
> Jincheng
>
>
>
>
> Chesnay Schepler  于2019年7月3日周三 下午8:46写道：
>
> > So this would not be a source release then, but a full-blown binary
> > release.
> >
> > Maybe it is just me, but I find it a bit suspect to ship an entire java
> > application via PyPI, just because there's a Python API for it.
> >
> > We definitely need input from more people here.
> >
> > On 03/07/2019 14:09, Dian Fu wrote:
> > > Hi Chesnay,
> > >
> > > Thanks a lot for the suggestions.
> > >
> > > Regarding “distributing java/scala code to PyPI”:
> > > The Python Table API is just a wrapper of the Java Table API and
> without
> > the java/scala code, two steps will be needed to set up an environment to
> > execute a Python Table API program:
> > > 1) Install pyflink using "pip install apache-flink"
> > > 2) Download the flink distribution and set the FLINK_HOME to it.
> > > Besides, users have to make sure that the manually installed Flink is
> > compatible with the pip installed pyflink.
> > >
> > > Bundle the java/scala code inside the Python package will eliminate
> step
> > 2) and makes it more simple for users to install pyflink. There was a
> short
> > discussion  on this in
> > Spark community and they finally decide to package the java/scala code in
> > the python package. (BTW, PySpark only bundle the jars of scala 2.11).
> > >
> > > Regards,
> > > Dian
> > >
> > >> 在 2019年7月3日，下午7:13，Chesnay Schepler  写道：
> > >>
> > >> The existing artifact in the pyflink project was neither released by
> > the Flink project / anyone affiliated with it nor approved by the Flink
> PMC.
> > >>
> > >> As such, if we were to use this account I believe we should delete it
> > to not mislead users that this is in any way an apache-provided
> > distribution. Since this goes against the users wishes, I would be in
> favor
> > of creating a separate account, and giving back control over the pyflink
> > account.
> > >>
> > >> My take on the raised points:
> > >> 1.1) "apache-flink"
> > >> 1.2)  option 2
> > >> 2) Given that we only distribute python code there should be no reason
> > to differentiate between scala versions. We should not be distributing
> any
> > java/scala code and/or modules to PyPi. Currently, I'm a bit confused
> about
> > this question and wonder what exactly we are trying to publish here.
> > >> 3) The should be treated as any other source release; i.e., it needs a
> > LICENSE and NOTICE file, signatures and a PMC vote. My suggestion would
> be
> > to make this part of our normal release process. There will be _one_
> source
> > release on dist.apache.org encompassing everything, and a separate
> python
> > of focused source release that we push to PyPi. The LICENSE and NOTICE
> > contained in the python source release must also be present in the source
> > release of Flink; so basically the python source release is just the
> > contents of flink-python module the maven pom.xml, with no other special
> > sauce added during the release process.
> > >>
> > >> On 02/07/2019 05:42, jincheng sun wrote:
> > >>> Hi all,
> > 

Re: Timestamp(timezone) conversion bug in non blink Table/SQL runtime

[Rong Rong]

Hi Shuyi,

I think there were some discussions in the mailing list [1,2] and JIRA
tickets [3,4] that might be related.
Since the table-blink planner doesn't produce such error, I think this
problem is valid and should be fixed.

Thanks,
Rong

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/event-time-timezone-is-not-correct-tt26457.html
[2]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/TimeZone-shift-problem-in-Flink-SQL-td25666.html#a25739
[3] https://issues.apache.org/jira/browse/FLINK-8353
[4] https://issues.apache.org/jira/browse/FLINK-8169

On Mon, Jul 22, 2019 at 1:49 PM Shuyi Chen  wrote:

> Hi Lasse,
>
> Thanks for the reply. If your input is in epoch time, you are not getting
> local time, instead, you are getting a wrong time that does not make sense.
> For example,  if the user input value is 0 (which means 00:00:00 UTC on 1
> January 1970), and your local timezone is UTC-8, converting 00:00:00 UTC on
> 1 January 1970 to your local timezone should yield 16:00:00 Dec 31, 1969.
> But actually, you will be getting 08:00:00 UTC on 1 January 1970  from
> Table/SQL runtime, which 00:00:00 on 1 January 1970 in your local timezone
> (UTC-8). Your input time just get shifted by 8 hours in output.
>
> Shuyi
>
> On Mon, Jul 22, 2019 at 12:49 PM Lasse Nedergaard <
> lassenederga...@gmail.com> wrote:
>
>> Hi.
>>
>> I have encountered the same problem when you input epoch time to window
>> table function and then use window.start and window.end the out doesn’t
>> output in epoch but local time and I located the problem to the same
>> internal function as you.
>>
>> Med venlig hilsen / Best regards
>> Lasse Nedergaard
>>
>>
>> Den 22. jul. 2019 kl. 20.46 skrev Shuyi Chen :
>>
>> Hi all,
>>
>> Currently, in the non-blink table/SQL runtime, Flink used
>> SqlFunctions.internalToTimestamp(long v) from Calcite to convert event time
>> (in long) to java.sql.Timestamp. However, as discussed in the recent
>> Calcite mailing list (Jul. 19, 2019), SqlFunctions.internalToTimestamp()
>> assumes the input timestamp value is in the current JVM’s default timezone
>> (which is unusual), NOT milliseconds since epoch. And
>> SqlFunctions.internalToTimestamp() is used to convert timestamp value in
>> the current JVM’s default timezone to milliseconds since epoch, which
>> java.sql.Timestamp constructor takes. Therefore, the results will not only
>> be wrong, but change if the job runs in machines on different timezones as
>> well. (The only exception is that all your production machines uses UTC
>> timezone.)
>>
>> Here is an example, if the user input value is 0 (00:00:00 UTC on 1
>> January 1970), and the table/SQL runtime runs in a machine in PST (UTC-8),
>> the output sql.Timestamp after SqlFunctions.internalToTimestamp() will
>> become 2880 millisec since epoch (08:00:00 UTC on 1 January 1970); And
>> with the same input, if the table/SQL runtime runs again in a different
>> machine in EST (UTC-5), the output sql.Timestamp after
>> SqlFunctions.internalToTimestamp() will become 1800 millisec since
>> epoch (05:00:00 UTC on 1 January 1970).
>>
>> More details are captured in
>> https://issues.apache.org/jira/browse/FLINK-13372. Please let me know
>> your thoughts and correct me if I am wrong. Thanks a lot.
>>
>> Shuyi
>>
>>

[jira] [Created] (FLINK-13389) Setting DataStream return type breaks some type conversion between Table and DataStream

[Rong Rong (JIRA)]

Rong Rong created FLINK-13389:
-

 Summary: Setting DataStream return type breaks some type 
conversion between Table and DataStream
 Key: FLINK-13389
 URL: https://issues.apache.org/jira/browse/FLINK-13389
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, Table SQL / API
Reporter: Rong Rong


When converting between data stream and table, there are situations where only 
GenericTypeInfo can be successfully applied, but not directly setting the 
specific RowTypeInfo.
For example the following code doesn't work

{code:java}
TypeInformation[] types = {
BasicTypeInfo.INT_TYPE_INFO,
TimeIndicatorTypeInfo.ROWTIME_INDICATOR(),
BasicTypeInfo.STRING_TYPE_INFO};
String[] names = {"a", "b", "c"};
RowTypeInfo typeInfo = new RowTypeInfo(types, names);
DataStream ds = env.fromCollection(data).returns(typeInfo);
Table sourceTable = tableEnv.fromDataStream(ds, "a,b,c");
tableEnv.registerTable("MyTableRow", sourceTable);

DataStream stream = tableEnv.toAppendStream(sourceTable, 
Row.class)
.map(a -> a)
// this line breaks the conversion, it sets the 
typeinfo to RowTypeInfo.
// without this line the output type is 
GenericTypeInfo(Row)
.returns(sourceTable.getSchema().toRowType());  
stream.addSink(new StreamITCase.StringSink());
env.execute();
{code}




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[zhijiang]

Thanks everyone! 

This is definitely a honor for me. It is very pleasant to work with you guys 
and I would try my best to contribute to the community future.

Best,
Zhijiang
--
From:Dawid Wysakowicz 
Send Time:2019年7月23日(星期二) 21:01
To:dev 
Subject:Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink 
project

Congratulations!

Best,

Dawid

On 23/07/2019 12:03, Danny Chan wrote:
> Congratulations Zhijiang!
>
> Best,
> Danny Chan
> 在 2019年7月22日 +0800 PM10:12，dev@flink.apache.org，写道：
>> Congratulations Zhijiang!

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Fabian Hueske]

Congrats Kurt!

Cheers, Fabian

Am Di., 23. Juli 2019 um 16:42 Uhr schrieb Rong Rong :

> Congratulations Kurt!!
>
> --
> Rong
>
> On Tue, Jul 23, 2019 at 7:31 AM zhijiang  .invalid>
> wrote:
>
> > Congrats Kurt!
> >
> > Best,
> > Zhijiang
> > --
> > From:Till Rohrmann 
> > Send Time:2019年7月23日(星期二) 21:08
> > To:dev 
> > Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC
> >
> > Congrats Kurt!
> >
> > Cheers,
> > Till
> >
> > On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz  >
> > wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 23/07/2019 13:39, Hequn Cheng wrote:
> > > > Congratulations Kurt!
> > > >
> > > > Best, Hequn
> > > >
> > > > On Tue, Jul 23, 2019 at 7:27 PM vino yang 
> > wrote:
> > > >
> > > >> Congratulations Kurt!
> > > >>
> > > >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> > > >>
> > > >>> Congratulations Kurt!
> > > >>>
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Bo WANG
> > > >>>
> > > >>>
> > > >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger <
> rmetz...@apache.org>
> > > >>> wrote:
> > > >>>
> > >  Hi all,
> > > 
> > >  On behalf of the Flink PMC, I'm happy to announce that Kete Young
> is
> > > >> now
> > >  part of the Apache Flink Project Management Committee (PMC).
> > > 
> > >  Kete has been a committer since February 2017, working a lot on
> > Table
> > > >>> API /
> > >  SQL. He's currently co-managing the 1.9 release! Thanks a lot for
> > your
> > > >>> work
> > >  for Flink!
> > > 
> > >  Congratulations & Welcome Kurt!
> > > 
> > >  Best,
> > >  Robert
> > > 
> > >
> > >
> >
> >
>

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Hugo Louro]

+1

> On Jul 23, 2019, at 6:15 AM, Till Rohrmann  wrote:
> 
> Good idea Jark. +1 for the proposal.
> 
> Cheers,
> Till
> 
>> On Tue, Jul 23, 2019 at 1:59 PM Hequn Cheng  wrote:
>> 
>> Hi Jark,
>> 
>> Good idea. +1!
>> 
>>> On Tue, Jul 23, 2019 at 6:23 PM Jark Wu  wrote:
>>> 
>>> Thank you all for your positive feedback.
>>> 
>>> We have three binding +1s, so I think, we can proceed with this.
>>> 
>>> Hi @Robert Metzger  , could you create a request to
>>> INFRA for the mailing list?
>>> I'm not sure if this needs a PMC permission.
>>> 
>>> Thanks,
>>> Jark
>>> 
>>> On Tue, 23 Jul 2019 at 16:42, jincheng sun 
>>> wrote:
>>> 
 +1
 
 Robert Metzger  于2019年7月23日周二 下午4:01写道：
 
> +1
> 
> On Mon, Jul 22, 2019 at 10:27 AM Biao Liu 
>> wrote:
> 
>> +1, make sense to me.
>> Mailing list seems to be a more "community" way.
>> 
>> Timo Walther  于2019年7月22日周一 下午4:06写道：
>> 
>>> +1 sounds good to inform people about instabilities or other
>> issues
>>> 
>>> Regards,
>>> Timo
>>> 
>>> 
 Am 22.07.19 um 09:58 schrieb Haibo Sun:
 +1. Sounds good.Letting the PR creators know the build results
>> of
 the
>>> master branch can help to determine more quickly whether Travis
> failures
>> of
>>> their PR are an exact failure or because of the instability of
>> test
> case.
>>> By the way, if the PR creator can abort their own Travis build, I
 think
>> it
>>> can improve the efficient use of Travis resources (of course,
>> this
>>> discussion does not deal with this issue).
 
 
 Best,
 Haibo
 At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> +1 sounds good, more people could be encouraged to involve in
 paying
>>> attention to failure commit.
> 
> Best
> Yun Tang
> 
> From: Becket Qin 
> Sent: Monday, July 22, 2019 9:44
> To: dev 
> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org
>> mailing
 list
>>> for travis builds
> 
> +1. Sounds a good idea to me.
> 
> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu <
>> dian0511...@gmail.com>
>> wrote:
> 
>> Thanks Jark for the proposal, sounds reasonable for me. +1.
>>> This
 ML
>>> could
>> be used for all the build notifications including master and
>>> CRON
>> jobs.
>> 
>>> 在 2019年7月20日，下午2:55，Xu Forward  写道：
>>> 
>>> +1 ,Thanks jark for the proposal.
>>> Best
>>> Forward
>>> 
>>> Jark Wu  于2019年7月20日周六 下午12:14写道：
>>> 
 Hi all,
 
 As far as I know, currently, email notifications of Travis
 builds
>> for
 master branch are sent to the commit author when a build
>> was
 just
>> broken or
 still is broken. And there is no email notifications for
>> CRON
>> builds.
 
 Recently, we are suffering from compile errors for
>> scala-2.12
 and
>>> java-9
 which are only ran in CRON jobs. So I'm figuring out a way
>> to
 get
 notifications of CRON builds (or all builds) to quick fix
 compile
>>> errors
 and failed cron tests.
 
 After reaching out to @Chesnay Schepler <
>> ches...@apache.org>
>> (thanks
>> for
 the helping), I know that we are using a Slack channel to
 receive
>> all
 failed build notifications. However, IMO, email
>> notification
> might
>>> be a
 better way than Slack channel to encourage more people to
>> pay
>>> attention
>> on
 the builds.
 
 So I'm here to propose to setup a bui...@flink.apache.org
> mailing
>>> list
>> for
 receiving build notifications. I also find that Beam has
>> such
>> mailing
>> list
 too[1]. After we have such a mailing list, we can integrate
>>> it
 to
>>> travis
 according to the travis doc[2].
 
 What do you think? Do we need a formal vote for this?
 
 Best and thanks,
 Jark
 
 [1]: https://beam.apache.org/community/contact-us/
 [2]:
 
 
>> 
>>> 
>> 
> 
 
>>> 
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
 <
 
>> 
>>> 
>> 
> 
 
>>> 
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
 <
 
>> 
>>> 
>> 
> 
 
>>> 
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
>> 
>>> 
>>> 
>> 
> 
 
>>> 
>> 

[jira] [Created] (FLINK-13388) Update UI screenshots in the documentation to the new default UI

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-13388:


 Summary: Update UI screenshots in the documentation to the new 
default UI
 Key: FLINK-13388
 URL: https://issues.apache.org/jira/browse/FLINK-13388
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Rong Rong]

Congratulations Kurt!!

--
Rong

On Tue, Jul 23, 2019 at 7:31 AM zhijiang 
wrote:

> Congrats Kurt!
>
> Best,
> Zhijiang
> --
> From:Till Rohrmann 
> Send Time:2019年7月23日(星期二) 21:08
> To:dev 
> Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC
>
> Congrats Kurt!
>
> Cheers,
> Till
>
> On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz 
> wrote:
>
> > Congratulations!
> >
> > Best,
> >
> > Dawid
> >
> > On 23/07/2019 13:39, Hequn Cheng wrote:
> > > Congratulations Kurt!
> > >
> > > Best, Hequn
> > >
> > > On Tue, Jul 23, 2019 at 7:27 PM vino yang 
> wrote:
> > >
> > >> Congratulations Kurt!
> > >>
> > >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> > >>
> > >>> Congratulations Kurt!
> > >>>
> > >>>
> > >>> Best,
> > >>>
> > >>> Bo WANG
> > >>>
> > >>>
> > >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> > >>> wrote:
> > >>>
> >  Hi all,
> > 
> >  On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> > >> now
> >  part of the Apache Flink Project Management Committee (PMC).
> > 
> >  Kete has been a committer since February 2017, working a lot on
> Table
> > >>> API /
> >  SQL. He's currently co-managing the 1.9 release! Thanks a lot for
> your
> > >>> work
> >  for Flink!
> > 
> >  Congratulations & Welcome Kurt!
> > 
> >  Best,
> >  Robert
> > 
> >
> >
>
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[zhijiang]

Congrats Kurt!

Best,
Zhijiang
--
From:Till Rohrmann 
Send Time:2019年7月23日(星期二) 21:08
To:dev 
Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

Congrats Kurt!

Cheers,
Till

On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz 
wrote:

> Congratulations!
>
> Best,
>
> Dawid
>
> On 23/07/2019 13:39, Hequn Cheng wrote:
> > Congratulations Kurt!
> >
> > Best, Hequn
> >
> > On Tue, Jul 23, 2019 at 7:27 PM vino yang  wrote:
> >
> >> Congratulations Kurt!
> >>
> >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> >>
> >>> Congratulations Kurt!
> >>>
> >>>
> >>> Best,
> >>>
> >>> Bo WANG
> >>>
> >>>
> >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> >>> wrote:
> >>>
>  Hi all,
> 
>  On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> >> now
>  part of the Apache Flink Project Management Committee (PMC).
> 
>  Kete has been a committer since February 2017, working a lot on Table
> >>> API /
>  SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> >>> work
>  for Flink!
> 
>  Congratulations & Welcome Kurt!
> 
>  Best,
>  Robert
> 
>
>

[jira] [Created] (FLINK-13387) Can not download taskmanger & jobmanager's logs in the old UI

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-13387:


 Summary: Can not download taskmanger & jobmanager's logs in the 
old UI
 Key: FLINK-13387
 URL: https://issues.apache.org/jira/browse/FLINK-13387
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Dawid Wysakowicz
 Fix For: 1.9.0


It is not possible to download the taskmanager & jobmanager logs via the old UI.

The exception is: "Unable to load requested file 
/old-version/taskmanagers/1234ddfcc0b1d06d2615d5431a08c7b8/stdout."



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13386) Friction in the new default API

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-13386:


 Summary: Friction in the new default API
 Key: FLINK-13386
 URL: https://issues.apache.org/jira/browse/FLINK-13386
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Dawid Wysakowicz
 Attachments: bug.png

While manually testing the new WebUI I found a few frictions.

* when using the UI the left panel hides unexpectedly at random moments
* mouse wheel does not work on the logs (taskmanager, jobmanager) pane
* the jobmanager configuration is not sorted
* different sorting of the operators (the old UI showed the sources first)
* the drop-down list for choosing operator/tasks metrics is not sorted, which 
makes it super hard to screen through available metrics
* arrow does not touch the rectangles in Chrome (see attached screenshot)

There are also some views missing in the new UI that I personally found useful 
in the old UI:
* can't see watermarks for all operators at once
* no numeric metrics (only graphs)




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13385) Align Hive data type mapping with FLIP-37

[Timo Walther (JIRA)]

Timo Walther created FLINK-13385:


 Summary: Align Hive data type mapping with FLIP-37
 Key: FLINK-13385
 URL: https://issues.apache.org/jira/browse/FLINK-13385
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Timo Walther


By looking at the Hive data type mapping of:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/catalog.html#data-type-mapping

Based on the information available in:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types

It seems that the type are not mapped correctly. The following changes should 
be performed (indicated by {{>>...<<}}):

{code}
CHAR(p) char(p)*
VARCHAR(p)  varchar(p)**
STRING  string
BOOLEAN boolean
>>TINYINT<< tinyint
>>SMALLINT<>INTERVAL?<<
BINARY  >>N/A<<
VARBINARY(p)>>N/A<<
>>BYTES BINARY<<
>>ARRAY  ARRAY<<
>>MAP MAP* we support more than primitives<<
ROW struct
MULTISETN/A
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: instable checkpointing after migration to flink 1.8

[Fabian Hueske]

Hi Bekir,

Another user reported checkpointing issues with Flink 1.8.0 [1].
These seem to be resolved with Flink 1.8.1.

Hope this helps,
Fabian

[1]
https://lists.apache.org/thread.html/991fe3b09fd6a052ff52e5f7d9cdd9418545e68b02e23493097d9bc4@%3Cuser.flink.apache.org%3E

Am Mi., 17. Juli 2019 um 09:16 Uhr schrieb Congxian Qiu <
qcx978132...@gmail.com>:

> Hi Bekir
>
> First of all, I think there is something wrong.  the state size is almost
> the same,  but the duration is different so much.
>
> The checkpoint for RocksDBStatebackend is dump sst files, then copy the
> needed sst files(if you enable incremental checkpoint, the sst files
> already on remote will not upload), then complete checkpoint. Can you check
> the network bandwidth usage during checkpoint?
>
> Best,
> Congxian
>
>
> Bekir Oguz  于2019年7月16日周二 下午10:45写道：
>
>> Hi all,
>> We have a flink job with user state, checkpointing to RocksDBBackend
>> which is externally stored in AWS S3.
>> After we have migrated our cluster from 1.6 to 1.8, we see occasionally
>> that some slots do to acknowledge the checkpoints quick enough. As an
>> example: All slots acknowledge between 30-50 seconds except only one slot
>> acknowledges in 15 mins. Checkpoint sizes are similar to each other, like
>> 200-400 MB.
>>
>> We did not experience this weird behaviour in Flink 1.6. We have 5 min
>> checkpoint interval and this happens sometimes once in an hour sometimes
>> more but not in all the checkpoint requests. Please see the screenshot
>> below.
>>
>> Also another point: For the faulty slots, the duration is consistently 15
>> mins and some seconds, we couldn’t find out where this 15 mins response
>> time comes from. And each time it is a different task manager, not always
>> the same one.
>>
>> Do you guys aware of any other users having similar issues with the new
>> version and also a suggested bug fix or solution?
>>
>>
>>
>>
>> Thanks in advance,
>> Bekir Oguz
>>
>

Re: [Requirement] CI report

[Andrey Zagrebin]

Hi,

@Tison I guess it was done because there were many associated comments
notified by everyone involved in PR.

I agree that it would be actually useful for a PR's author to get
notifications about failed/succeeded CI builds, not sure how easy it is
though.

Best,
Andrey


On Tue, Jul 23, 2019 at 10:15 AM Zili Chen  wrote:

> Hi,
>
> Currently, our flinkbot updates CI report on status changing.
>
> However, it updates via editing GitHub comment, which would not send
> a notification to pr creator once status updated.
>
> Said the "PENDING" status is not quite useful, is it possible that
> flinkbot updates a final status(FAILURE/SUCCESS) by adding a new
> comment? This will be like hadoop bot updates on JIRA.
>
>
> Best,
> tison.
>

?????? [DISSCUSS] Tolerate temporarily suspended ZooKeeper connections

[????????]

Ok, If you have any suggestions, we can talk aobut the details under 
FLINK-10052.


Best.


--  --
??: "Till Rohrmann";
: 2019??7??23??(??) 9:19
??: "dev";

: Re: [DISSCUSS] Tolerate temporarily suspended ZooKeeper connections



Hi Lamber-Ken,

thanks for starting this discussion. I think there is benefit of not
directly losing leadership if the ZooKeeper connection goes into the
SUSPENDED state. In particular if we can guarantee that there is only a
single JobMaster, it might make sense to not overly eagerly give up
leadership. I would suggest to continue the technical discussion on the
JIRA issue thread since it already contains a good amount of details.

Cheers,
Till

On Sat, Jul 20, 2019 at 12:55 PM QQ <2217232...@qq.com> wrote:

> Hi All,
>
> Desc
> We deploy flink streaming jobs on hadoop cluster on per-job model and use
> zookeeper as HighAvailabilityService, but we found that flink job will
> restart because of the network disconnected temporarily between jobmanager
> and zookeeper.So we analyze this problem deeply. Flink JobManager use
> curator's `LeaderLatch` to maintain the leadership. When network
> disconncet, the `LeaderLatch` will change leadership to false directly. We
> think it's too brutally that many flink longrunning jobs will restart
> because of the network shake.Instead of directly revoking the leadership
> upon a SUSPENDED ZooKeeper connection, it would be better to wait until the
> ZooKeeper connection is LOST.
>
> Here're two jiras about the problem, FLINK-10052 and FLINK-13189, they are
> duplicate. Thanks to @Elias Levy told us that FLINK-13189, so close
> FLINK-13189.
>
> Solution
> Back to this problem, there're two ways to solve this currently, one is
> rewrite LeaderLatch#handleStateChange method, another is upgrade
> curator-4.2.0. The first way is hackly but right, the second way need to
> consider the
> compatibility. For more detail, please see FLINK-10052.
>
> Hope
> The FLINK-10052 was reported at 2018-08-03(about a year ago), so we hope
> this problem can fix as soon as possible.
> btw, thanks @TisonKun for talking about this problem and review pr.
>
> Links
> FLINK-10052 https://issues.apache.org/jira/browse/FLINK-10052 <
> https://issues.apache.org/jira/browse/FLINK-10052>
> FLINK-13189 https://issues.apache.org/jira/browse/FLINK-13189 <
> https://issues.apache.org/jira/browse/FLINK-13189>
>
> Any suggestion is welcome, what do you think?
>
> Best, lamber-ken.

Re: [DISSCUSS] Tolerate temporarily suspended ZooKeeper connections

[Till Rohrmann]

Hi Lamber-Ken,

thanks for starting this discussion. I think there is benefit of not
directly losing leadership if the ZooKeeper connection goes into the
SUSPENDED state. In particular if we can guarantee that there is only a
single JobMaster, it might make sense to not overly eagerly give up
leadership. I would suggest to continue the technical discussion on the
JIRA issue thread since it already contains a good amount of details.

Cheers,
Till

On Sat, Jul 20, 2019 at 12:55 PM QQ邮箱 <2217232...@qq.com> wrote:

> Hi All,
>
> Desc
> We deploy flink streaming jobs on hadoop cluster on per-job model and use
> zookeeper as HighAvailabilityService, but we found that flink job will
> restart because of the network disconnected temporarily between jobmanager
> and zookeeper.So we analyze this problem deeply. Flink JobManager use
> curator's `LeaderLatch` to maintain the leadership. When network
> disconncet, the `LeaderLatch` will change leadership to false directly. We
> think it's too brutally that many flink longrunning jobs will restart
> because of the network shake.Instead of directly revoking the leadership
> upon a SUSPENDED ZooKeeper connection, it would be better to wait until the
> ZooKeeper connection is LOST.
>
> Here're two jiras about the problem, FLINK-10052 and FLINK-13189, they are
> duplicate. Thanks to @Elias Levy told us that FLINK-13189, so close
> FLINK-13189.
>
> Solution
> Back to this problem, there're two ways to solve this currently, one is
> rewrite LeaderLatch#handleStateChange method, another is upgrade
> curator-4.2.0. The first way is hackly but right, the second way need to
> consider the
> compatibility. For more detail, please see FLINK-10052.
>
> Hope
> The FLINK-10052 was reported at 2018-08-03(about a year ago), so we hope
> this problem can fix as soon as possible.
> btw, thanks @TisonKun for talking about this problem and review pr.
>
> Links
> FLINK-10052 https://issues.apache.org/jira/browse/FLINK-10052 <
> https://issues.apache.org/jira/browse/FLINK-10052>
> FLINK-13189 https://issues.apache.org/jira/browse/FLINK-13189 <
> https://issues.apache.org/jira/browse/FLINK-13189>
>
> Any suggestion is welcome, what do you think?
>
> Best, lamber-ken.

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Till Rohrmann]

Good idea Jark. +1 for the proposal.

Cheers,
Till

On Tue, Jul 23, 2019 at 1:59 PM Hequn Cheng  wrote:

> Hi Jark,
>
> Good idea. +1!
>
> On Tue, Jul 23, 2019 at 6:23 PM Jark Wu  wrote:
>
> > Thank you all for your positive feedback.
> >
> > We have three binding +1s, so I think, we can proceed with this.
> >
> > Hi @Robert Metzger  , could you create a request to
> > INFRA for the mailing list?
> > I'm not sure if this needs a PMC permission.
> >
> > Thanks,
> > Jark
> >
> > On Tue, 23 Jul 2019 at 16:42, jincheng sun 
> > wrote:
> >
> > > +1
> > >
> > > Robert Metzger  于2019年7月23日周二 下午4:01写道：
> > >
> > > > +1
> > > >
> > > > On Mon, Jul 22, 2019 at 10:27 AM Biao Liu 
> wrote:
> > > >
> > > > > +1, make sense to me.
> > > > > Mailing list seems to be a more "community" way.
> > > > >
> > > > > Timo Walther  于2019年7月22日周一 下午4:06写道：
> > > > >
> > > > > > +1 sounds good to inform people about instabilities or other
> issues
> > > > > >
> > > > > > Regards,
> > > > > > Timo
> > > > > >
> > > > > >
> > > > > > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > > > > > +1. Sounds good.Letting the PR creators know the build results
> of
> > > the
> > > > > > master branch can help to determine more quickly whether Travis
> > > > failures
> > > > > of
> > > > > > their PR are an exact failure or because of the instability of
> test
> > > > case.
> > > > > > By the way, if the PR creator can abort their own Travis build, I
> > > think
> > > > > it
> > > > > > can improve the efficient use of Travis resources (of course,
> this
> > > > > > discussion does not deal with this issue).
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Haibo
> > > > > > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > > > > > >> +1 sounds good, more people could be encouraged to involve in
> > > paying
> > > > > > attention to failure commit.
> > > > > > >>
> > > > > > >> Best
> > > > > > >> Yun Tang
> > > > > > >> 
> > > > > > >> From: Becket Qin 
> > > > > > >> Sent: Monday, July 22, 2019 9:44
> > > > > > >> To: dev 
> > > > > > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org
> mailing
> > > list
> > > > > > for travis builds
> > > > > > >>
> > > > > > >> +1. Sounds a good idea to me.
> > > > > > >>
> > > > > > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu <
> dian0511...@gmail.com>
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Thanks Jark for the proposal, sounds reasonable for me. +1.
> > This
> > > ML
> > > > > > could
> > > > > > >>> be used for all the build notifications including master and
> > CRON
> > > > > jobs.
> > > > > > >>>
> > > > > >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > > > > > 
> > > > > >  +1 ,Thanks jark for the proposal.
> > > > > >  Best
> > > > > >  Forward
> > > > > > 
> > > > > >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > > > > > 
> > > > > > > Hi all,
> > > > > > >
> > > > > > > As far as I know, currently, email notifications of Travis
> > > builds
> > > > > for
> > > > > > > master branch are sent to the commit author when a build
> was
> > > just
> > > > > > >>> broken or
> > > > > > > still is broken. And there is no email notifications for
> CRON
> > > > > builds.
> > > > > > >
> > > > > > > Recently, we are suffering from compile errors for
> scala-2.12
> > > and
> > > > > > java-9
> > > > > > > which are only ran in CRON jobs. So I'm figuring out a way
> to
> > > get
> > > > > > > notifications of CRON builds (or all builds) to quick fix
> > > compile
> > > > > > errors
> > > > > > > and failed cron tests.
> > > > > > >
> > > > > > > After reaching out to @Chesnay Schepler <
> ches...@apache.org>
> > > > > (thanks
> > > > > > >>> for
> > > > > > > the helping), I know that we are using a Slack channel to
> > > receive
> > > > > all
> > > > > > > failed build notifications. However, IMO, email
> notification
> > > > might
> > > > > > be a
> > > > > > > better way than Slack channel to encourage more people to
> pay
> > > > > > attention
> > > > > > >>> on
> > > > > > > the builds.
> > > > > > >
> > > > > > > So I'm here to propose to setup a bui...@flink.apache.org
> > > > mailing
> > > > > > list
> > > > > > >>> for
> > > > > > > receiving build notifications. I also find that Beam has
> such
> > > > > mailing
> > > > > > >>> list
> > > > > > > too[1]. After we have such a mailing list, we can integrate
> > it
> > > to
> > > > > > travis
> > > > > > > according to the travis doc[2].
> > > > > > >
> > > > > > > What do you think? Do we need a formal vote for this?
> > > > > > >
> > > > > > > Best and thanks,
> > > > > > > Jark
> > > > > > >
> > > > > > > [1]: https://beam.apache.org/community/contact-us/
> > > > > > > [2]:
> > > > > > >
> > > > > > >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > > > 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Till Rohrmann]

Congrats Kurt!

Cheers,
Till

On Tue, Jul 23, 2019 at 3:00 PM Dawid Wysakowicz 
wrote:

> Congratulations!
>
> Best,
>
> Dawid
>
> On 23/07/2019 13:39, Hequn Cheng wrote:
> > Congratulations Kurt!
> >
> > Best, Hequn
> >
> > On Tue, Jul 23, 2019 at 7:27 PM vino yang  wrote:
> >
> >> Congratulations Kurt!
> >>
> >> Bo WANG  于2019年7月23日周二 下午7:13写道：
> >>
> >>> Congratulations Kurt!
> >>>
> >>>
> >>> Best,
> >>>
> >>> Bo WANG
> >>>
> >>>
> >>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> >>> wrote:
> >>>
>  Hi all,
> 
>  On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> >> now
>  part of the Apache Flink Project Management Committee (PMC).
> 
>  Kete has been a committer since February 2017, working a lot on Table
> >>> API /
>  SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> >>> work
>  for Flink!
> 
>  Congratulations & Welcome Kurt!
> 
>  Best,
>  Robert
> 
>
>

Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Dawid Wysakowicz]

Congratulations!

Best,

Dawid

On 23/07/2019 12:03, Danny Chan wrote:
> Congratulations Zhijiang!
>
> Best,
> Danny Chan
> 在 2019年7月22日 +0800 PM10:12，dev@flink.apache.org，写道：
>> Congratulations Zhijiang!


signature.asc
Description: OpenPGP digital signature

Re: Looking for reviewer: FLINK-13127

[Till Rohrmann]

I've assigned the issue to you David. I think this feature makes sense. The
only question I have is why we need to sort the values. But let's discuss
the issue on the Github PR.

@Tison and @Xintong, let me know once the PR is in mergeable state.

Cheers,
Till

On Tue, Jul 23, 2019 at 4:14 AM Xintong Song  wrote:

> David,
>
> Thank you for opening this PR. I also left a few comments.
>
> And I think we need a committer to assign this jira ticket to David. Maybe
> Till or any other committer could look into this?
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Jul 22, 2019 at 8:37 PM Zili Chen  wrote:
>
> > Hi David,
> >
> > Just reviewed and left several comments. Thanks for your contribution!
> >
> > Best,
> > tison.
> >
> >
> > David Morávek  于2019年7月22日周一 下午5:59写道：
> >
> > > Hi, I've prepared a small patch related to Yarn deployment. Would
> anyone
> > > please have a time to take a look at it? Thanks
> > >
> > > https://github.com/apache/flink/pull/9022
> > >
> > > Regards,
> > > D.
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Dawid Wysakowicz]

Congratulations!

Best,

Dawid

On 23/07/2019 13:39, Hequn Cheng wrote:
> Congratulations Kurt!
>
> Best, Hequn
>
> On Tue, Jul 23, 2019 at 7:27 PM vino yang  wrote:
>
>> Congratulations Kurt!
>>
>> Bo WANG  于2019年7月23日周二 下午7:13写道：
>>
>>> Congratulations Kurt!
>>>
>>>
>>> Best,
>>>
>>> Bo WANG
>>>
>>>
>>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
>>> wrote:
>>>
 Hi all,

 On behalf of the Flink PMC, I'm happy to announce that Kete Young is
>> now
 part of the Apache Flink Project Management Committee (PMC).

 Kete has been a committer since February 2017, working a lot on Table
>>> API /
 SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
>>> work
 for Flink!

 Congratulations & Welcome Kurt!

 Best,
 Robert




signature.asc
Description: OpenPGP digital signature

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Andrey Zagrebin]

Congratulations Zhijiang! Good work!

Best,
Andrey

On Tue, Jul 23, 2019 at 1:40 PM Hequn Cheng  wrote:

> Congratulations Zhijiang!
>
> Best, Hequn
>
> On Tue, Jul 23, 2019 at 6:17 PM JingsongLee  .invalid>
> wrote:
>
> > Congratulations Zhijiang!
> >
> > Best, Jingsong Lee
> >
> >
> > --
> > From:wenlong.lwl 
> > Send Time:2019年7月23日(星期二) 17:34
> > To:dev 
> > Subject:Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to
> > the Flink project
> >
> > Congrats, Zhijiang!
> >
> > On Tue, 23 Jul 2019 at 15:59, Becket Qin  wrote:
> >
> > > Congrats, Zhijiang!
> > >
> > > On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:
> > >
> > > > Congratulations Zhijiang!
> > > >
> > > >
> > > > On Tue, 23 Jul 2019 at 11:30, vino yang 
> wrote:
> > > >
> > > > > Congratulations Zhijiang!
> > > > >
> > > > > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> > > > >
> > > > > > Congrats, Zhejiang!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Haibo
> > > > > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > > > > >Congratulations Zhijiang, well deserved.
> > > > > > >
> > > > > > >Best
> > > > > > >
> > > > > > >From: Yingjie Cao 
> > > > > > >Sent: Tuesday, July 23, 2019 10:23
> > > > > > >To: dev@flink.apache.org 
> > > > > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a
> > committer
> > > to
> > > > > > the Flink project
> > > > > > >
> > > > > > >Congratulations Zhijiang!
> > > > > > >
> > > > > > >yangtao.yt  于2019年7月23日周二
> 上午10:17写道：
> > > > > > >
> > > > > > >> Congrats, Zhejiang!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Tao Yang
> > > > > > >>
> > > > > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > > > > >> >
> > > > > > >> > Congratulations Zhijiang
> > > > > > >> >
> > > > > > >> > 发自我的 iPhone
> > > > > > >> >
> > > > > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > > > > >> >>
> > > > > > >> >> Congratulations, Zhijiang!
> > > > > > >> >>
> > > > > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> > > > wbeaglewatc...@gmail.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >> >>>
> > > > > > >> >>> Congratulations Zhijiang!
> > > > > > >> >>>
> > > > > > >> >>>
> > > > > > >> >>> Best,
> > > > > > >> >>>
> > > > > > >> >>> Bo WANG
> > > > > > >> >>>
> > > > > > >> >>>
> > > > > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > > > > rmetz...@apache.org>
> > > > > > >> >>> wrote:
> > > > > > >> >>>
> > > > > > >>  Hey all,
> > > > > > >> 
> > > > > > >>  We've added another committer to the Flink project:
> > Zhijiang
> > > > > Wang.
> > > > > > >> 
> > > > > > >>  Congratulations Zhijiang!
> > > > > > >> 
> > > > > > >>  Best,
> > > > > > >>  Robert
> > > > > > >>  (on behalf of the Flink PMC)
> > > > > > >> 
> > > > > > >> >>>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Xuefu Zhang
> > > > > > >> >>
> > > > > > >> >> "In Honey We Trust!"
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Support temporary tables in SQL API

[Dawid Wysakowicz]

I think we all agree so far that we should implement one of the short
term solutions for 1.9 release (#2 or #3) and continue the discussion on
option #1 for the next release. Personally I prefer option #2, because
it is closest to the current behavior and as Kurt said it is the most
intuitive one, but I am also fine with option #3

To sum up the opinions so far:

/option #2: 3 votes(Kurt, Aljoscha, me)/

/option #3: 2 votes(Timo, Jingsong)/

I wasn't sure which option out of the two Xuefu prefers.

I would like to conclude the discussion by the end of tomorrow, so that
we can prepare a proper fix as soon as possible. Therefore I would
suggest to proceed with the option that gets the most votes until
tomorrow (*July 24th 12:00 CET*), unless there are some hard objections.


Comment on option #1 concerns:

I agree with Jingsong on that. I think there are some benefits of the
approach, as it makes Flink in control of the temporary tables.

1. We have a unified behavior across all catalogs. Also for the catalogs
that do not support temporary tables natively.

2. As Flink is in control of the temporary tables it makes it easier to
control their lifecycle.

Best,

Dawid

On 23/07/2019 11:40, JingsongLee wrote:
> And I think we should recommend user to use catalog api to
>  createTable and createFunction,(I guess most scenarios do
>  not use temporary objects) in this way, it is good to option #3
>
> Best, JingsongLee
>
>
> --
> From:JingsongLee 
> Send Time:2019年7月23日(星期二) 17:35
> To:dev 
> Subject:Re: [DISCUSS] Support temporary tables in SQL API
>
> Thanks Dawid and other people.
> +1 for using option #3 for 1.9.0 and go with option #1
>  in 1.10.0.
>
> Regarding Xuefu's concern, I don't know how necessary it is for each catalog 
> to
>  deal with tmpView. I think Catalog is different from DB, we can have single 
> concept for tmpView, that make user easier to understand.
>
> Regarding option #2, It is hard to use if we let user to use fully qualified 
> name for hive catalog. Would this experience be too bad to use?
>
> Best, Jingsong Lee
>
>
> --
> From:Kurt Young 
> Send Time:2019年7月23日(星期二) 17:03
> To:dev 
> Subject:Re: [DISCUSS] Support temporary tables in SQL API
>
> Thanks Dawid for driving this discussion.
> Personally, I would +1 for using option #2 for 1.9.0 and go with option #1
> in 1.10.0.
>
> Regarding Xuefu's concern about option #1, I think we could also try to
> reuse the in-memory catalog
> for the builtin temporary table storage.
>
> Regarding to option #2 and option #3, from user's perspective, IIUC option
> #2 allows user to have
> simple name to reference temporary table and should use fully qualified
> name for external catalogs.
> But option #3 provide the opposite behavior, user can use simple name for
> external tables after he
> changed current catalog and current database, but have to use fully
> qualified name for temporary
> tables. IMO, option #2 will be more straightforward.
>
> Best,
> Kurt
>
>
> On Tue, Jul 23, 2019 at 4:01 PM Aljoscha Krettek 
> wrote:
>
>> I would be fine with option 3) but I think option 2) is the more implicit
>> solution that has less surprising behaviour.
>>
>> Aljoscha
>>
>>> On 22. Jul 2019, at 23:59, Xuefu Zhang  wrote:
>>>
>>> Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
>>> that for 1.9 we should have some quick and simple solution, leaving time
>>> for more thorough discussions for 1.10.
>>>
>>> In particular, I'm not fully with solution #1. For one thing, it seems
>>> proposing storing all temporary objects in a memory map in
>> CatalogManager,
>>> and the memory map duplicates the functionality of the in-memory catalog,
>>> which also store temporary objects. For another, as pointed out by the
>>> google doc, different db may handle the temporary tables differently, and
>>> accordingly it may make more sense to let each catalog to handle its
>>> temporary objects.
>>>
>>> Therefore, postponing the fix buys us time to flush out all the details.
>>>
>>> Thanks,
>>> Xuefu
>>>
>>> On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  wrote:
>>>
 Thanks for summarizing our offline discussion Dawid! Even though I would
 prefer solution 1 instead of releasing half-baked features, I also
 understand that the Table API should not further block the next release.
 Therefore, I would be fine with solution 3 but introduce the new
 user-facing `createTemporaryTable` methods as synonyms of the existing
 ones already. This allows us to deprecate the methods with undefined
 behavior as early as possible.

 Thanks,
 Timo


 Am 22.07.19 um 16:13 schrieb Dawid Wysakowicz:
> Hi all,
>
> When working on FLINK-13279[1] we realized we could benefit from a
> better temporary objects support in the Catalog API/Table API.
> Unfortunately we are already 

[jira] [Created] (FLINK-13384) The backpressure monitoring tab in WebUI does not show correct results

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-13384:


 Summary: The backpressure monitoring tab in WebUI does not show 
correct results
 Key: FLINK-13384
 URL: https://issues.apache.org/jira/browse/FLINK-13384
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics, Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Hequn Cheng]

Hi Jark,

Good idea. +1!

On Tue, Jul 23, 2019 at 6:23 PM Jark Wu  wrote:

> Thank you all for your positive feedback.
>
> We have three binding +1s, so I think, we can proceed with this.
>
> Hi @Robert Metzger  , could you create a request to
> INFRA for the mailing list?
> I'm not sure if this needs a PMC permission.
>
> Thanks,
> Jark
>
> On Tue, 23 Jul 2019 at 16:42, jincheng sun 
> wrote:
>
> > +1
> >
> > Robert Metzger  于2019年7月23日周二 下午4:01写道：
> >
> > > +1
> > >
> > > On Mon, Jul 22, 2019 at 10:27 AM Biao Liu  wrote:
> > >
> > > > +1, make sense to me.
> > > > Mailing list seems to be a more "community" way.
> > > >
> > > > Timo Walther  于2019年7月22日周一 下午4:06写道：
> > > >
> > > > > +1 sounds good to inform people about instabilities or other issues
> > > > >
> > > > > Regards,
> > > > > Timo
> > > > >
> > > > >
> > > > > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > > > > +1. Sounds good.Letting the PR creators know the build results of
> > the
> > > > > master branch can help to determine more quickly whether Travis
> > > failures
> > > > of
> > > > > their PR are an exact failure or because of the instability of test
> > > case.
> > > > > By the way, if the PR creator can abort their own Travis build, I
> > think
> > > > it
> > > > > can improve the efficient use of Travis resources (of course, this
> > > > > discussion does not deal with this issue).
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Haibo
> > > > > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > > > > >> +1 sounds good, more people could be encouraged to involve in
> > paying
> > > > > attention to failure commit.
> > > > > >>
> > > > > >> Best
> > > > > >> Yun Tang
> > > > > >> 
> > > > > >> From: Becket Qin 
> > > > > >> Sent: Monday, July 22, 2019 9:44
> > > > > >> To: dev 
> > > > > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing
> > list
> > > > > for travis builds
> > > > > >>
> > > > > >> +1. Sounds a good idea to me.
> > > > > >>
> > > > > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu 
> > > > wrote:
> > > > > >>
> > > > > >>> Thanks Jark for the proposal, sounds reasonable for me. +1.
> This
> > ML
> > > > > could
> > > > > >>> be used for all the build notifications including master and
> CRON
> > > > jobs.
> > > > > >>>
> > > > >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > > > > 
> > > > >  +1 ,Thanks jark for the proposal.
> > > > >  Best
> > > > >  Forward
> > > > > 
> > > > >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > > > > 
> > > > > > Hi all,
> > > > > >
> > > > > > As far as I know, currently, email notifications of Travis
> > builds
> > > > for
> > > > > > master branch are sent to the commit author when a build was
> > just
> > > > > >>> broken or
> > > > > > still is broken. And there is no email notifications for CRON
> > > > builds.
> > > > > >
> > > > > > Recently, we are suffering from compile errors for scala-2.12
> > and
> > > > > java-9
> > > > > > which are only ran in CRON jobs. So I'm figuring out a way to
> > get
> > > > > > notifications of CRON builds (or all builds) to quick fix
> > compile
> > > > > errors
> > > > > > and failed cron tests.
> > > > > >
> > > > > > After reaching out to @Chesnay Schepler 
> > > > (thanks
> > > > > >>> for
> > > > > > the helping), I know that we are using a Slack channel to
> > receive
> > > > all
> > > > > > failed build notifications. However, IMO, email notification
> > > might
> > > > > be a
> > > > > > better way than Slack channel to encourage more people to pay
> > > > > attention
> > > > > >>> on
> > > > > > the builds.
> > > > > >
> > > > > > So I'm here to propose to setup a bui...@flink.apache.org
> > > mailing
> > > > > list
> > > > > >>> for
> > > > > > receiving build notifications. I also find that Beam has such
> > > > mailing
> > > > > >>> list
> > > > > > too[1]. After we have such a mailing list, we can integrate
> it
> > to
> > > > > travis
> > > > > > according to the travis doc[2].
> > > > > >
> > > > > > What do you think? Do we need a formal vote for this?
> > > > > >
> > > > > > Best and thanks,
> > > > > > Jark
> > > > > >
> > > > > > [1]: https://beam.apache.org/community/contact-us/
> > > > > > [2]:
> > > > > >
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > > <
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > > <
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > >>>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Hequn Cheng]

Congratulations Kurt!

Best, Hequn

On Tue, Jul 23, 2019 at 7:27 PM vino yang  wrote:

> Congratulations Kurt!
>
> Bo WANG  于2019年7月23日周二 下午7:13写道：
>
> > Congratulations Kurt!
> >
> >
> > Best,
> >
> > Bo WANG
> >
> >
> > On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> > wrote:
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Kete has been a committer since February 2017, working a lot on Table
> > API /
> > > SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> > work
> > > for Flink!
> > >
> > > Congratulations & Welcome Kurt!
> > >
> > > Best,
> > > Robert
> > >
> >
>

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Hequn Cheng]

Congratulations Zhijiang!

Best, Hequn

On Tue, Jul 23, 2019 at 6:17 PM JingsongLee 
wrote:

> Congratulations Zhijiang!
>
> Best, Jingsong Lee
>
>
> --
> From:wenlong.lwl 
> Send Time:2019年7月23日(星期二) 17:34
> To:dev 
> Subject:Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to
> the Flink project
>
> Congrats, Zhijiang!
>
> On Tue, 23 Jul 2019 at 15:59, Becket Qin  wrote:
>
> > Congrats, Zhijiang!
> >
> > On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:
> >
> > > Congratulations Zhijiang!
> > >
> > >
> > > On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:
> > >
> > > > Congratulations Zhijiang!
> > > >
> > > > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> > > >
> > > > > Congrats, Zhejiang!
> > > > >
> > > > >
> > > > > Best,
> > > > > Haibo
> > > > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > > > >Congratulations Zhijiang, well deserved.
> > > > > >
> > > > > >Best
> > > > > >
> > > > > >From: Yingjie Cao 
> > > > > >Sent: Tuesday, July 23, 2019 10:23
> > > > > >To: dev@flink.apache.org 
> > > > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a
> committer
> > to
> > > > > the Flink project
> > > > > >
> > > > > >Congratulations Zhijiang!
> > > > > >
> > > > > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > > > > >
> > > > > >> Congrats, Zhejiang!
> > > > > >>
> > > > > >> Best,
> > > > > >> Tao Yang
> > > > > >>
> > > > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > > > >> >
> > > > > >> > Congratulations Zhijiang
> > > > > >> >
> > > > > >> > 发自我的 iPhone
> > > > > >> >
> > > > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > > > >> >>
> > > > > >> >> Congratulations, Zhijiang!
> > > > > >> >>
> > > > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> > > wbeaglewatc...@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >> >>>
> > > > > >> >>> Congratulations Zhijiang!
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> Best,
> > > > > >> >>>
> > > > > >> >>> Bo WANG
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > > > rmetz...@apache.org>
> > > > > >> >>> wrote:
> > > > > >> >>>
> > > > > >>  Hey all,
> > > > > >> 
> > > > > >>  We've added another committer to the Flink project:
> Zhijiang
> > > > Wang.
> > > > > >> 
> > > > > >>  Congratulations Zhijiang!
> > > > > >> 
> > > > > >>  Best,
> > > > > >>  Robert
> > > > > >>  (on behalf of the Flink PMC)
> > > > > >> 
> > > > > >> >>>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Xuefu Zhang
> > > > > >> >>
> > > > > >> >> "In Honey We Trust!"
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[vino yang]

Congratulations Kurt!

Bo WANG  于2019年7月23日周二 下午7:13写道：

> Congratulations Kurt!
>
>
> Best,
>
> Bo WANG
>
>
> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> wrote:
>
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Kete has been a committer since February 2017, working a lot on Table
> API /
> > SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> work
> > for Flink!
> >
> > Congratulations & Welcome Kurt!
> >
> > Best,
> > Robert
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Guowei Ma]

Congratulations Kurt!

发自我的 iPhone

> 在 2019年7月23日，下午7:13，Bo WANG  写道：
> 
> Congratulations Kurt!
> 
> 
> Best,
> 
> Bo WANG
> 
> 
>> On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger  wrote:
>> 
>> Hi all,
>> 
>> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
>> part of the Apache Flink Project Management Committee (PMC).
>> 
>> Kete has been a committer since February 2017, working a lot on Table API /
>> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
>> for Flink!
>> 
>> Congratulations & Welcome Kurt!
>> 
>> Best,
>> Robert
>> 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Bo WANG]

Congratulations Kurt!


Best,

Bo WANG


On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger  wrote:

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Kete has been a committer since February 2017, working a lot on Table API /
> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
> for Flink!
>
> Congratulations & Welcome Kurt!
>
> Best,
> Robert
>

Re:[ANNOUNCE] Kete Young is now part of the Flink PMC

[Haibo Sun]

Congrats Kurt!


Best,
Haibo
At 2019-07-23 17:24:18, "Robert Metzger"  wrote:
>Hi all,
>
>On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
>part of the Apache Flink Project Management Committee (PMC).
>
>Kete has been a committer since February 2017, working a lot on Table API /
>SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
>for Flink!
>
>Congratulations & Welcome Kurt!
>
>Best,
>Robert

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Xintong Song]

Congratulations, Kurt~!

Thank you~

Xintong Song



On Tue, Jul 23, 2019 at 6:55 PM aihua li  wrote:

> Congratulations Kurt, Well deserved.
>
> > 在 2019年7月23日，下午5:24，Robert Metzger  写道：
> >
> > Hi all,
> >
> > On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> > part of the Apache Flink Project Management Committee (PMC).
> >
> > Kete has been a committer since February 2017, working a lot on Table
> API /
> > SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> work
> > for Flink!
> >
> > Congratulations & Welcome Kurt!
> >
> > Best,
> > Robert
>
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[aihua li]

Congratulations Kurt, Well deserved.

> 在 2019年7月23日，下午5:24，Robert Metzger  写道：
> 
> Hi all,
> 
> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> part of the Apache Flink Project Management Committee (PMC).
> 
> Kete has been a committer since February 2017, working a lot on Table API /
> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
> for Flink!
> 
> Congratulations & Welcome Kurt!
> 
> Best,
> Robert

Use sql to access the flink state

[zhang yue]

Does flink have a plan to support sql to access flink state

Re: Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

[Kostas Kloudas]

Hi Elkhan,

As Till pointed out, the fact that your files remain in-progress is the
expected behaviour as the StreamingFileSink assumes
checkpointing is enabled in order to work. There are no plans to change
this behaviour but an issue that may be relevant
for you is https://issues.apache.org/jira/browse/FLINK-13027, but it still
assumes checkpointing is enabled.

Could you elaborate though on what is the reason for not enabling
checkpointing, even if exactly-once is not a necessity?

Cheers,
Kostas


On Tue, Jul 23, 2019 at 11:58 AM Till Rohrmann 
wrote:

> Hi,
>
> as the documentation for the StreamingFileSink indicates [1], it is
> required to enable checkpoints if you want to use bulk encoded output
> formats atm.
>
> I'm not sure whether there are concrete plans to change this behaviour in
> the future because it breaks with exactly once processing guarantees. Klou
> might know more.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html
>
> Cheers,
> Till
>
> On Mon, Jul 22, 2019 at 10:21 PM Elkhan Dadashov <
> elkhan.dadas...@gmail.com> wrote:
>
>> Hi Flink Dev team,
>>
>> Will StreamingFileSink.forBulkFormat(...) support overriding
>> OnCheckpointRollingPolicy?
>>
>> Does anyone use StreamingFileSink *with checkpoint disabled *for writing
>> Parquet output files?
>>
>> The output parquet files are generated, but they are empty, and stay in
>> *inprogress* state, even when the job completes:
>>
>> .part-0-0.inprogress.3e31ba42-588c-48cc-ad6d-d0ebcf1d8632
>> .part-1-0.inprogress.78e1f1dc-3c1c-417b-8270-2bf0298f985a
>> .part-2-0.inprogress.087cf3f1-7e2d-4a03-a518-62f576ed7eea
>>
>> Exactly-once semantics is not important for my case, would then using
>> *BucketingSink* is the only option ?
>>
>> Thanks.
>>
>

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Jark Wu]

Thank you all for your positive feedback.

We have three binding +1s, so I think, we can proceed with this.

Hi @Robert Metzger  , could you create a request to
INFRA for the mailing list?
I'm not sure if this needs a PMC permission.

Thanks,
Jark

On Tue, 23 Jul 2019 at 16:42, jincheng sun  wrote:

> +1
>
> Robert Metzger  于2019年7月23日周二 下午4:01写道：
>
> > +1
> >
> > On Mon, Jul 22, 2019 at 10:27 AM Biao Liu  wrote:
> >
> > > +1, make sense to me.
> > > Mailing list seems to be a more "community" way.
> > >
> > > Timo Walther  于2019年7月22日周一 下午4:06写道：
> > >
> > > > +1 sounds good to inform people about instabilities or other issues
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > >
> > > > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > > > +1. Sounds good.Letting the PR creators know the build results of
> the
> > > > master branch can help to determine more quickly whether Travis
> > failures
> > > of
> > > > their PR are an exact failure or because of the instability of test
> > case.
> > > > By the way, if the PR creator can abort their own Travis build, I
> think
> > > it
> > > > can improve the efficient use of Travis resources (of course, this
> > > > discussion does not deal with this issue).
> > > > >
> > > > >
> > > > > Best,
> > > > > Haibo
> > > > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > > > >> +1 sounds good, more people could be encouraged to involve in
> paying
> > > > attention to failure commit.
> > > > >>
> > > > >> Best
> > > > >> Yun Tang
> > > > >> 
> > > > >> From: Becket Qin 
> > > > >> Sent: Monday, July 22, 2019 9:44
> > > > >> To: dev 
> > > > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing
> list
> > > > for travis builds
> > > > >>
> > > > >> +1. Sounds a good idea to me.
> > > > >>
> > > > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu 
> > > wrote:
> > > > >>
> > > > >>> Thanks Jark for the proposal, sounds reasonable for me. +1. This
> ML
> > > > could
> > > > >>> be used for all the build notifications including master and CRON
> > > jobs.
> > > > >>>
> > > >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > > > 
> > > >  +1 ,Thanks jark for the proposal.
> > > >  Best
> > > >  Forward
> > > > 
> > > >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > > > 
> > > > > Hi all,
> > > > >
> > > > > As far as I know, currently, email notifications of Travis
> builds
> > > for
> > > > > master branch are sent to the commit author when a build was
> just
> > > > >>> broken or
> > > > > still is broken. And there is no email notifications for CRON
> > > builds.
> > > > >
> > > > > Recently, we are suffering from compile errors for scala-2.12
> and
> > > > java-9
> > > > > which are only ran in CRON jobs. So I'm figuring out a way to
> get
> > > > > notifications of CRON builds (or all builds) to quick fix
> compile
> > > > errors
> > > > > and failed cron tests.
> > > > >
> > > > > After reaching out to @Chesnay Schepler 
> > > (thanks
> > > > >>> for
> > > > > the helping), I know that we are using a Slack channel to
> receive
> > > all
> > > > > failed build notifications. However, IMO, email notification
> > might
> > > > be a
> > > > > better way than Slack channel to encourage more people to pay
> > > > attention
> > > > >>> on
> > > > > the builds.
> > > > >
> > > > > So I'm here to propose to setup a bui...@flink.apache.org
> > mailing
> > > > list
> > > > >>> for
> > > > > receiving build notifications. I also find that Beam has such
> > > mailing
> > > > >>> list
> > > > > too[1]. After we have such a mailing list, we can integrate it
> to
> > > > travis
> > > > > according to the travis doc[2].
> > > > >
> > > > > What do you think? Do we need a formal vote for this?
> > > > >
> > > > > Best and thanks,
> > > > > Jark
> > > > >
> > > > > [1]: https://beam.apache.org/community/contact-us/
> > > > > [2]:
> > > > >
> > > > >
> > > > >>>
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > <
> > > > >
> > > > >>>
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > <
> > > > >
> > > > >>>
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > >>>
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-13383) Customize the problem in the trigger

[jinguishi (JIRA)]

jinguishi created FLINK-13383:
-

 Summary: Customize the problem in the trigger
 Key: FLINK-13383
 URL: https://issues.apache.org/jira/browse/FLINK-13383
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.8.0
 Environment: The development environment is idea, the flink version is 
1.8
Reporter: jinguishi
 Fix For: 1.8.0
 Attachments: WX20190723-174236.png, WechatIMG2.png

Using a Tumbling time window, I created a time-based and counter trigger. The 
parameters in the onElement method TriggerContext, ctx.getCurrentWatermark(), 
get negative values, 
There are screenshots in the attachment。

code show as below


{code:java}
public class CountTrigger extends Trigger {

private static final long serialVersionUID = 1L;

private CountTrigger(int count) {
this.threshold = count;
}

private int count = 0;
private int threshold;

@Override
public TriggerResult onElement(Object element, long timestamp, TimeWindow 
window, TriggerContext ctx) {

long watermark = ctx.getCurrentWatermark();
ctx.registerEventTimeTimer(window.maxTimestamp());
if (count > threshold) {
count = 0;
return TriggerResult.FIRE;
} else {
count++;
}
System.out.println("onElement: " + element);
return TriggerResult.CONTINUE;
}

@Override
public TriggerResult onEventTime(long time, TimeWindow window, 
Trigger.TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}

@Override
public TriggerResult onProcessingTime(long time, TimeWindow window, 
Trigger.TriggerContext ctx) throws Exception {
return TriggerResult.FIRE;
}

@Override
public void clear(TimeWindow window, Trigger.TriggerContext ctx) throws 
Exception {
ctx.deleteProcessingTimeTimer(window.maxTimestamp());
}

@Override
public String toString() {
return "CountTrigger";
}

public static  CountTrigger of(int threshold) {
return new CountTrigger(threshold);
}
}

{code}






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: contributor permission application

[Danny Chan]

Hi, zhang yue

There is no need to apply permission for JIRA issues now, you can left your 
message under the JIRA issues you are interested in, the committers would 
assign it to you if they saw your message.

Best,
Danny Chan
在 2019年7月23日 +0800 PM6:06，zhang yue ，写道：
> No one responded to me,What else do I need to do?
>
> > 在 2019年7月17日，下午12:16，zhang yue  写道：
> >
> > Hi,
> >
> > I want to contribute to Apache Flink.
> > Would you please give me the contributor permission?
> > My JIRA ID is mooonzhang.
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Yun Gao]

Congratulations Kurt!

best,
Yun


--
From:Jeff Zhang 
Send Time:2019 Jul. 23 (Tue.) 18:14
To:dev 
Subject:Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

Congratulations Kurt

Biao Liu  于2019年7月23日周二 下午6:07写道：

> Congrats Kurt!
> Well deserved!
>
> Danny Chan  于2019年7月23日周二 下午6:01写道：
>
> > Congratulations Kurt, Well deserved.
> >
> > Best,
> > Danny Chan
> > 在 2019年7月23日 +0800 PM5:24，dev@flink.apache.org，写道：
> > >
> > > Congratulations Kurt, Well deserved.
> >
>


-- 
Best Regards

Jeff Zhang

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[JingsongLee]

Congratulations Zhijiang!

Best, Jingsong Lee


--
From:wenlong.lwl 
Send Time:2019年7月23日(星期二) 17:34
To:dev 
Subject:Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the 
Flink project

Congrats, Zhijiang!

On Tue, 23 Jul 2019 at 15:59, Becket Qin  wrote:

> Congrats, Zhijiang!
>
> On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:
>
> > Congratulations Zhijiang!
> >
> >
> > On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:
> >
> > > Congratulations Zhijiang!
> > >
> > > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> > >
> > > > Congrats, Zhejiang!
> > > >
> > > >
> > > > Best,
> > > > Haibo
> > > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > > >Congratulations Zhijiang, well deserved.
> > > > >
> > > > >Best
> > > > >
> > > > >From: Yingjie Cao 
> > > > >Sent: Tuesday, July 23, 2019 10:23
> > > > >To: dev@flink.apache.org 
> > > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer
> to
> > > > the Flink project
> > > > >
> > > > >Congratulations Zhijiang!
> > > > >
> > > > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > > > >
> > > > >> Congrats, Zhejiang!
> > > > >>
> > > > >> Best,
> > > > >> Tao Yang
> > > > >>
> > > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > > >> >
> > > > >> > Congratulations Zhijiang
> > > > >> >
> > > > >> > 发自我的 iPhone
> > > > >> >
> > > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > > >> >>
> > > > >> >> Congratulations, Zhijiang!
> > > > >> >>
> > > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> > wbeaglewatc...@gmail.com
> > > >
> > > > >> wrote:
> > > > >> >>>
> > > > >> >>> Congratulations Zhijiang!
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> Best,
> > > > >> >>>
> > > > >> >>> Bo WANG
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > > rmetz...@apache.org>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >>  Hey all,
> > > > >> 
> > > > >>  We've added another committer to the Flink project: Zhijiang
> > > Wang.
> > > > >> 
> > > > >>  Congratulations Zhijiang!
> > > > >> 
> > > > >>  Best,
> > > > >>  Robert
> > > > >>  (on behalf of the Flink PMC)
> > > > >> 
> > > > >> >>>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >> Xuefu Zhang
> > > > >> >>
> > > > >> >> "In Honey We Trust!"
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: contributor permission application

[highfei2011]

Hi zhang yue,
   The new JIRA workflow is in effect, and contributors don't have to go to dev 
to apply for contributor permissions. If you are interested in JIRA, you can 
reply directly to JIRA (of course, for complex issues, also to clarify the 
implementation). Then there will be a committer/pmc assign issue for you.




Best,
Yang


原始邮件
发件人:zhang yuezhang...@silvrr.com
收件人:dev...@flink.apache.org
发送时间:2019年7月23日(周二) 18:06
主题:Re: contributor permission application


No one responded to me,What else do I need to do?  在 2019年7月17日，下午12:16，zhang 
yue zhang...@silvrr.com 写道：   Hi,   I want to contribute to Apache Flink.  
Would you please give me the contributor permission?  My JIRA ID is mooonzhang.

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Jeff Zhang]

Congratulations Kurt

Biao Liu  于2019年7月23日周二 下午6:07写道：

> Congrats Kurt!
> Well deserved!
>
> Danny Chan  于2019年7月23日周二 下午6:01写道：
>
> > Congratulations Kurt, Well deserved.
> >
> > Best,
> > Danny Chan
> > 在 2019年7月23日 +0800 PM5:24，dev@flink.apache.org，写道：
> > >
> > > Congratulations Kurt, Well deserved.
> >
>


-- 
Best Regards

Jeff Zhang

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Terry Wang]

Congrats Kurt!
Well deserved!

> 在 2019年7月23日，下午6:07，Biao Liu  写道：
> 
> Congrats Kurt!
> Well deserved!
> 
> Danny Chan  于2019年7月23日周二 下午6:01写道：
> 
>> Congratulations Kurt, Well deserved.
>> 
>> Best,
>> Danny Chan
>> 在 2019年7月23日 +0800 PM5:24，dev@flink.apache.org，写道：
>>> 
>>> Congratulations Kurt, Well deserved.
>> 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Biao Liu]

Congrats Kurt!
Well deserved!

Danny Chan  于2019年7月23日周二 下午6:01写道：

> Congratulations Kurt, Well deserved.
>
> Best,
> Danny Chan
> 在 2019年7月23日 +0800 PM5:24，dev@flink.apache.org，写道：
> >
> > Congratulations Kurt, Well deserved.
>

Re: contributor permission application

[zhang yue]

No one responded to me,What else do I need to do?

> 在 2019年7月17日，下午12:16，zhang yue  写道：
> 
> Hi,
> 
> I want to contribute to Apache Flink.
> Would you please give me the contributor permission?
> My JIRA ID is mooonzhang.

Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Danny Chan]

Congratulations Zhijiang!

Best,
Danny Chan
在 2019年7月22日 +0800 PM10:12，dev@flink.apache.org，写道：
>
> Congratulations Zhijiang!

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Danny Chan]

Congratulations Kurt, Well deserved.

Best,
Danny Chan
在 2019年7月23日 +0800 PM5:24，dev@flink.apache.org，写道：
>
> Congratulations Kurt, Well deserved.

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Yun Tang]

Congratulations Kurt, Well deserved.

Best

From: Xu Forward 
Sent: Tuesday, July 23, 2019 17:53
To: dev 
Subject: Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

Congratulations Kurt!


best

forward

Jark Wu  于2019年7月23日周二 下午5:49写道：

> Congratulations Kurt! Well deserved.
>
> Cheers,
> Jark
>
> On Tue, 23 Jul 2019 at 17:43, LakeShen  wrote:
>
> > Congratulations Kurt!
> >
> > Congxian Qiu  于2019年7月23日周二 下午5:37写道：
> >
> > > Congratulations Kurt!
> > > Best,
> > > Congxian
> > >
> > >
> > > Dian Fu  于2019年7月23日周二 下午5:36写道：
> > >
> > > > Congrats, Kurt!
> > > >
> > > > > 在 2019年7月23日，下午5:33，Zili Chen  写道：
> > > > >
> > > > > Congratulations Kurt!
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > JingsongLee  于2019年7月23日周二
> > 下午5:29写道：
> > > > >
> > > > >> Congratulations Kurt!
> > > > >>
> > > > >> Best, Jingsong Lee
> > > > >>
> > > > >>
> > > > >> --
> > > > >> From:Robert Metzger 
> > > > >> Send Time:2019年7月23日(星期二) 17:24
> > > > >> To:dev 
> > > > >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> On behalf of the Flink PMC, I'm happy to announce that Kete Young
> is
> > > now
> > > > >> part of the Apache Flink Project Management Committee (PMC).
> > > > >>
> > > > >> Kete has been a committer since February 2017, working a lot on
> > Table
> > > > API /
> > > > >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for
> > your
> > > > work
> > > > >> for Flink!
> > > > >>
> > > > >> Congratulations & Welcome Kurt!
> > > > >>
> > > > >> Best,
> > > > >> Robert
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

[Till Rohrmann]

Hi,

as the documentation for the StreamingFileSink indicates [1], it is
required to enable checkpoints if you want to use bulk encoded output
formats atm.

I'm not sure whether there are concrete plans to change this behaviour in
the future because it breaks with exactly once processing guarantees. Klou
might know more.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html

Cheers,
Till

On Mon, Jul 22, 2019 at 10:21 PM Elkhan Dadashov 
wrote:

> Hi Flink Dev team,
>
> Will StreamingFileSink.forBulkFormat(...) support overriding
> OnCheckpointRollingPolicy?
>
> Does anyone use StreamingFileSink *with checkpoint disabled *for writing
> Parquet output files?
>
> The output parquet files are generated, but they are empty, and stay in
> *inprogress* state, even when the job completes:
>
> .part-0-0.inprogress.3e31ba42-588c-48cc-ad6d-d0ebcf1d8632
> .part-1-0.inprogress.78e1f1dc-3c1c-417b-8270-2bf0298f985a
> .part-2-0.inprogress.087cf3f1-7e2d-4a03-a518-62f576ed7eea
>
> Exactly-once semantics is not important for my case, would then using
> *BucketingSink* is the only option ?
>
> Thanks.
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Xu Forward]

Congratulations Kurt!


best

forward

Jark Wu  于2019年7月23日周二 下午5:49写道：

> Congratulations Kurt! Well deserved.
>
> Cheers,
> Jark
>
> On Tue, 23 Jul 2019 at 17:43, LakeShen  wrote:
>
> > Congratulations Kurt!
> >
> > Congxian Qiu  于2019年7月23日周二 下午5:37写道：
> >
> > > Congratulations Kurt!
> > > Best,
> > > Congxian
> > >
> > >
> > > Dian Fu  于2019年7月23日周二 下午5:36写道：
> > >
> > > > Congrats, Kurt!
> > > >
> > > > > 在 2019年7月23日，下午5:33，Zili Chen  写道：
> > > > >
> > > > > Congratulations Kurt!
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > JingsongLee  于2019年7月23日周二
> > 下午5:29写道：
> > > > >
> > > > >> Congratulations Kurt!
> > > > >>
> > > > >> Best, Jingsong Lee
> > > > >>
> > > > >>
> > > > >> --
> > > > >> From:Robert Metzger 
> > > > >> Send Time:2019年7月23日(星期二) 17:24
> > > > >> To:dev 
> > > > >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> On behalf of the Flink PMC, I'm happy to announce that Kete Young
> is
> > > now
> > > > >> part of the Apache Flink Project Management Committee (PMC).
> > > > >>
> > > > >> Kete has been a committer since February 2017, working a lot on
> > Table
> > > > API /
> > > > >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for
> > your
> > > > work
> > > > >> for Flink!
> > > > >>
> > > > >> Congratulations & Welcome Kurt!
> > > > >>
> > > > >> Best,
> > > > >> Robert
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Jark Wu]

Congratulations Kurt! Well deserved.

Cheers,
Jark

On Tue, 23 Jul 2019 at 17:43, LakeShen  wrote:

> Congratulations Kurt!
>
> Congxian Qiu  于2019年7月23日周二 下午5:37写道：
>
> > Congratulations Kurt!
> > Best,
> > Congxian
> >
> >
> > Dian Fu  于2019年7月23日周二 下午5:36写道：
> >
> > > Congrats, Kurt!
> > >
> > > > 在 2019年7月23日，下午5:33，Zili Chen  写道：
> > > >
> > > > Congratulations Kurt!
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > JingsongLee  于2019年7月23日周二
> 下午5:29写道：
> > > >
> > > >> Congratulations Kurt!
> > > >>
> > > >> Best, Jingsong Lee
> > > >>
> > > >>
> > > >> --
> > > >> From:Robert Metzger 
> > > >> Send Time:2019年7月23日(星期二) 17:24
> > > >> To:dev 
> > > >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
> > > >>
> > > >> Hi all,
> > > >>
> > > >> On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> > now
> > > >> part of the Apache Flink Project Management Committee (PMC).
> > > >>
> > > >> Kete has been a committer since February 2017, working a lot on
> Table
> > > API /
> > > >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for
> your
> > > work
> > > >> for Flink!
> > > >>
> > > >> Congratulations & Welcome Kurt!
> > > >>
> > > >> Best,
> > > >> Robert
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[LakeShen]

Congratulations Kurt!

Congxian Qiu  于2019年7月23日周二 下午5:37写道：

> Congratulations Kurt!
> Best,
> Congxian
>
>
> Dian Fu  于2019年7月23日周二 下午5:36写道：
>
> > Congrats, Kurt!
> >
> > > 在 2019年7月23日，下午5:33，Zili Chen  写道：
> > >
> > > Congratulations Kurt!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > JingsongLee  于2019年7月23日周二 下午5:29写道：
> > >
> > >> Congratulations Kurt!
> > >>
> > >> Best, Jingsong Lee
> > >>
> > >>
> > >> --
> > >> From:Robert Metzger 
> > >> Send Time:2019年7月23日(星期二) 17:24
> > >> To:dev 
> > >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
> > >>
> > >> Hi all,
> > >>
> > >> On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> now
> > >> part of the Apache Flink Project Management Committee (PMC).
> > >>
> > >> Kete has been a committer since February 2017, working a lot on Table
> > API /
> > >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> > work
> > >> for Flink!
> > >>
> > >> Congratulations & Welcome Kurt!
> > >>
> > >> Best,
> > >> Robert
> > >>
> >
> >
>

Re: [DISCUSS] Support temporary tables in SQL API

[JingsongLee]

And I think we should recommend user to use catalog api to
 createTable and createFunction,(I guess most scenarios do
 not use temporary objects) in this way, it is good to option #3

Best, JingsongLee


--
From:JingsongLee 
Send Time:2019年7月23日(星期二) 17:35
To:dev 
Subject:Re: [DISCUSS] Support temporary tables in SQL API

Thanks Dawid and other people.
+1 for using option #3 for 1.9.0 and go with option #1
 in 1.10.0.

Regarding Xuefu's concern, I don't know how necessary it is for each catalog to
 deal with tmpView. I think Catalog is different from DB, we can have single 
concept for tmpView, that make user easier to understand.

Regarding option #2, It is hard to use if we let user to use fully qualified 
name for hive catalog. Would this experience be too bad to use?

Best, Jingsong Lee


--
From:Kurt Young 
Send Time:2019年7月23日(星期二) 17:03
To:dev 
Subject:Re: [DISCUSS] Support temporary tables in SQL API

Thanks Dawid for driving this discussion.
Personally, I would +1 for using option #2 for 1.9.0 and go with option #1
in 1.10.0.

Regarding Xuefu's concern about option #1, I think we could also try to
reuse the in-memory catalog
for the builtin temporary table storage.

Regarding to option #2 and option #3, from user's perspective, IIUC option
#2 allows user to have
simple name to reference temporary table and should use fully qualified
name for external catalogs.
But option #3 provide the opposite behavior, user can use simple name for
external tables after he
changed current catalog and current database, but have to use fully
qualified name for temporary
tables. IMO, option #2 will be more straightforward.

Best,
Kurt


On Tue, Jul 23, 2019 at 4:01 PM Aljoscha Krettek 
wrote:

> I would be fine with option 3) but I think option 2) is the more implicit
> solution that has less surprising behaviour.
>
> Aljoscha
>
> > On 22. Jul 2019, at 23:59, Xuefu Zhang  wrote:
> >
> > Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
> > that for 1.9 we should have some quick and simple solution, leaving time
> > for more thorough discussions for 1.10.
> >
> > In particular, I'm not fully with solution #1. For one thing, it seems
> > proposing storing all temporary objects in a memory map in
> CatalogManager,
> > and the memory map duplicates the functionality of the in-memory catalog,
> > which also store temporary objects. For another, as pointed out by the
> > google doc, different db may handle the temporary tables differently, and
> > accordingly it may make more sense to let each catalog to handle its
> > temporary objects.
> >
> > Therefore, postponing the fix buys us time to flush out all the details.
> >
> > Thanks,
> > Xuefu
> >
> > On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  wrote:
> >
> >> Thanks for summarizing our offline discussion Dawid! Even though I would
> >> prefer solution 1 instead of releasing half-baked features, I also
> >> understand that the Table API should not further block the next release.
> >> Therefore, I would be fine with solution 3 but introduce the new
> >> user-facing `createTemporaryTable` methods as synonyms of the existing
> >> ones already. This allows us to deprecate the methods with undefined
> >> behavior as early as possible.
> >>
> >> Thanks,
> >> Timo
> >>
> >>
> >> Am 22.07.19 um 16:13 schrieb Dawid Wysakowicz:
> >>> Hi all,
> >>>
> >>> When working on FLINK-13279[1] we realized we could benefit from a
> >>> better temporary objects support in the Catalog API/Table API.
> >>> Unfortunately we are already long past the feature freeze that's why I
> >>> wanted to get some opinions from the community how should we proceed
> >>> with this topic. I tried to prepare a summary of the current state and
> 3
> >>> different suggested approaches that we could take. Please see the
> >>> attached document[2]
> >>>
> >>> I will appreciate your thoughts!
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/FLINK-13279
> >>>
> >>> [2]
> >>>
> >>
> https://docs.google.com/document/d/1RxLj4tDB9GXVjF5qrkM38SKUPkvJt_BSefGYTQ-cVX4/edit?usp=sharing
> >>>
> >>>
> >>
> >>
>
>

[jira] [Created] (FLINK-13382) Port DecimalITCase to unit tests

[Jark Wu (JIRA)]

Jark Wu created FLINK-13382:
---

 Summary: Port DecimalITCase to unit tests
 Key: FLINK-13382
 URL: https://issues.apache.org/jira/browse/FLINK-13382
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Jark Wu
 Fix For: 1.10.0


There are 81 query tests in 
{{org.apache.flink.table.runtime.batch.sql.DecimalITCase}} and 67 query tests 
in {{org.apache.flink.table.runtime.batch.table.DecimalITCase}}. 

Note that each query will be executed on a MiniCluster which will consume a lot 
of testing time. However, decimals can be tested as unit tests similar to other 
Scalar functions tests.

So I would suggest to move Decimal IT cases to unit tests 
{{org.apache.flink.table.expressions.DecimalTypeTest}}. This should save much 
testing time. 

In the meanwhile, IMO, this is not required by 1.9 release. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Congxian Qiu]

Congratulations Kurt!
Best,
Congxian


Dian Fu  于2019年7月23日周二 下午5:36写道：

> Congrats, Kurt!
>
> > 在 2019年7月23日，下午5:33，Zili Chen  写道：
> >
> > Congratulations Kurt!
> >
> > Best,
> > tison.
> >
> >
> > JingsongLee  于2019年7月23日周二 下午5:29写道：
> >
> >> Congratulations Kurt!
> >>
> >> Best, Jingsong Lee
> >>
> >>
> >> --
> >> From:Robert Metzger 
> >> Send Time:2019年7月23日(星期二) 17:24
> >> To:dev 
> >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
> >>
> >> Hi all,
> >>
> >> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> >> part of the Apache Flink Project Management Committee (PMC).
> >>
> >> Kete has been a committer since February 2017, working a lot on Table
> API /
> >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> work
> >> for Flink!
> >>
> >> Congratulations & Welcome Kurt!
> >>
> >> Best,
> >> Robert
> >>
>
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Dian Fu]

Congrats, Kurt!

> 在 2019年7月23日，下午5:33，Zili Chen  写道：
> 
> Congratulations Kurt!
> 
> Best,
> tison.
> 
> 
> JingsongLee  于2019年7月23日周二 下午5:29写道：
> 
>> Congratulations Kurt!
>> 
>> Best, Jingsong Lee
>> 
>> 
>> --
>> From:Robert Metzger 
>> Send Time:2019年7月23日(星期二) 17:24
>> To:dev 
>> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
>> 
>> Hi all,
>> 
>> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
>> part of the Apache Flink Project Management Committee (PMC).
>> 
>> Kete has been a committer since February 2017, working a lot on Table API /
>> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
>> for Flink!
>> 
>> Congratulations & Welcome Kurt!
>> 
>> Best,
>> Robert
>> 

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[wenlong.lwl]

Congratulations Kurt!

On Tue, 23 Jul 2019 at 17:24, Robert Metzger  wrote:

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Kete has been a committer since February 2017, working a lot on Table API /
> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
> for Flink!
>
> Congratulations & Welcome Kurt!
>
> Best,
> Robert
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Zili Chen]

Congratulations Kurt!

Best,
tison.


JingsongLee  于2019年7月23日周二 下午5:29写道：

> Congratulations Kurt!
>
> Best, Jingsong Lee
>
>
> --
> From:Robert Metzger 
> Send Time:2019年7月23日(星期二) 17:24
> To:dev 
> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC
>
> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Kete has been a committer since February 2017, working a lot on Table API /
> SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
> for Flink!
>
> Congratulations & Welcome Kurt!
>
> Best,
> Robert
>

Re: [DISCUSS] Support temporary tables in SQL API

[JingsongLee]

Thanks Dawid and other people.
+1 for using option #3 for 1.9.0 and go with option #1
 in 1.10.0.

Regarding Xuefu's concern, I don't know how necessary it is for each catalog to
 deal with tmpView. I think Catalog is different from DB, we can have single 
concept for tmpView, that make user easier to understand.

Regarding option #2, It is hard to use if we let user to use fully qualified 
name for hive catalog. Would this experience be too bad to use?

Best, Jingsong Lee


--
From:Kurt Young 
Send Time:2019年7月23日(星期二) 17:03
To:dev 
Subject:Re: [DISCUSS] Support temporary tables in SQL API

Thanks Dawid for driving this discussion.
Personally, I would +1 for using option #2 for 1.9.0 and go with option #1
in 1.10.0.

Regarding Xuefu's concern about option #1, I think we could also try to
reuse the in-memory catalog
for the builtin temporary table storage.

Regarding to option #2 and option #3, from user's perspective, IIUC option
#2 allows user to have
simple name to reference temporary table and should use fully qualified
name for external catalogs.
But option #3 provide the opposite behavior, user can use simple name for
external tables after he
changed current catalog and current database, but have to use fully
qualified name for temporary
tables. IMO, option #2 will be more straightforward.

Best,
Kurt


On Tue, Jul 23, 2019 at 4:01 PM Aljoscha Krettek 
wrote:

> I would be fine with option 3) but I think option 2) is the more implicit
> solution that has less surprising behaviour.
>
> Aljoscha
>
> > On 22. Jul 2019, at 23:59, Xuefu Zhang  wrote:
> >
> > Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
> > that for 1.9 we should have some quick and simple solution, leaving time
> > for more thorough discussions for 1.10.
> >
> > In particular, I'm not fully with solution #1. For one thing, it seems
> > proposing storing all temporary objects in a memory map in
> CatalogManager,
> > and the memory map duplicates the functionality of the in-memory catalog,
> > which also store temporary objects. For another, as pointed out by the
> > google doc, different db may handle the temporary tables differently, and
> > accordingly it may make more sense to let each catalog to handle its
> > temporary objects.
> >
> > Therefore, postponing the fix buys us time to flush out all the details.
> >
> > Thanks,
> > Xuefu
> >
> > On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  wrote:
> >
> >> Thanks for summarizing our offline discussion Dawid! Even though I would
> >> prefer solution 1 instead of releasing half-baked features, I also
> >> understand that the Table API should not further block the next release.
> >> Therefore, I would be fine with solution 3 but introduce the new
> >> user-facing `createTemporaryTable` methods as synonyms of the existing
> >> ones already. This allows us to deprecate the methods with undefined
> >> behavior as early as possible.
> >>
> >> Thanks,
> >> Timo
> >>
> >>
> >> Am 22.07.19 um 16:13 schrieb Dawid Wysakowicz:
> >>> Hi all,
> >>>
> >>> When working on FLINK-13279[1] we realized we could benefit from a
> >>> better temporary objects support in the Catalog API/Table API.
> >>> Unfortunately we are already long past the feature freeze that's why I
> >>> wanted to get some opinions from the community how should we proceed
> >>> with this topic. I tried to prepare a summary of the current state and
> 3
> >>> different suggested approaches that we could take. Please see the
> >>> attached document[2]
> >>>
> >>> I will appreciate your thoughts!
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/FLINK-13279
> >>>
> >>> [2]
> >>>
> >>
> https://docs.google.com/document/d/1RxLj4tDB9GXVjF5qrkM38SKUPkvJt_BSefGYTQ-cVX4/edit?usp=sharing
> >>>
> >>>
> >>
> >>
>
>

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[wenlong.lwl]

Congrats, Zhijiang!

On Tue, 23 Jul 2019 at 15:59, Becket Qin  wrote:

> Congrats, Zhijiang!
>
> On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:
>
> > Congratulations Zhijiang!
> >
> >
> > On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:
> >
> > > Congratulations Zhijiang!
> > >
> > > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> > >
> > > > Congrats, Zhejiang!
> > > >
> > > >
> > > > Best,
> > > > Haibo
> > > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > > >Congratulations Zhijiang, well deserved.
> > > > >
> > > > >Best
> > > > >
> > > > >From: Yingjie Cao 
> > > > >Sent: Tuesday, July 23, 2019 10:23
> > > > >To: dev@flink.apache.org 
> > > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer
> to
> > > > the Flink project
> > > > >
> > > > >Congratulations Zhijiang!
> > > > >
> > > > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > > > >
> > > > >> Congrats, Zhejiang!
> > > > >>
> > > > >> Best,
> > > > >> Tao Yang
> > > > >>
> > > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > > >> >
> > > > >> > Congratulations Zhijiang
> > > > >> >
> > > > >> > 发自我的 iPhone
> > > > >> >
> > > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > > >> >>
> > > > >> >> Congratulations, Zhijiang!
> > > > >> >>
> > > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> > wbeaglewatc...@gmail.com
> > > >
> > > > >> wrote:
> > > > >> >>>
> > > > >> >>> Congratulations Zhijiang!
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> Best,
> > > > >> >>>
> > > > >> >>> Bo WANG
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > > rmetz...@apache.org>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >>  Hey all,
> > > > >> 
> > > > >>  We've added another committer to the Flink project: Zhijiang
> > > Wang.
> > > > >> 
> > > > >>  Congratulations Zhijiang!
> > > > >> 
> > > > >>  Best,
> > > > >>  Robert
> > > > >>  (on behalf of the Flink PMC)
> > > > >> 
> > > > >> >>>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >> Xuefu Zhang
> > > > >> >>
> > > > >> >> "In Honey We Trust!"
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[JingsongLee]

Congratulations Kurt!

Best, Jingsong Lee


--
From:Robert Metzger 
Send Time:2019年7月23日(星期二) 17:24
To:dev 
Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC

Hi all,

On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
part of the Apache Flink Project Management Committee (PMC).

Kete has been a committer since February 2017, working a lot on Table API /
SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
for Flink!

Congratulations & Welcome Kurt!

Best,
Robert

[ANNOUNCE] Kete Young is now part of the Flink PMC

[Robert Metzger]

Hi all,

On behalf of the Flink PMC, I'm happy to announce that Kete Young is now
part of the Apache Flink Project Management Committee (PMC).

Kete has been a committer since February 2017, working a lot on Table API /
SQL. He's currently co-managing the 1.9 release! Thanks a lot for your work
for Flink!

Congratulations & Welcome Kurt!

Best,
Robert

Re: [DISCUSS] Create a Flink ecosystem website

[Robert Metzger]

Thanks a lot Marta for offering to write a blog post about the community
site!

I'm not sure if multi-language support for the site is a good idea. I see
the packages site as something similar to GitHub or Jira. The page itself
contains very view things we could actually translate. The package owners
usually are only able to provide one or two languages for their package
description.
For the comments, we don't want disjoint discussions to happen.

I've also kicked off a discussion with Apache legal on the initiative [1].
We might not be able to host this at Apache, but let's see where the
discussion goes.


[1]
https://lists.apache.org/thread.html/ee76a02257b51292ab61f6ac8d3d69307e83cc569cdeebde80596207@%3Clegal-discuss.apache.org%3E


On Sat, Jul 20, 2019 at 5:25 AM Becket Qin  wrote:

> [Sorry for the incomplete message. Clicked send by mistake...]
>
> I agree with Marta that it might be good to have multi-language support as
> a mid-term goal.
>
> Jiangjie (Becket) Qin
>
> On Sat, Jul 20, 2019 at 11:22 AM Becket Qin  wrote:
>
>> The website is awesome! I really like its conciseness and yet fairly
>> useful information and functionalities. I cannot think of much to improve
>> at the moment. Just one thought, do we need an "others" category, just in
>> case a package does not fit into any of the current given categories?
>>
>> Thanks Robert and Daryl for the great effort. Looking forward to seeing
>> this get published soon!!
>>
>> I agree with Marta that
>>
>> Jiangjie (Becket) Qin
>>
>> On Sat, Jul 20, 2019 at 1:34 AM Marta Paes Moreira 
>> wrote:
>>
>>> Hey, Robert.
>>>
>>> I will keep an eye on the overall progress and get started on the blog
>>> post
>>> to make the community announcement. Are there (mid-term) plans to
>>> translate/localize this website as well? It might be a point worth
>>> mentioning in the blogpost.
>>>
>>> Hats off to you and Daryl — this turned out amazing!
>>>
>>> Marta
>>>
>>> On Thu, Jul 18, 2019 at 10:57 AM Congxian Qiu 
>>> wrote:
>>>
>>> > Robert and Daryl, thanks for the great work, I tried the website and
>>> filed
>>> > some issues on Github.
>>> > Best,
>>> > Congxian
>>> >
>>> >
>>> > Robert Metzger  于2019年7月17日周三 下午11:28写道：
>>> >
>>> >> Hey all,
>>> >>
>>> >> Daryl and I have great news to share. We are about to finish adding
>>> the
>>> >> basic features to the ecosystem page.
>>> >> We are at a stage where it is ready to be reviewed and made public.
>>> >>
>>> >> You can either check out a development instance of the ecosystem page
>>> >> here: https://flink-ecosystem-demo.flink-resources.org/
>>> >> Or you run it locally, with the instructions from the README.md:
>>> >> https://github.com/sorahn/flink-ecosystem
>>> >>
>>> >> Please report all issues you find here:
>>> >> https://github.com/sorahn/flink-ecosystem/issues or in this thread.
>>> >>
>>> >> The next steps in this project are the following:
>>> >> - We fix all issues reported through this testing
>>> >> - We set up the site on the INFRA resources Becket has secured [1], do
>>> >> some further testing (including email notifications) and pre-fill the
>>> page
>>> >> with some packages.
>>> >> - We set up a packages.flink.apache.org or flink.apache.org/packages
>>> >> domain
>>> >> - We announce the packages through a short blog post
>>> >>
>>> >> Happy testing!
>>> >>
>>> >> Best,
>>> >> Robert
>>> >>
>>> >> [1] https://issues.apache.org/jira/browse/INFRA-18010
>>> >>
>>> >>
>>> >> On Thu, Apr 25, 2019 at 6:23 AM Becket Qin 
>>> wrote:
>>> >>
>>> >>> Thanks for the update, Robert. Looking forward to the website. If
>>> there
>>> >>> is already a list of software we need to run the website, we can ask
>>> Apache
>>> >>> infra team to prepare the VM for us, as that may also take some time.
>>> >>>
>>> >>> On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger >> >
>>> >>> wrote:
>>> >>>
>>>  Hey all,
>>> 
>>>  quick update on this project: The frontend and backend code have
>>> been
>>>  put together into this repository:
>>>  https://github.com/sorahn/flink-ecosystem
>>>  We also just agreed on an API specification, and will now work on
>>>  finishing the backend.
>>> 
>>>  It will probably take a few more weeks for this to finish, but we
>>> are
>>>  making progress :)
>>> 
>>>  Best,
>>>  Robert
>>> 
>>> 
>>>  On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger <
>>> rmetz...@apache.org>
>>>  wrote:
>>> 
>>> > Hey Daryl,
>>> >
>>> > thanks a lot for posting a link to this first prototype on the
>>> mailing
>>> > list! I really like it!
>>> >
>>> > Becket: Our plan forward is that Congxian is implementing the
>>> backend
>>> > for the website. He has already started with the work, but needs
>>> at least
>>> > one more week.
>>> >
>>> >
>>> > [Re-sending this email because the first one was blocked on
>>> dev@f.a.o]
>>> >
>>> >
>>> > On Mon, Apr 15, 2019 at 7:59 AM Becket Qin 
>>> > 

Re: [DISCUSS] Support temporary tables in SQL API

[Kurt Young]

Thanks Dawid for driving this discussion.
Personally, I would +1 for using option #2 for 1.9.0 and go with option #1
in 1.10.0.

Regarding Xuefu's concern about option #1, I think we could also try to
reuse the in-memory catalog
for the builtin temporary table storage.

Regarding to option #2 and option #3, from user's perspective, IIUC option
#2 allows user to have
simple name to reference temporary table and should use fully qualified
name for external catalogs.
But option #3 provide the opposite behavior, user can use simple name for
external tables after he
changed current catalog and current database, but have to use fully
qualified name for temporary
tables. IMO, option #2 will be more straightforward.

Best,
Kurt


On Tue, Jul 23, 2019 at 4:01 PM Aljoscha Krettek 
wrote:

> I would be fine with option 3) but I think option 2) is the more implicit
> solution that has less surprising behaviour.
>
> Aljoscha
>
> > On 22. Jul 2019, at 23:59, Xuefu Zhang  wrote:
> >
> > Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
> > that for 1.9 we should have some quick and simple solution, leaving time
> > for more thorough discussions for 1.10.
> >
> > In particular, I'm not fully with solution #1. For one thing, it seems
> > proposing storing all temporary objects in a memory map in
> CatalogManager,
> > and the memory map duplicates the functionality of the in-memory catalog,
> > which also store temporary objects. For another, as pointed out by the
> > google doc, different db may handle the temporary tables differently, and
> > accordingly it may make more sense to let each catalog to handle its
> > temporary objects.
> >
> > Therefore, postponing the fix buys us time to flush out all the details.
> >
> > Thanks,
> > Xuefu
> >
> > On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  wrote:
> >
> >> Thanks for summarizing our offline discussion Dawid! Even though I would
> >> prefer solution 1 instead of releasing half-baked features, I also
> >> understand that the Table API should not further block the next release.
> >> Therefore, I would be fine with solution 3 but introduce the new
> >> user-facing `createTemporaryTable` methods as synonyms of the existing
> >> ones already. This allows us to deprecate the methods with undefined
> >> behavior as early as possible.
> >>
> >> Thanks,
> >> Timo
> >>
> >>
> >> Am 22.07.19 um 16:13 schrieb Dawid Wysakowicz:
> >>> Hi all,
> >>>
> >>> When working on FLINK-13279[1] we realized we could benefit from a
> >>> better temporary objects support in the Catalog API/Table API.
> >>> Unfortunately we are already long past the feature freeze that's why I
> >>> wanted to get some opinions from the community how should we proceed
> >>> with this topic. I tried to prepare a summary of the current state and
> 3
> >>> different suggested approaches that we could take. Please see the
> >>> attached document[2]
> >>>
> >>> I will appreciate your thoughts!
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/FLINK-13279
> >>>
> >>> [2]
> >>>
> >>
> https://docs.google.com/document/d/1RxLj4tDB9GXVjF5qrkM38SKUPkvJt_BSefGYTQ-cVX4/edit?usp=sharing
> >>>
> >>>
> >>
> >>
>
>

[jira] [Created] (FLINK-13381) BinaryHashTableTest and BinaryExternalSorterTest is crashed on Travis

[Jark Wu (JIRA)]

Jark Wu created FLINK-13381:
---

 Summary: BinaryHashTableTest and BinaryExternalSorterTest  is 
crashed on Travis
 Key: FLINK-13381
 URL: https://issues.apache.org/jira/browse/FLINK-13381
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Jark Wu
 Fix For: 1.9.0, 1.10.0


Here is an instance of master: 
https://api.travis-ci.org/v3/job/562437128/log.txt
Here is an instance of 1.9: https://api.travis-ci.org/v3/job/562380020/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[DISCUSS] Support computed column for Flink SQL

[Danny Chan]

In umbrella task FLINK-10232[1] we have introduced CREATE TABLE grammar in our 
new module flink-sql-parser. And we proposed to use computed column to describe 
the time attribute of process time in the design doc FLINK SQL DDL[2], so user 
may create a table with process time attribute as following:

create table T1(
a int,
b bigint,
c varchar,
d as PROC_TIME,
) with (
k1 = v1,
k2 = v2
);

The column d would be a process time attribute for table T1. There are also 
many other use cases for computed columns[3].

It may not be a big change here, but may touch the TableSchema, which is a 
public API for user now, so i'm very appreciate for your suggestions(especially 
its relationship with the TableSchema).

I write a simple design doc here[3].

[1] https://issues.apache.org/jira/browse/FLINK-10232
[2] 
https://docs.google.com/document/d/1OmVyuPk9ibGUC-CnPHbXvCg_fdG1TeC3lXSnqcUEYmM
[3] 
https://docs.google.com/document/d/110TseRtTCphxETPY7uhiHpu-dph3NEesh3mYKtJ7QOY/edit?usp=sharing

Best,
Danny Chan

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[jincheng sun]

+1

Robert Metzger  于2019年7月23日周二 下午4:01写道：

> +1
>
> On Mon, Jul 22, 2019 at 10:27 AM Biao Liu  wrote:
>
> > +1, make sense to me.
> > Mailing list seems to be a more "community" way.
> >
> > Timo Walther  于2019年7月22日周一 下午4:06写道：
> >
> > > +1 sounds good to inform people about instabilities or other issues
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > > +1. Sounds good.Letting the PR creators know the build results of the
> > > master branch can help to determine more quickly whether Travis
> failures
> > of
> > > their PR are an exact failure or because of the instability of test
> case.
> > > By the way, if the PR creator can abort their own Travis build, I think
> > it
> > > can improve the efficient use of Travis resources (of course, this
> > > discussion does not deal with this issue).
> > > >
> > > >
> > > > Best,
> > > > Haibo
> > > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > > >> +1 sounds good, more people could be encouraged to involve in paying
> > > attention to failure commit.
> > > >>
> > > >> Best
> > > >> Yun Tang
> > > >> 
> > > >> From: Becket Qin 
> > > >> Sent: Monday, July 22, 2019 9:44
> > > >> To: dev 
> > > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list
> > > for travis builds
> > > >>
> > > >> +1. Sounds a good idea to me.
> > > >>
> > > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu 
> > wrote:
> > > >>
> > > >>> Thanks Jark for the proposal, sounds reasonable for me. +1. This ML
> > > could
> > > >>> be used for all the build notifications including master and CRON
> > jobs.
> > > >>>
> > >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > > 
> > >  +1 ,Thanks jark for the proposal.
> > >  Best
> > >  Forward
> > > 
> > >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > > 
> > > > Hi all,
> > > >
> > > > As far as I know, currently, email notifications of Travis builds
> > for
> > > > master branch are sent to the commit author when a build was just
> > > >>> broken or
> > > > still is broken. And there is no email notifications for CRON
> > builds.
> > > >
> > > > Recently, we are suffering from compile errors for scala-2.12 and
> > > java-9
> > > > which are only ran in CRON jobs. So I'm figuring out a way to get
> > > > notifications of CRON builds (or all builds) to quick fix compile
> > > errors
> > > > and failed cron tests.
> > > >
> > > > After reaching out to @Chesnay Schepler 
> > (thanks
> > > >>> for
> > > > the helping), I know that we are using a Slack channel to receive
> > all
> > > > failed build notifications. However, IMO, email notification
> might
> > > be a
> > > > better way than Slack channel to encourage more people to pay
> > > attention
> > > >>> on
> > > > the builds.
> > > >
> > > > So I'm here to propose to setup a bui...@flink.apache.org
> mailing
> > > list
> > > >>> for
> > > > receiving build notifications. I also find that Beam has such
> > mailing
> > > >>> list
> > > > too[1]. After we have such a mailing list, we can integrate it to
> > > travis
> > > > according to the travis doc[2].
> > > >
> > > > What do you think? Do we need a formal vote for this?
> > > >
> > > > Best and thanks,
> > > > Jark
> > > >
> > > > [1]: https://beam.apache.org/community/contact-us/
> > > > [2]:
> > > >
> > > >
> > > >>>
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > <
> > > >
> > > >>>
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > <
> > > >
> > > >>>
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > >>>
> > >
> > >
> >
>

[jira] [Created] (FLINK-13380) Improve the usability of Flink session cluster on Kubernetes

[Yun Tang (JIRA)]

Yun Tang created FLINK-13380:


 Summary: Improve the usability of Flink session cluster on 
Kubernetes
 Key: FLINK-13380
 URL: https://issues.apache.org/jira/browse/FLINK-13380
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: Yun Tang
 Fix For: 1.10.0


Recently, I was launching a Flink session cluster on Kubernetes. However, I 
found current documentation of 
[flink-session-cluster-on-kubernetes|https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html#flink-session-cluster-on-kubernetes]
 and its entrypoint shell script lacks usability:
* flink-conf.yaml is not supported to configure, user have to rebuild docker 
image.
* No explicit documentation of how to submit jobs to session cluster. There 
existed some related question on 
[stackoverflow|https://stackoverflow.com/questions/56910990/flink-on-kubernetes-how-to-submit-jobs-to-session-clusters].
 I think we should add nodeport for 8081 port in deployment yaml.
* Cannot view logs via Flink web UI.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13379) Make Flink work without flink-shaded-hadoop jar

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13379:
--

 Summary: Make Flink work without flink-shaded-hadoop jar
 Key: FLINK-13379
 URL: https://issues.apache.org/jira/browse/FLINK-13379
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.9.0
Reporter: Jeff Zhang


Talk with [~aljoscha] offline, flink should be able to run in yarn mode without 
flink-shaded-hadoop jar but with the native hadoop jar. What we need to do is 
to put that in the flink classpath. 

But I didn't find any document mention about this, this ticket is to verify 
this and add related document. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[Requirement] CI report

[Zili Chen]

Hi,

Currently, our flinkbot updates CI report on status changing.

However, it updates via editing GitHub comment, which would not send
a notification to pr creator once status updated.

Said the "PENDING" status is not quite useful, is it possible that
flinkbot updates a final status(FAILURE/SUCCESS) by adding a new
comment? This will be like hadoop bot updates on JIRA.


Best,
tison.

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Robert Metzger]

+1

On Mon, Jul 22, 2019 at 10:27 AM Biao Liu  wrote:

> +1, make sense to me.
> Mailing list seems to be a more "community" way.
>
> Timo Walther  于2019年7月22日周一 下午4:06写道：
>
> > +1 sounds good to inform people about instabilities or other issues
> >
> > Regards,
> > Timo
> >
> >
> > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > +1. Sounds good.Letting the PR creators know the build results of the
> > master branch can help to determine more quickly whether Travis failures
> of
> > their PR are an exact failure or because of the instability of test case.
> > By the way, if the PR creator can abort their own Travis build, I think
> it
> > can improve the efficient use of Travis resources (of course, this
> > discussion does not deal with this issue).
> > >
> > >
> > > Best,
> > > Haibo
> > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > >> +1 sounds good, more people could be encouraged to involve in paying
> > attention to failure commit.
> > >>
> > >> Best
> > >> Yun Tang
> > >> 
> > >> From: Becket Qin 
> > >> Sent: Monday, July 22, 2019 9:44
> > >> To: dev 
> > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list
> > for travis builds
> > >>
> > >> +1. Sounds a good idea to me.
> > >>
> > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu 
> wrote:
> > >>
> > >>> Thanks Jark for the proposal, sounds reasonable for me. +1. This ML
> > could
> > >>> be used for all the build notifications including master and CRON
> jobs.
> > >>>
> >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > 
> >  +1 ,Thanks jark for the proposal.
> >  Best
> >  Forward
> > 
> >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > 
> > > Hi all,
> > >
> > > As far as I know, currently, email notifications of Travis builds
> for
> > > master branch are sent to the commit author when a build was just
> > >>> broken or
> > > still is broken. And there is no email notifications for CRON
> builds.
> > >
> > > Recently, we are suffering from compile errors for scala-2.12 and
> > java-9
> > > which are only ran in CRON jobs. So I'm figuring out a way to get
> > > notifications of CRON builds (or all builds) to quick fix compile
> > errors
> > > and failed cron tests.
> > >
> > > After reaching out to @Chesnay Schepler 
> (thanks
> > >>> for
> > > the helping), I know that we are using a Slack channel to receive
> all
> > > failed build notifications. However, IMO, email notification might
> > be a
> > > better way than Slack channel to encourage more people to pay
> > attention
> > >>> on
> > > the builds.
> > >
> > > So I'm here to propose to setup a bui...@flink.apache.org mailing
> > list
> > >>> for
> > > receiving build notifications. I also find that Beam has such
> mailing
> > >>> list
> > > too[1]. After we have such a mailing list, we can integrate it to
> > travis
> > > according to the travis doc[2].
> > >
> > > What do you think? Do we need a formal vote for this?
> > >
> > > Best and thanks,
> > > Jark
> > >
> > > [1]: https://beam.apache.org/community/contact-us/
> > > [2]:
> > >
> > >
> > >>>
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > <
> > >
> > >>>
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > <
> > >
> > >>>
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > >>>
> >
> >
>

Re: [DISCUSS] Support temporary tables in SQL API

[Aljoscha Krettek]

I would be fine with option 3) but I think option 2) is the more implicit 
solution that has less surprising behaviour.

Aljoscha

> On 22. Jul 2019, at 23:59, Xuefu Zhang  wrote:
> 
> Thanks to Dawid for initiating the discussion. Overall, I agree with Timo
> that for 1.9 we should have some quick and simple solution, leaving time
> for more thorough discussions for 1.10.
> 
> In particular, I'm not fully with solution #1. For one thing, it seems
> proposing storing all temporary objects in a memory map in CatalogManager,
> and the memory map duplicates the functionality of the in-memory catalog,
> which also store temporary objects. For another, as pointed out by the
> google doc, different db may handle the temporary tables differently, and
> accordingly it may make more sense to let each catalog to handle its
> temporary objects.
> 
> Therefore, postponing the fix buys us time to flush out all the details.
> 
> Thanks,
> Xuefu
> 
> On Mon, Jul 22, 2019 at 7:19 AM Timo Walther  wrote:
> 
>> Thanks for summarizing our offline discussion Dawid! Even though I would
>> prefer solution 1 instead of releasing half-baked features, I also
>> understand that the Table API should not further block the next release.
>> Therefore, I would be fine with solution 3 but introduce the new
>> user-facing `createTemporaryTable` methods as synonyms of the existing
>> ones already. This allows us to deprecate the methods with undefined
>> behavior as early as possible.
>> 
>> Thanks,
>> Timo
>> 
>> 
>> Am 22.07.19 um 16:13 schrieb Dawid Wysakowicz:
>>> Hi all,
>>> 
>>> When working on FLINK-13279[1] we realized we could benefit from a
>>> better temporary objects support in the Catalog API/Table API.
>>> Unfortunately we are already long past the feature freeze that's why I
>>> wanted to get some opinions from the community how should we proceed
>>> with this topic. I tried to prepare a summary of the current state and 3
>>> different suggested approaches that we could take. Please see the
>>> attached document[2]
>>> 
>>> I will appreciate your thoughts!
>>> 
>>> 
>>> [1] https://issues.apache.org/jira/browse/FLINK-13279
>>> 
>>> [2]
>>> 
>> https://docs.google.com/document/d/1RxLj4tDB9GXVjF5qrkM38SKUPkvJt_BSefGYTQ-cVX4/edit?usp=sharing
>>> 
>>> 
>> 
>> 

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Becket Qin]

Congrats, Zhijiang!

On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:

> Congratulations Zhijiang!
>
>
> On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:
>
> > Congratulations Zhijiang!
> >
> > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> >
> > > Congrats, Zhejiang!
> > >
> > >
> > > Best,
> > > Haibo
> > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > >Congratulations Zhijiang, well deserved.
> > > >
> > > >Best
> > > >
> > > >From: Yingjie Cao 
> > > >Sent: Tuesday, July 23, 2019 10:23
> > > >To: dev@flink.apache.org 
> > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to
> > > the Flink project
> > > >
> > > >Congratulations Zhijiang!
> > > >
> > > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > > >
> > > >> Congrats, Zhejiang!
> > > >>
> > > >> Best,
> > > >> Tao Yang
> > > >>
> > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > >> >
> > > >> > Congratulations Zhijiang
> > > >> >
> > > >> > 发自我的 iPhone
> > > >> >
> > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > >> >>
> > > >> >> Congratulations, Zhijiang!
> > > >> >>
> > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> wbeaglewatc...@gmail.com
> > >
> > > >> wrote:
> > > >> >>>
> > > >> >>> Congratulations Zhijiang!
> > > >> >>>
> > > >> >>>
> > > >> >>> Best,
> > > >> >>>
> > > >> >>> Bo WANG
> > > >> >>>
> > > >> >>>
> > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > rmetz...@apache.org>
> > > >> >>> wrote:
> > > >> >>>
> > > >>  Hey all,
> > > >> 
> > > >>  We've added another committer to the Flink project: Zhijiang
> > Wang.
> > > >> 
> > > >>  Congratulations Zhijiang!
> > > >> 
> > > >>  Best,
> > > >>  Robert
> > > >>  (on behalf of the Flink PMC)
> > > >> 
> > > >> >>>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> Xuefu Zhang
> > > >> >>
> > > >> >> "In Honey We Trust!"
> > > >>
> > > >>
> > >
> >
>

[jira] [Created] (FLINK-13378) Wrong comment in SingleValueAggFunction.accumulateExpressions()

[Louis Xu (JIRA)]

Louis Xu created FLINK-13378:


 Summary: Wrong comment in 
SingleValueAggFunction.accumulateExpressions()
 Key: FLINK-13378
 URL: https://issues.apache.org/jira/browse/FLINK-13378
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0, 1.10.0
Reporter: Louis Xu
 Fix For: 1.10.0


1.The comment in the method is "value = count == 0 ? exception : operand(0)", 
but actually it need to be "value = count > 0 ? exception : operand(0)" 
according to the code and logic.

2.And the "throwException" expression's parameter msg is never used. And if we 
only think about "throwException" expression, the param "type" is the return 
type of agg function or msg's type? I don't known the definition of 
"throwException" expression, but I think this might be some problem.

3.If we need use param msg, the expression is call(THROW_EXCEPTION, 
literal(msg, type)) or 

call(THROW_EXCEPTION, literal(msg), typeLiteral(type)).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13377) Streaming SQL e2e test failed on travis

[Zhenghua Gao (JIRA)]

Zhenghua Gao created FLINK-13377:


 Summary: Streaming SQL e2e test failed on travis
 Key: FLINK-13377
 URL: https://issues.apache.org/jira/browse/FLINK-13377
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhenghua Gao
 Fix For: 1.9.0, 1.10.0


This is an instance: [https://api.travis-ci.org/v3/job/562011491/log.txt]

== 
Running 'Streaming SQL end-to-end test' 
== 
TEST_DATA_DIR: 
/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314
 Flink dist directory: 
/home/travis/build/apache/flink/flink-dist/target/flink-1.9-SNAPSHOT-bin/flink-1.9-SNAPSHOT
 Starting cluster. Starting standalonesession daemon on host 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon 
on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Waiting for dispatcher 
REST endpoint to come up... Waiting for dispatcher REST endpoint to come up... 
Waiting for dispatcher REST endpoint to come up... Waiting for dispatcher REST 
endpoint to come up... Waiting for dispatcher REST endpoint to come up... 
Waiting for dispatcher REST endpoint to come up... Dispatcher REST endpoint is 
up. [INFO] 1 instance(s) of taskexecutor are already running on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon 
on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 2 instance(s) 
of taskexecutor are already running on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon 
on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [INFO] 3 instance(s) 
of taskexecutor are already running on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting taskexecutor daemon 
on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Starting execution of 
program Program execution finished Job with JobID 
7c7b66dd4e8dc17e229700b1c746aba6 has finished. Job Runtime: 77371 ms cat: 
'/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/.part-*':
 No such file or directory cat: 
'/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-47211990314/out/result/20/part-*':
 No such file or directory FAIL StreamSQL: Output hash mismatch. Got 
d41d8cd98f00b204e9800998ecf8427e, expected b29f14ed221a936211202ff65b51ee26. 
head hexdump of actual: Stopping taskexecutor daemon (pid: 9983) on host 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping standalonesession 
daemon (pid: 8088) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. 
Skipping taskexecutor daemon (pid: 21571), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 22154), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 22595), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 30622), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 3850), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 4405), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Skipping taskexecutor daemon 
(pid: 4839), because it is not running anymore on 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon 
(pid: 8531) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping 
taskexecutor daemon (pid: 9077) on host 
travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. Stopping taskexecutor daemon 
(pid: 9518) on host travis-job-3ac15c01-1a7d-48b2-b4a9-86575f5d4641. [FAIL] 
Test script contains errors. Checking of logs skipped. [FAIL] 'Streaming SQL 
end-to-end test' failed after 1 minutes and 51 seconds! Test exited with exit 
code 1



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Jark Wu]

Congratulations Zhijiang!


On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:

> Congratulations Zhijiang!
>
> Haibo Sun  于2019年7月23日周二 上午10:48写道：
>
> > Congrats, Zhejiang!
> >
> >
> > Best,
> > Haibo
> > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > >Congratulations Zhijiang, well deserved.
> > >
> > >Best
> > >
> > >From: Yingjie Cao 
> > >Sent: Tuesday, July 23, 2019 10:23
> > >To: dev@flink.apache.org 
> > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to
> > the Flink project
> > >
> > >Congratulations Zhijiang!
> > >
> > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > >
> > >> Congrats, Zhejiang!
> > >>
> > >> Best,
> > >> Tao Yang
> > >>
> > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > >> >
> > >> > Congratulations Zhijiang
> > >> >
> > >> > 发自我的 iPhone
> > >> >
> > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > >> >>
> > >> >> Congratulations, Zhijiang!
> > >> >>
> > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG  >
> > >> wrote:
> > >> >>>
> > >> >>> Congratulations Zhijiang!
> > >> >>>
> > >> >>>
> > >> >>> Best,
> > >> >>>
> > >> >>> Bo WANG
> > >> >>>
> > >> >>>
> > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > rmetz...@apache.org>
> > >> >>> wrote:
> > >> >>>
> > >>  Hey all,
> > >> 
> > >>  We've added another committer to the Flink project: Zhijiang
> Wang.
> > >> 
> > >>  Congratulations Zhijiang!
> > >> 
> > >>  Best,
> > >>  Robert
> > >>  (on behalf of the Flink PMC)
> > >> 
> > >> >>>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Xuefu Zhang
> > >> >>
> > >> >> "In Honey We Trust!"
> > >>
> > >>
> >
>