[RESULT][VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Dian Fu]

Hi all,

The voting time for FLIP-120[1] has passed. I'm closing the vote now.

There are 5 +1 votes, 3 of which are binding:
- Jincheng (binding)
- Hequn (binding)
- Dian Fu (binding)
- Xingbo Huang (non-binding)
- Jeff (non-binding)

There are no disapproving votes and thus FLIP-120 has been accepted.

Thanks a lot for everyone for joining the discussion and votes.

Regards,
Dian

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
 

[jira] [Created] (FLINK-17108) Exception: Cannot determine simple type name "com"

[xiemeilong (Jira)]

xiemeilong created FLINK-17108:
--

 Summary: Exception: Cannot determine simple type name "com"
 Key: FLINK-17108
 URL: https://issues.apache.org/jira/browse/FLINK-17108
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: xiemeilong


The below code will throw exception in cluster mode , but not in local mode or 
when checkpoint disabled.

 
{code:java}
package com.test

import org.apache.flink.streaming.api.scala._
import org.apache.flink.table.api.EnvironmentSettings
import org.apache.flink.table.api.scala.{StreamTableEnvironment, _}
import org.apache.flink.types.Row

case class Test(int:Int)
object Main {
  def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment
  .enableCheckpointing(32 * 1000)

val settings = EnvironmentSettings.newInstance()
  .useBlinkPlanner()
  .inStreamingMode().build()
val tableEnv = StreamTableEnvironment.create(env, settings)

tableEnv.createTemporaryView("test", env.fromCollection(List(Test(1
val deviceSchemaTable = tableEnv.from("test")
deviceSchemaTable.toRetractStream[Row]
.print()
  env.execute("test")
 }
}
{code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Dian Fu]

Hi all,

Thanks a lot for the discussion and votes. I'll summarize the voting results in 
a separate email.

Regards,
Dian

> 在 2020年4月13日，下午2:40，Dian Fu  写道：
> 
> +1 (binding)
> 
> Regards,
> Dian
> 
>> 在 2020年4月13日，下午1:31，Hequn Cheng  写道：
>> 
>> +1 (binding)
>> 
>> Best,
>> Hequn
>> 
>> On Mon, Apr 13, 2020 at 1:28 PM Jeff Zhang  wrote:
>> 
>>> +1
>>> 
>>> 
>>> jincheng sun  于2020年4月13日周一 下午1:24写道：
>>> 
 +1(binding)
 
 Best,
 Jincheng
 
 
 
 Xingbo Huang  于2020年4月9日周四 下午8:27写道：
 
> Hi Dian,
> 
> +1 (non-binding)
> Thanks a lot for driving this.
> 
> Best,
> Xingbo
> 
> Dian Fu  于2020年4月8日周三 上午10:03写道：
> 
>> Hi all,
>> 
>> I'd like to start the vote for FLIP-120[1] which is discussed and
 reached
>> consensus in the discussion thread[2].
>> 
>> The vote will be open for at least 72 hours unless there is an
 objection
>> or we have not received sufficient votes.
>> 
>> Regards,
>> Dian
>> 
>> [1]
>> 
> 
 
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
>> <
>> 
> 
 
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
>>> 
>> [2]
>> 
> 
 
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
>> <
>> 
> 
 
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
>>> 
> 
 
>>> 
>>> 
>>> --
>>> Best Regards
>>> 
>>> Jeff Zhang
>>> 
> 

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Dian Fu]

+1 (binding)

Regards,
Dian

> 在 2020年4月13日，下午1:31，Hequn Cheng  写道：
> 
> +1 (binding)
> 
> Best,
> Hequn
> 
> On Mon, Apr 13, 2020 at 1:28 PM Jeff Zhang  wrote:
> 
>> +1
>> 
>> 
>> jincheng sun  于2020年4月13日周一 下午1:24写道：
>> 
>>> +1(binding)
>>> 
>>> Best,
>>> Jincheng
>>> 
>>> 
>>> 
>>> Xingbo Huang  于2020年4月9日周四 下午8:27写道：
>>> 
 Hi Dian,
 
 +1 (non-binding)
 Thanks a lot for driving this.
 
 Best,
 Xingbo
 
 Dian Fu  于2020年4月8日周三 上午10:03写道：
 
> Hi all,
> 
> I'd like to start the vote for FLIP-120[1] which is discussed and
>>> reached
> consensus in the discussion thread[2].
> 
> The vote will be open for at least 72 hours unless there is an
>>> objection
> or we have not received sufficient votes.
> 
> Regards,
> Dian
> 
> [1]
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> <
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
>> 
> [2]
> 
 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> <
> 
 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
>> 
 
>>> 
>> 
>> 
>> --
>> Best Regards
>> 
>> Jeff Zhang
>> 

[jira] [Created] (FLINK-17107) CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent with StreamConfig#getCheckpointMode()

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-17107:
---

 Summary: CheckpointCoordinatorConfiguration#isExactlyOnce() is 
inconsistent with StreamConfig#getCheckpointMode()
 Key: FLINK-17107
 URL: https://issues.apache.org/jira/browse/FLINK-17107
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Reporter: Yingjie Cao


CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent with 
StreamConfig#getCheckpointMode() when checkpoint is disabled. 
CheckpointCoordinatorConfiguration#isExactlyOnce() returns true if checkpoint 
mode is  EXACTLY_ONCE mode and return false if checkpoint mode is AT_LEAST_ONCE 
while StreamConfig#getCheckpointMode() will always return AT_LEAST_ONCE which 
means always not exactly once.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Generating java code?

[Yang Wang]

Hi Niels,

Thanks a lot for starting this discussion. Although the latter option is
more stable,
i think it is not acceptable for all the developers to execute `mvn
generate-sources`
first. Otherwise, the Flink project is just broken and could not run tests,
Flink jobs
in IDE.

So i think the version properties file is enough for now. +1 for the first
option.

Best,
Yang

Niels Basjes  于2020年4月9日周四 下午4:47写道：

> Hi,
>
> I'm working on https://issues.apache.org/jira/browse/FLINK-16871 to make
> more build time variables (like the scala version) into the code available
> at runtime.
>
> During the review process there was discussion around a basic question: *Is
> generating java code during the build ok?*
> See
>
>- https://github.com/apache/flink/pull/11245#discussion_r400035133
>- https://github.com/apache/flink/pull/11592
>- https://github.com/apache/flink/pull/11592#issuecomment-610282450
>
> As suggested by Chesnay Schepler I'm putting this question to the mailing
> list.
>   https://github.com/apache/flink/pull/11592#issuecomment-610963947
>
> The main discussion was around the ease of use when running in an IDE like
> IntelliJ.
>
> So essentially we have two solution directions available:
>
>1. *Generate a properties file and then use the classloader to load this
>file as a resource and then parse it as a property file.*
>This is the currently used solution direction for this part of the code.
>A rough version of this (to be improved) :
>
> https://github.com/apache/flink/commit/47099f663b7644056e9d87b262cd4dba034f513e
>This method has several effects:
>   1. The developer can run the project immediately from within the IDE
>   as fallback values are provided if the 'actual' values are missing.
>   2. This property file (with stuff that should never be overwritten)
>   can be modified by placing a different one in the classpath. In
> fact it IS
>   modified in the flink-dist as it generates a new file with the same
> name
>   into the binary distribution (I consider this to be bad).
>   3. Loading resources means loading, parsing and a lot of error
>   handling. Lots of things "can be null" or  be a default value. So the
>   values are unreliable and lots of code needs to handle this. In fact
> when
>   running from IntelliJ the properties file is generated poorly most
> of the
>   time, only during a normal maven build will it work correctly.
>2. *Generate a Java source file and then simply compile this and make it
>part of the project.*
>Essentially the same model as you would have when using Apache Avro,
>Protobuf, Antlr 4 and javacc (several of those are used in Flink!).
>A rough version of this (to be improved) :
>
> https://github.com/apache/flink/commit/d215e4df60dc9d647dcee1aa9a2114cbf49d0566
>This method has several effects:
>1. The developer MUST run 'mvn generate-sources' before the actual the
>   project immediately from within the IDE as fallback values are
> provided if
>   the 'actual' values are missing.
>   2. The code/test will not run until this step is done.
>   3. Because the file is generated by a plugin it is always correct. As
>   a consequence all variables are always available and the downstream
> users
>   no longer have to handle the "can be null" or "default value"
> situations.
>
> So is generating code similar to what I created a desired change?
> My opinion is that it is the better solution, the data available is more
> reliable and as a consequence the rest of the code is simpler.
> It would probably mean that during development of flink you should be aware
> of this and do an 'extra step' to get it running.
>
> What do you guys think?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

[jira] [Created] (FLINK-17106) Support create/drop view in Flink SQL

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-17106:


 Summary: Support create/drop view in Flink SQL
 Key: FLINK-17106
 URL: https://issues.apache.org/jira/browse/FLINK-17106
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API, Table SQL / Legacy Planner, Table SQL / 
Planner
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17105) FLIP-71: E2E viewsupport

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-17105:


 Summary: FLIP-71: E2E viewsupport 
 Key: FLINK-17105
 URL: https://issues.apache.org/jira/browse/FLINK-17105
 Project: Flink
  Issue Type: New Feature
Reporter: Zhenghua Gao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-121: Support Cython Optimizing Python User Defined Function

[jincheng sun]

+1(binding)

Best,
Jincheng



Dian Fu  于2020年4月8日周三 上午9:59写道：

> +1 (binding)
>
> > 在 2020年4月8日，上午9:53，Xingbo Huang  写道：
> >
> > Hi all,
> > I would like to start the vote for FLIP-121[1], which is discussed and
> > reached a consensus in the discussion thread[2].
> >
> > The vote will be open for at least 72h, unless there is an objection or
> not
> > enough votes.
> >
> > Best,
> > Xingbo
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-121-Support-Cython-Optimizing-Python-User-Defined-Function-tt39577.html
>
>

Re: [DISCUSS] FLIP-118: Improve Flink’s ID system

[Yangze Guo]

Hi everyone,
After an offline discussion with ZhuZhu, I have some comments on this
investigation.

Regarding the maximum parallelism went from 760 to 685, it may because
of that the tasks are not scheduled evenly. The result is inconsistent
in multiple experiments. So, this phenomenon would be irrelevant to
our changes.

I think what we really care about is the framesize for Akka(Should
user enlarge it after our change for the same job). The size of TDD
after serialization seems to be smaller after change. I don't know the
root reason of this phenomenon at the moment. The way I measure it is:
```
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(deployment);
oos.flush();
LOG.info("BENCHMARK TDD_SERIAL_SIZE {}.", bos.toByteArray().length);
```
Please correct me if I'm wrong.

I'll experiment with higher parallelism to see if there is any
regression regarding Akka framesize.

Regarding the TDD building time, the parallelism in my investigation
might be too small. Meanwhile, this time might be influence by many
factors. Thus, I'll
- experiment with higher parallelism.
- measure the time spent from "Starting scheduling" to the last task
change state to running.

Best,
Yangze Guo


On Fri, Apr 10, 2020 at 12:53 PM Yangze Guo  wrote:
>
> Hi there,
>
> Sorry for the belated reply. I just make a preliminary investigation
> of the infect of refactoring IntermediateResultPartitionID.
>
> The key change is making it being composed of IntermediateDataSetID
> and a partitionNum.
> public class IntermediateResultPartitionID {
>private final IntermediateDataSetID intermediateDataSetID;
>private final int partitionNum;
> }
>
> In this benchmark, I use examples/streaming/WordCount.jar as the test
> job and run Flink on Yarn. All the configurations are kept default
> except for "taskmanager.numberOfTaskSlots", which is set to 2.
>
> The result shows it does have an impact on performance.
> - After this change, the maximum parallelism went from 760 to 685,
> which limited by the total number of network buffers. For the same
> parallelism, user needs more network buffer than before.
> - The netty message "PartitionRequest" and "TaskEventRequest" increase
> by 4 bytes. For "PartitionRequest", it means 7% increase.
> - The TDD takes longer to construct. With 600 parallelisms, it takes
> twice as long to construct TDD than before.
>
> Details record in
> https://docs.google.com/spreadsheets/d/13lt6J29uikWoux79047lkvbGi2fS4ioWHSbvT1kTL2M/edit?usp=sharing
>
> The same issue could happen in ExecutionAttemptID, which will increase
> the "PartitionRequest" and "TaskEventRequest" by 8 bytes(subtaskIndex
> and attemptNumber). But it may not infect the TDD as much as
> IntermediateResultPartitionID, since there is only one
> ExecutionAttemptID per TDD.
>
> After that preliminary investigation, I tend to not refactor
> ExecutionAttemptID and IntermediateResultPartitionID at the moment or
> treat it as future work.
>
> WDYT? @ZhuZhu @Till
>
> Best,
> Yangze Guo
>
> On Wed, Apr 1, 2020 at 11:53 AM Zhu Zhu  wrote:
> >
> > >> However, it seems the JobVertexID is derived from hashcode ...
> > You are right. JobVertexID is widely used and reworking it may affect the
> > public
> > interfaces, e.g. REST api. We can take it as a long term goal and exclude
> > it from this FLIP.
> > This same applies to IntermediateDataSetID, which can be also composed of a
> > JobID
> > and an index as Till proposed.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Till Rohrmann  于2020年3月31日周二 下午8:36写道：
> >
> > > For the IntermediateDataSetID I was just thinking that it might actually 
> > > be
> > > interesting to know which job produced the result which, by using cluster
> > > partitions, could be used across different jobs. Not saying that we have 
> > > to
> > > do it, though.
> > >
> > > A small addition to Zhu Zhu's comment about TDD sizes: For the problem 
> > > with
> > > too large TDDs there is already an issue [1]. The current suspicion is 
> > > that
> > > the size of TDDs for jobs with a large parallelism can indeed become
> > > problematic for Flink. Hence, it would be great to investigate the impacts
> > > of the proposed changes.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-16069
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Mar 31, 2020 at 11:50 AM Yangze Guo  wrote:
> > >
> > > > Hi, Zhu,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > > make JobVertexID a composition of JobID and a topology index
> > > > I think it is a good idea. However, it seems the JobVertexID is
> > > > derived from hashcode which used to identify them across submission.
> > > > I'm not familiar with that component though. I prefer to keep this
> > > > idea out the scope of this FLIP if no one could help us to figure it
> > > > out.
> > > >
> > > > > How about we still keep IntermediateDataSetID independent from
> > > > JobVertexID,
> > > > > but just print the producing re

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Mon, Apr 13, 2020 at 1:28 PM Jeff Zhang  wrote:

> +1
>
>
> jincheng sun  于2020年4月13日周一 下午1:24写道：
>
> > +1(binding)
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Xingbo Huang  于2020年4月9日周四 下午8:27写道：
> >
> > > Hi Dian,
> > >
> > > +1 (non-binding)
> > > Thanks a lot for driving this.
> > >
> > > Best,
> > > Xingbo
> > >
> > > Dian Fu  于2020年4月8日周三 上午10:03写道：
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for FLIP-120[1] which is discussed and
> > reached
> > > > consensus in the discussion thread[2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection
> > > > or we have not received sufficient votes.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > >
> > > > [2]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > <
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Jeff Zhang]

+1


jincheng sun  于2020年4月13日周一 下午1:24写道：

> +1(binding)
>
> Best,
> Jincheng
>
>
>
> Xingbo Huang  于2020年4月9日周四 下午8:27写道：
>
> > Hi Dian,
> >
> > +1 (non-binding)
> > Thanks a lot for driving this.
> >
> > Best,
> > Xingbo
> >
> > Dian Fu  于2020年4月8日周三 上午10:03写道：
> >
> > > Hi all,
> > >
> > > I'd like to start the vote for FLIP-120[1] which is discussed and
> reached
> > > consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > > or we have not received sufficient votes.
> > >
> > > Regards,
> > > Dian
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > >
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > <
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > >
> >
>


-- 
Best Regards

Jeff Zhang

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[jincheng sun]

+1(binding)

Best,
Jincheng



Xingbo Huang  于2020年4月9日周四 下午8:27写道：

> Hi Dian,
>
> +1 (non-binding)
> Thanks a lot for driving this.
>
> Best,
> Xingbo
>
> Dian Fu  于2020年4月8日周三 上午10:03写道：
>
> > Hi all,
> >
> > I'd like to start the vote for FLIP-120[1] which is discussed and reached
> > consensus in the discussion thread[2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> > or we have not received sufficient votes.
> >
> > Regards,
> > Dian
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > >
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > <
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > >
>

[jira] [Created] (FLINK-17104) Support registering custom JobStatusListeners from config

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17104:


 Summary: Support registering custom JobStatusListeners from config
 Key: FLINK-17104
 URL: https://issues.apache.org/jira/browse/FLINK-17104
 Project: Flink
  Issue Type: New Feature
  Components: API / Core
Reporter: Canbin Zheng


Currently, a variety of users are asking for registering custom 
JobStatusListener support to get timely feedback on the status transition of 
the jobs. This could be an important feature for effective Flink cluster 
monitoring systems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[RESULT][VOTE] FLIP-113: Support dynamic table options for Flink SQL

[Danny Chan]

Hi all,

The voting time for FLIP-113 has passed. I'm closing the vote now.

There were 5 +1 votes, 4 of which are binding:

- Timo (binding)
- Dawid (binding)
- Jark (binding)
- Kurt (binding)
- Benchao Li (non-binding)

There were no disapproving votes.

Thus, FLIP-113 has been accepted.

Thanks everyone for joining the discussion and giving feedback!

Best,
Danny Chan

[jira] [Created] (FLINK-17103) Supports dynamic table options for Blink planner

[Danny Chen (Jira)]

Danny Chen created FLINK-17103:
--

 Summary: Supports dynamic table options for Blink planner
 Key: FLINK-17103
 URL: https://issues.apache.org/jira/browse/FLINK-17103
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Danny Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17102) Add -Dkubernetes.container.image= for the start-flink-session section

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17102:


 Summary: Add -Dkubernetes.container.image= for the 
start-flink-session section
 Key: FLINK-17102
 URL: https://issues.apache.org/jira/browse/FLINK-17102
 Project: Flink
  Issue Type: Sub-task
Reporter: Canbin Zheng


Add {{-Dkubernetes.container.image=}} as a guide for new users in 
the existing command:
{quote}{{}}
 
{{./bin/kubernetes-session.sh \}}

{{-Dkubernetes.cluster-id= \-Dtaskmanager.memory.process.size=4096m 
\-Dkubernetes.taskmanager.cpu=2 \-Dtaskmanager.numberOfTaskSlots=4 
\-Dresourcemanager.taskmanager-timeout=360}}{{}}
{quote}
Details could refer to 
[https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html#start-flink-session]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17101) Supports dynamic table options for Flink SQL

[Danny Chen (Jira)]

Danny Chen created FLINK-17101:
--

 Summary: Supports dynamic table options for Flink SQL
 Key: FLINK-17101
 URL: https://issues.apache.org/jira/browse/FLINK-17101
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: Danny Chen
 Fix For: 1.11.0


Supports syntax:

{code:sql}
... table /*+ OPTIONS('k1' = 'v1', 'k2' = 'v2') */
{code}

to specify dynamic options within the scope of the appended table. The dynamic 
options would override the static options defined in the CREATE TABLE DDL or 
connector API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17100) Document Native Kubernetes Improvements

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-17100:


 Summary: Document Native Kubernetes Improvements
 Key: FLINK-17100
 URL: https://issues.apache.org/jira/browse/FLINK-17100
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes, Documentation
Reporter: Canbin Zheng


This is the umbrella issue for the native Kubernetes documentation improvements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17099) Refactoring State TTL solution in Group Agg、TopN operator, etc replace Timer with StateTtlConfig

[dalongliu (Jira)]

dalongliu created FLINK-17099:
-

 Summary: Refactoring State TTL solution in Group Agg、TopN 
operator, etc replace Timer with StateTtlConfig
 Key: FLINK-17099
 URL: https://issues.apache.org/jira/browse/FLINK-17099
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Affects Versions: 1.10.0, 1.9.0
Reporter: dalongliu
 Fix For: 1.11.0


At the moment, there are 2 ways to cleanup states.

1) registering a processing-time timer, and cleanup entries when the timer is 
callback.
 - pros: can cleanup multiple states at the same time (state consistent)
 - cons: timer space depends on the key size, which may lead to OOM (heap 
timer).
 - used in Group Aggregation, Over Aggregation, TopN

2) using the {{StateTtlConfig}} provided by DataStream [1].
 - pros: decouple the logic of state ttl with the record processing, easy to 
program (take a look at old planner NonWindowJoin which bundles ttl timestamp 
with records in MapState).
 - cons: can't cleanup multiple states at the same time.
 - useed in Sream-Stream Joins.

For timer solution, although it can cleanup multiple states at the same time, 
but it also will lead to OOM when there have a great many state keys, besides, 
StateTtlConfig is used in stream-stream join case, and will be used in more 
operator. Therefore，in order to unify the state ttl solution, simplify the code 
implemention, and improve the readability of codes, so we should refactor state 
cleanup way which use StateTtlConfig to replace processing-time timer in Group 
Aggregation、Over Aggregation、TopN operator, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17098) CatalogManager#dropTemporaryTable and dropTemporaryView should use ObjectIdentifier as its argument

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-17098:


 Summary: CatalogManager#dropTemporaryTable and dropTemporaryView 
should use ObjectIdentifier as its argument
 Key: FLINK-17098
 URL: https://issues.apache.org/jira/browse/FLINK-17098
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Zhenghua Gao
 Fix For: 1.11.0


Since CatalogManager#createTable, createTemporaryTable and dropTable use the 
given 

fully qualified ObjectIdentifier to create or drop tables/temporary tables, we 
should also use ObjectIdentifier (instead of UnresolvedIdentifier) in 
dropTemporaryTable and dropTemporaryView. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[Zhenghua Gao]

Hi everyone,

Thanks for the votes. So far, we have
- 3 binding +1 votes (Timo, Jingsong, Jark)
- 4 non-binding +1 votes (Danny, zoudan, Benchao, godfrey)
- no -1 vote

The voting time has past and there is enough +1 votes to consider the
FLIP-71 approved.
Thanks you all.

*Best Regards,*
*Zhenghua Gao*


On Mon, Apr 13, 2020 at 10:30 AM Jark Wu  wrote:

> +1
>
> Best,
> Jark
>
> On Sun, 12 Apr 2020 at 12:28, Benchao Li  wrote:
>
> > +1 (non-binding)
> >
> > zoudan  于2020年4月12日周日 上午9:52写道：
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Dan Zou
> > >
> > >
> > > > 在 2020年4月10日，09:30，Danny Chan  写道：
> > > >
> > > > +1 from my side.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2020年4月9日 +0800 PM9:23，Timo Walther ，写道：
> > > >> +1 (binding)
> > > >>
> > > >> Thanks for your efforts.
> > > >>
> > > >> Regards,
> > > >> Timo
> > > >>
> > > >>
> > > >> On 09.04.20 14:46, Zhenghua Gao wrote:
> > > >>> Hi all,
> > > >>>
> > > >>> I'd like to start the vote for FLIP-71[1] which adds E2E view
> support
> > > in
> > > >>> Flink SQL.
> > > >>> This FLIP is discussed in the thread[2].
> > > >>>
> > > >>> The vote will be open for at least 72 hours. Unless there is an
> > > objection.
> > > >>> I will try to
> > > >>> close it by April 13, 2020 09:00 UTC if we have received sufficient
> > > votes.
> > > >>>
> > > >>> [1]
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL
> > > >>>
> > > >>> [2]
> > > >>>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787
> > > >>>
> > > >>> *Best Regards,*
> > > >>> *Zhenghua Gao*
> > > >>>
> > > >>
> > >
> > >
> >
> > --
> >
> > Benchao Li
> > School of Electronics Engineering and Computer Science, Peking University
> > Tel:+86-15650713730
> > Email: libenc...@gmail.com; libenc...@pku.edu.cn
> >
>

[jira] [Created] (FLINK-17097) Flink HBase Connector String field size at least equal 8

[xingoo (Jira)]

xingoo created FLINK-17097:
--

 Summary: Flink HBase Connector String field size at least equal 8
 Key: FLINK-17097
 URL: https://issues.apache.org/jira/browse/FLINK-17097
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.10.0
Reporter: xingoo


when using string field in hbase connector, the rowkey length at least 8, 
becuase byte[] size at least 8.

example:
{code:java}
//代码占位符
rowkey: "1"
{code}
when using it as lookup function:
{code:java}
//代码占位符
Caused by: java.lang.IllegalArgumentException: offset (0) + length (8) exceed 
the capacity of the array: 1
at 
org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:779)
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:753)
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:726)
at 
org.apache.flink.addons.hbase.util.HBaseTypeUtils.deserializeToObject(HBaseTypeUtils.java:57)
at 
org.apache.flink.addons.hbase.util.HBaseReadWriteHelper.parseToRow(HBaseReadWriteHelper.java:158)
at 
org.apache.flink.addons.hbase.HBaseLookupFunction.eval(HBaseLookupFunction.java:78)
at StreamExecCorrelate$144.processElement(Unknown Source)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
at StreamExecCalc$134.processElement(Unknown Source)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
at SourceConversion$127.processElement(Unknown Source)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:310)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:409)
at 
org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at 
org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:91)
at 
org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:156)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:715)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:196)

{code}
trace code:
{code:java}
//代码占位符
@Internal
public class HBaseTypeUtils {

 private static final byte[] EMPTY_BYTES = new byte[]{};

 /**
  * Deserialize byte array to Java Object with the given type.
  */
 public static Object deserializeToObject(byte[] value, int typeIdx, Charset 
stringCharset) {
  switch (typeIdx) {
   case 0: // byte[]
return value;
   case 1: // String
return new String(value, stringCharset);

...

public String(byte bytes[], Charset charset) {
this(bytes, 0, bytes.length, charset);
}{code}



--
This mes

[jira] [Created] (FLINK-17096) Minibatch Group Agg support state ttl

[dalongliu (Jira)]

dalongliu created FLINK-17096:
-

 Summary: Minibatch Group Agg support state ttl
 Key: FLINK-17096
 URL: https://issues.apache.org/jira/browse/FLINK-17096
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Runtime
Affects Versions: 1.10.0, 1.9.0
Reporter: dalongliu
 Fix For: 1.11.0


At the moment, MiniBatch Group Agg include Local/Global doesn`t support State 
TTL, for streaming job, it will lead to OOM in long time running, so we need to 
make state data expire after ttl, the solution is that use incremental cleanup 
feature refer to [link 
FLINK-16581|https://issues.apache.org/jira/browse/FLINK-16581]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[Jark Wu]

+1

Best,
Jark

On Sun, 12 Apr 2020 at 12:28, Benchao Li  wrote:

> +1 (non-binding)
>
> zoudan  于2020年4月12日周日 上午9:52写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Dan Zou
> >
> >
> > > 在 2020年4月10日，09:30，Danny Chan  写道：
> > >
> > > +1 from my side.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2020年4月9日 +0800 PM9:23，Timo Walther ，写道：
> > >> +1 (binding)
> > >>
> > >> Thanks for your efforts.
> > >>
> > >> Regards,
> > >> Timo
> > >>
> > >>
> > >> On 09.04.20 14:46, Zhenghua Gao wrote:
> > >>> Hi all,
> > >>>
> > >>> I'd like to start the vote for FLIP-71[1] which adds E2E view support
> > in
> > >>> Flink SQL.
> > >>> This FLIP is discussed in the thread[2].
> > >>>
> > >>> The vote will be open for at least 72 hours. Unless there is an
> > objection.
> > >>> I will try to
> > >>> close it by April 13, 2020 09:00 UTC if we have received sufficient
> > votes.
> > >>>
> > >>> [1]
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL
> > >>>
> > >>> [2]
> > >>>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787
> > >>>
> > >>> *Best Regards,*
> > >>> *Zhenghua Gao*
> > >>>
> > >>
> >
> >
>
> --
>
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenc...@gmail.com; libenc...@pku.edu.cn
>

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

[godfrey he]

+1 (non-binding)

Best,
Godfrey

Benchao Li  于2020年4月12日周日 下午12:28写道：

> +1 (non-binding)
>
> zoudan  于2020年4月12日周日 上午9:52写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Dan Zou
> >
> >
> > > 在 2020年4月10日，09:30，Danny Chan  写道：
> > >
> > > +1 from my side.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2020年4月9日 +0800 PM9:23，Timo Walther ，写道：
> > >> +1 (binding)
> > >>
> > >> Thanks for your efforts.
> > >>
> > >> Regards,
> > >> Timo
> > >>
> > >>
> > >> On 09.04.20 14:46, Zhenghua Gao wrote:
> > >>> Hi all,
> > >>>
> > >>> I'd like to start the vote for FLIP-71[1] which adds E2E view support
> > in
> > >>> Flink SQL.
> > >>> This FLIP is discussed in the thread[2].
> > >>>
> > >>> The vote will be open for at least 72 hours. Unless there is an
> > objection.
> > >>> I will try to
> > >>> close it by April 13, 2020 09:00 UTC if we have received sufficient
> > votes.
> > >>>
> > >>> [1]
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL
> > >>>
> > >>> [2]
> > >>>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-71-E2E-View-support-in-Flink-SQL-td33131.html#a39787
> > >>>
> > >>> *Best Regards,*
> > >>> *Zhenghua Gao*
> > >>>
> > >>
> >
> >
>
> --
>
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenc...@gmail.com; libenc...@pku.edu.cn
>

Re: [VOTE] FLIP-108: Add GPU support in Flink

[Xintong Song]

Thanks for updating the FLIP, Yangze.
The latest FLIP looks good to me.

nit: Javadoc of `ExternalResourceDriver#retrieveResourceInfo` is out of
sync.

> Retrieve the information of the external resources according to the
> resourceProfile.


Thank you~

Xintong Song



On Sat, Apr 11, 2020 at 11:04 AM Becket Qin  wrote:

> Good feedback form Xintong. The latest FLIP looks good to me.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sat, Apr 11, 2020 at 9:20 AM Yangze Guo  wrote:
>
> > Hi there,
> > I've updated the FLIP accordingly. Please take a look. If you have any
> > further concerns please let me know.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Apr 10, 2020 at 6:40 PM Yangze Guo  wrote:
> > >
> > > Thanks for the feedback, Xintong.
> > >
> > > - Should we have a factory interface for `ExternalResourceDriver`, that
> > > takes the configuration and returns a driver instance? Otherwise, if we
> > are
> > > creating the driver instance with reflection, we kind of implicitly
> > > requires the driver to have a public non-argument constructor. If we
> > > decided to go with this approach, then we will not need
> > > `ExternalResourceDriver#open`.
> > >
> > > True, we could have an `ExternalResourceDriverFactory`, like
> > > interface ExternalResourceDriverFactory {
> > > ExternalResourceDriver fromConfiguration(Configuration config);
> > > }
> > > Regarding the configuration, the user should provide
> > > "external-resource.{resourceName}.driver-factory.class" instead.
> > >
> > > - Not sure about the necessity of `ExternalResourceDriver#close`. I
> would
> > > suggest to avoid introduce more interfaces if not absolutely necessary.
> > >
> > > I add `ExternalResourceDriver#close` in case user needs to clean up
> > > internal states and any other resources. It's true that it may not
> > > absolutely necessary for our GPUDriver. From my side, I'm ok to remove
> > > it.
> > >
> > > - `ExternalResourceDriver#retrieveResourceInfo` should not take
> > > `ResourceProfile` as argument. This exposes more information than it
> > needs.
> > > In addition, it requires the runtime/core to understand how to properly
> > > wrap the external resource into `ResourceProfile`. E.g.,
> > > `ResourceProfile#extendedResources` takes `Resource`, which is an
> > abstract
> > > class. Runtime/core has to known which implementation of `Resource` to
> > use.
> > >
> > > True, at the moment, I think the amount of the resource is enough for
> > > the `ExternalResourceDriver#retrieveResourceInfo`. In the future, if
> > > the fine-grained external resource management is supported, the amount
> > > of the resource seems to be enough either. If we want to leverage some
> > > external resources which could not be measured by a single long value,
> > > we might enrich this. But I'd like to keep it out of the scope of this
> > > FLIP.
> > >
> > > - Do we really need `ExternalResourceInfo#getInformation`? I think it
> > > should be good enough to make `ExternalResourceInfo` an empty
> interface.
> > > User can define their own `ExternalResourceInfo` implementation and how
> > it
> > > is used by the operator user codes.
> > >
> > > Sounds good.
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Fri, Apr 10, 2020 at 6:04 PM Xintong Song 
> > wrote:
> > > >
> > > > Sorry to pull this back. I have some concerns about the recent
> updated
> > > > interface details.
> > > >
> > > > - Should we have a factory interface for `ExternalResourceDriver`,
> that
> > > > takes the configuration and returns a driver instance? Otherwise, if
> > we are
> > > > creating the driver instance with reflection, we kind of implicitly
> > > > requires the driver to have a public non-argument constructor. If we
> > > > decided to go with this approach, then we will not need
> > > > `ExternalResourceDriver#open`.
> > > > - Not sure about the necessity of `ExternalResourceDriver#close`. I
> > would
> > > > suggest to avoid introduce more interfaces if not absolutely
> necessary.
> > > > - `ExternalResourceDriver#retrieveResourceInfo` should not take
> > > > `ResourceProfile` as argument. This exposes more information than it
> > needs.
> > > > In addition, it requires the runtime/core to understand how to
> properly
> > > > wrap the external resource into `ResourceProfile`. E.g.,
> > > > `ResourceProfile#extendedResources` takes `Resource`, which is an
> > abstract
> > > > class. Runtime/core has to known which implementation of `Resource`
> to
> > use.
> > > > - Do we really need `ExternalResourceInfo#getInformation`? I think it
> > > > should be good enough to make `ExternalResourceInfo` an empty
> > interface.
> > > > User can define their own `ExternalResourceInfo` implementation and
> > how it
> > > > is used by the operator user codes.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Thu, Apr 9, 2020 at 2:25 PM Becket Qin 
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Thanks for driving this effort, Ynagze. The la

[jira] [Created] (FLINK-17095) KafkaProducerExactlyOnceITCase fails with "address already in use"

[Zhijiang (Jira)]

Zhijiang created FLINK-17095:


 Summary: KafkaProducerExactlyOnceITCase fails with "address 
already in use"
 Key: FLINK-17095
 URL: https://issues.apache.org/jira/browse/FLINK-17095
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Tests
Reporter: Zhijiang
 Fix For: 1.11.0


Logs: [https://travis-ci.org/github/apache/flink/jobs/673786814]
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.256 s 
<<< FAILURE! - in 
org.apache.flink.streaming.connectors.kafka.KafkaProducerExactlyOnceITCase
[ERROR] 
org.apache.flink.streaming.connectors.kafka.KafkaProducerExactlyOnceITCase  
Time elapsed: 7.256 s  <<< ERROR!
org.apache.kafka.common.KafkaException: Socket server failed to bind to 
0.0.0.0:42733: Address already in use.
at kafka.network.Acceptor.openServerSocket(SocketServer.scala:573)
at kafka.network.Acceptor.(SocketServer.scala:451)
at kafka.network.SocketServer.createAcceptor(SocketServer.scala:245)
at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:215)
at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:214)
at 
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at 
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:214)
at kafka.network.SocketServer.startup(SocketServer.scala:114)
at kafka.server.KafkaServer.startup(KafkaServer.scala:253)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.getKafkaServer(KafkaTestEnvironmentImpl.java:404)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.prepare(KafkaTestEnvironmentImpl.java:131)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.startClusters(KafkaTestBase.java:142)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.startClusters(KafkaTestBase.java:131)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.prepare(KafkaTestBase.java:100)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.prepare(KafkaTestBase.java:92)
at 
org.apache.flink.streaming.connectors.kafka.KafkaProducerExactlyOnceITCase.prepare(KafkaProducerExactlyOnceITCase.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
at kafka.network.Acceptor.openServerSocket(SocketServer.s

[jira] [Created] (FLINK-17094) OverWindowITCase#testRowTimeBoundedPartitionedRowsOver failed by FileNotFoundException

[Zhijiang (Jira)]

Zhijiang created FLINK-17094:


 Summary: OverWindowITCase#testRowTimeBoundedPartitionedRowsOver 
failed by FileNotFoundException
 Key: FLINK-17094
 URL: https://issues.apache.org/jira/browse/FLINK-17094
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends, Tests
Reporter: Zhijiang
 Fix For: 1.11.0


Build: [https://travis-ci.org/github/apache/flink/jobs/673786805]

logs
{code:java}
[ERROR] 
testRowTimeBoundedPartitionedRowsOver[StateBackend=ROCKSDB](org.apache.flink.table.planner.runtime.stream.sql.OverWindowITCase)
  Time elapsed: 0.754 s  <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
at 
org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:659)
at 
org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1643)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1625)
at 
org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:673)
at 
org.apache.flink.table.planner.runtime.stream.sql.OverWindowITCase.testRowTimeBoundedPartitionedRowsOver(OverWindowITCase.scala:417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
   

Re: [DISCUSS] Releasing Flink 1.10.1

[Yu Li]

Thanks Weike and all others for the efforts!

Here comes the latest status, we are in good shape and plan to produce RC1
next week.

* Blockers (1 left)
  - [Closed] FLINK-16018 Improve error reporting when submitting batch job
(instead of AskTimeoutException)
  - [Closed] FLINK-16142 Memory Leak causes Metaspace OOM error on repeated
job submission
  - [Closed] FLINK-16170 SearchTemplateRequest ClassNotFoundException when
use flink-sql-connector-elasticsearch7
  - [Closed] FLINK-16262 Class loader problem with
FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
  - [Closed] FLINK-16406 Increase default value for JVM Metaspace to
minimise its OutOfMemoryError
  - [Closed] FLINK-16454 Update the copyright year in NOTICE files
  - [Closed] FLINK-16705 LocalExecutor tears down MiniCluster before client
can retrieve JobResult
  - [Closed] FLINK-16913 ReadableConfigToConfigurationAdapter#getEnum
throws UnsupportedOperationException
  - [Closed] FLINK-16626 Exception encountered when cancelling a job in
yarn per-job mode
  - [Fix for 1.10.1 is Done] FLINK-17093 Python UDF doesn't work when the
input column is of composite type
  - [PR reviewed] FLINK-16576 State inconsistency on restore with memory
state backends

* Critical (1 left)
  - [Closed] FLINK-16047 Blink planner produces wrong aggregate results
with state clean up
  - [Closed] FLINK-16070 Blink planner can not extract correct unique key
for UpsertStreamTableSink
  - [Fix for 1.10.1 is Done] FLINK-16225 Metaspace Out Of Memory should be
handled as Fatal Error in TaskManager
  - [Closed] FLINK-14316 stuck in "Job leader ... lost leadership" error
  - [May Postpone] FLINK-16408 Bind user code class loader to lifetime of a
slot

Please let me know if any concerns/comments. Thanks.

Best Regards,
Yu


On Fri, 3 Apr 2020 at 21:35, DONG, Weike  wrote:

> Hi Yu,
>
> Thanks for your updates. I am still working on the fix for FLINK-16626 and
> it is expected to be completed by this Sunday after thorough testing.
>
> Sincerely,
> Weike
>
> On Fri, Apr 3, 2020 at 8:43 PM Yu Li  wrote:
>
>> Updates for 1.10.1 watched issues (we are in good progress and almost
>> there
>> to produce the first RC, thanks all for the efforts):
>>
>> * Blockers (3 left)
>>   - [Closed] FLINK-16018 Improve error reporting when submitting batch job
>> (instead of AskTimeoutException)
>>   - [Closed] FLINK-16142 Memory Leak causes Metaspace OOM error on
>> repeated
>> job submission
>>   - [Closed] FLINK-16170 SearchTemplateRequest ClassNotFoundException when
>> use flink-sql-connector-elasticsearch7
>>   - [Closed] FLINK-16262 Class loader problem with
>> FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
>>   - [Closed] FLINK-16406 Increase default value for JVM Metaspace to
>> minimise its OutOfMemoryError
>>   - [Closed] FLINK-16454 Update the copyright year in NOTICE files
>>   - [PR reviewed] FLINK-16576 State inconsistency on restore with memory
>> state backends
>>   - [Under Discussion] FLINK-16626 Exception encountered when cancelling a
>> job in yarn per-job mode
>>   - [Closed] FLINK-16705 LocalExecutor tears down MiniCluster before
>> client
>> can retrieve JobResult
>>   - [PR reviewed] FLINK-16913 ReadableConfigToConfigurationAdapter#getEnum
>> throws UnsupportedOperationException
>>
>> * Critical (1 left)
>>   - [Closed] FLINK-14316 stuck in "Job leader ... lost leadership" error
>>   - [Closed] FLINK-16047 Blink planner produces wrong aggregate results
>> with state clean up
>>   - [Closed] FLINK-16070 Blink planner can not extract correct unique key
>> for UpsertStreamTableSink
>>   - [Fix for 1.10.1 is Done] FLINK-16225 Metaspace Out Of Memory should be
>> handled as Fatal Error in TaskManager
>>   - [May Postpone] FLINK-16408 Bind user code class loader to lifetime of
>> a
>> slot
>>
>> Please let me know if you find any missing ones, thanks.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Fri, 27 Mar 2020 at 21:06, Yu Li  wrote:
>>
>> > Here comes the latest status of issues in 1.10.1 watch list:
>> >
>> > * Blockers (4 left)
>> >   - [Under Discussion] FLINK-16018 Improve error reporting when
>> submitting
>> > batch job (instead of AskTimeoutException)
>> >   - [Closed] FLINK-16142 Memory Leak causes Metaspace OOM error on
>> > repeated job submission
>> >   - [Closed] FLINK-16170 SearchTemplateRequest ClassNotFoundException
>> when
>> > use flink-sql-connector-elasticsearch7
>> >   - [PR Approved] FLINK-16262 Class loader problem with
>> > FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
>> >   - [Closed] FLINK-16406 Increase default value for JVM Metaspace to
>> > minimise its OutOfMemoryError
>> >   - [Closed] FLINK-16454 Update the copyright year in NOTICE files
>> >   - [In Progress] FLINK-16576 State inconsistency on restore with memory
>> > state backends
>> >   - [New] [PR Submitted] FLINK-16705 LocalExecutor tears down
>> MiniCluster
>> > before client can retrieve JobResult
>> >
>> > * Critical (2 left)
>> >   - [Closed] FLINK-16047