Re: [RESULT] [VOTE] Apache Flink 1.9.0, release candidate #3

[Shaoxuan Wang]

Congratulations and thanks all for the great efforts on release1.9.

I have verified the RC#3 with the following items:

- Verified signatures and hashes. (OK)
- Built from source archive. (OK)
- Repository contains all artifacts. (OK)
- Test WordCount on local cluster. (OK)
a. Both streaming and batch
b. Web ui works fine
- Test WordCount on yarn cluster. (OK)
a. Both streaming and batch
b. Web ui works fine
c. Test session mode and non-session mode

So +1 (binding) from my side.

Regards,
Shaoxuan


On Thu, Aug 22, 2019 at 1:23 AM Tzu-Li (Gordon) Tai 
wrote:

> I'm happy to announce that we have unanimously approved this candidate as
> the 1.9.0 release.
>
> There are 12 approving votes, 5 of which are binding:
> - Yu Li
> - Zili Chen
> - Gordon Tai
> - Stephan Ewen
> - Jark Wu
> - Vino Yang
> - Gary Yao
> - Bowen Li
> - Chesnay Schepler
> - Till Rohrmann
> - Aljoscha Krettek
> - David Anderson
>
> There are no disapproving votes.
>
> Thanks everyone who has contributed to this release!
>
> I will wait until tomorrow morning for the artifacts to be available in
> Maven central before announcing the release in a separate thread.
>
> The release blog post will also be merged tomorrow along with the official
> announcement.
>
> Cheers,
> Gordon
>
> On Wed, Aug 21, 2019, 5:37 PM David Anderson  wrote:
>
> > +1 (non-binding)
> >
> > I upgraded the flink-training-exercises project.
> >
> > I encountered a few rough edges, including problems in the docs, but
> > nothing serious.
> >
> > I had to make some modifications to deal with changes in the Table API:
> >
> > ExternalCatalogTable.builder became new ExternalCatalogTableBuilder
> > TableEnvironment.getTableEnvironment became StreamTableEnvironment.create
> > StreamTableDescriptorValidator.UPDATE_MODE() became
> > StreamTableDescriptorValidator.UPDATE_MODE
> > org.apache.flink.table.api.java.Slide moved to
> > org.apache.flink.table.api.Slide
> >
> > I also found myself forced to change a CoProcessFunction to a
> > KeyedCoProcessFunction (which it should have been).
> >
> > I also tried a few complex queries in the SQL console, and wrote a
> > simple job using the State Processor API. Everything worked.
> >
> > David
> >
> >
> > David Anderson | Training Coordinator
> >
> > Follow us @VervericaData
> >
> > --
> > Join Flink Forward - The Apache Flink Conference
> > Stream Processing | Event Driven | Real Time
> >
> >
> > On Wed, Aug 21, 2019 at 1:45 PM Aljoscha Krettek 
> > wrote:
> > >
> > > +1
> > >
> > > I checked the last RC on a GCE cluster and was satisfied with the
> > testing. The cherry-picked commits didn’t change anything related, so I’m
> > forwarding my vote from there.
> > >
> > > Aljoscha
> > >
> > > > On 21. Aug 2019, at 13:34, Chesnay Schepler 
> > wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > On 21/08/2019 08:09, Bowen Li wrote:
> > > >> +1 non-binding
> > > >>
> > > >> - built from source with default profile
> > > >> - manually ran SQL and Table API tests for Flink's metadata
> > integration
> > > >> with Hive Metastore in local cluster
> > > >> - manually ran SQL tests for batch capability with Blink planner and
> > Hive
> > > >> integration (source/sink/udf) in local cluster
> > > >> - file formats include: csv, orc, parquet
> > > >>
> > > >>
> > > >> On Tue, Aug 20, 2019 at 10:23 PM Gary Yao 
> wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> Reran Jepsen tests 10 times.
> > > >>>
> > > >>> On Wed, Aug 21, 2019 at 5:35 AM vino yang 
> > wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  - checkout source code and build successfully
> > >  - started a local cluster and ran some example jobs successfully
> > >  - verified signatures and hashes
> > >  - checked release notes and post
> > > 
> > >  Best,
> > >  Vino
> > > 
> > >  Stephan Ewen  于2019年8月21日周三 上午4:20写道：
> > > 
> > > > +1 (binding)
> > > >
> > > >  - Downloaded the binary release tarball
> > > >  - started a standalone cluster with four nodes
> > > >  - ran some examples through the Web UI
> > > >  - checked the logs
> > > >  - created a project from the Java quickstarts maven archetype
> > > >  - ran a multi-stage DataSet job in batch mode
> > > >  - killed as TaskManager and verified correct restart behavior,
> > > >>> including
> > > > failover region backtracking
> > > >
> > > >
> > > > I found a few issues, and a common theme here is confusing error
> > >  reporting
> > > > and logging.
> > > >
> > > > (1) When testing batch failover and killing a TaskManager, the
> job
> > >  reports
> > > > as the failure cause "org.apache.flink.util.FlinkException: The
> > > >>> assigned
> > > > slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
> > > > I think that is a pretty bad error message, as a user I don't
> > know
> > >  what
> > > > that means. Some internal book keeping thing?
> > > 

Re: [DISCUSS] Flink client api enhancement for downstream project

[Yang Wang]

>From the user's perspective, it is really confused about the scope of
per-job cluster.


If it means a flink cluster with single job, so that we could get better
isolation.

Now it does not matter how we deploy the cluster, directly deploy(mode1)

or start a flink cluster and then submit job through cluster client(mode2).


Otherwise, if it just means directly deploy, how should we name the mode2,

session with job or something else?

We could also benefit from the mode2. Users could get the same isolation
with mode1.

The user code and dependencies will be loaded by user class loader

to avoid class conflict with framework.



Anyway, both of the two submission modes are useful.

We just need to clarify the concepts.




Best,

Yang

Zili Chen  于2019年8月20日周二 下午5:58写道：

> Thanks for the clarification.
>
> The idea JobDeployer ever came into my mind when I was muddled with
> how to execute per-job mode and session mode with the same user code
> and framework codepath.
>
> With the concept JobDeployer we back to the statement that environment
> knows every configs of cluster deployment and job submission. We
> configure or generate from configuration a specific JobDeployer in
> environment and then code align on
>
> *JobClient client = env.execute().get();*
>
> which in session mode returned by clusterClient.submitJob and in per-job
> mode returned by clusterDescriptor.deployJobCluster.
>
> Here comes a problem that currently we directly run ClusterEntrypoint
> with extracted job graph. Follow the JobDeployer way we'd better
> align entry point of per-job deployment at JobDeployer. Users run
> their main method or by a Cli(finally call main method) to deploy the
> job cluster.
>
> Best,
> tison.
>
>
> Stephan Ewen  于2019年8月20日周二 下午4:40写道：
>
> > Till has made some good comments here.
> >
> > Two things to add:
> >
> >   - The job mode is very nice in the way that it runs the client inside
> the
> > cluster (in the same image/process that is the JM) and thus unifies both
> > applications and what the Spark world calls the "driver mode".
> >
> >   - Another thing I would add is that during the FLIP-6 design, we were
> > thinking about setups where Dispatcher and JobManager are separate
> > processes.
> > A Yarn or Mesos Dispatcher of a session could run independently (even
> > as privileged processes executing no code).
> > Then you the "per-job" mode could still be helpful: when a job is
> > submitted to the dispatcher, it launches the JM again in a per-job mode,
> so
> > that JM and TM processes are bound to teh job only. For higher security
> > setups, it is important that processes are not reused across jobs.
> >
> > On Tue, Aug 20, 2019 at 10:27 AM Till Rohrmann 
> > wrote:
> >
> > > I would not be in favour of getting rid of the per-job mode since it
> > > simplifies the process of running Flink jobs considerably. Moreover, it
> > is
> > > not only well suited for container deployments but also for deployments
> > > where you want to guarantee job isolation. For example, a user could
> use
> > > the per-job mode on Yarn to execute his job on a separate cluster.
> > >
> > > I think that having two notions of cluster deployments (session vs.
> > per-job
> > > mode) does not necessarily contradict your ideas for the client api
> > > refactoring. For example one could have the following interfaces:
> > >
> > > - ClusterDeploymentDescriptor: encapsulates the logic how to deploy a
> > > cluster.
> > > - ClusterClient: allows to interact with a cluster
> > > - JobClient: allows to interact with a running job
> > >
> > > Now the ClusterDeploymentDescriptor could have two methods:
> > >
> > > - ClusterClient deploySessionCluster()
> > > - JobClusterClient/JobClient deployPerJobCluster(JobGraph)
> > >
> > > where JobClusterClient is either a supertype of ClusterClient which
> does
> > > not give you the functionality to submit jobs or deployPerJobCluster
> > > returns directly a JobClient.
> > >
> > > When setting up the ExecutionEnvironment, one would then not provide a
> > > ClusterClient to submit jobs but a JobDeployer which, depending on the
> > > selected mode, either uses a ClusterClient (session mode) to submit
> jobs
> > or
> > > a ClusterDeploymentDescriptor to deploy per a job mode cluster with the
> > job
> > > to execute.
> > >
> > > These are just some thoughts how one could make it working because I
> > > believe there is some value in using the per job mode from the
> > > ExecutionEnvironment.
> > >
> > > Concerning the web submission, this is indeed a bit tricky. From a
> > cluster
> > > management stand point, I would in favour of not executing user code on
> > the
> > > REST endpoint. Especially when considering security, it would be good
> to
> > > have a well defined cluster behaviour where it is explicitly stated
> where
> > > user code and, thus, potentially risky code is executed. Ideally we
> limit
> > > it to the TaskExecutor and JobMaster.
> > >
> > > Cheers,
> > > Till
> > >
> 

[jira] [Created] (FLINK-13814) HiveTableSink should strip quotes from partition values

[Rui Li (Jira)]

Rui Li created FLINK-13814:
--

 Summary: HiveTableSink should strip quotes from partition values
 Key: FLINK-13814
 URL: https://issues.apache.org/jira/browse/FLINK-13814
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: CiBot Update

[Zili Chen]

Thanks for your announcement. Nice work!

Best,
tison.


vino yang  于2019年8月22日周四 上午8:14写道：

> +1 for "@flinkbot run travis", it is very convenient.
>
> Chesnay Schepler  于2019年8月21日周三 下午9:12写道：
>
> > Hi everyone,
> >
> > this is an update on recent changes to the CI bot.
> >
> >
> > The bot now cancels builds if a new commit was added to a PR, and
> > cancels all builds if the PR was closed.
> > (This was implemented a while ago; I'm just mentioning it again for
> > discoverability)
> >
> >
> > Additionally, starting today you can now re-trigger a Travis run by
> > writing a comment "@flinkbot run travis"; this means you no longer have
> > to commit an empty commit or do other shenanigans to get another build
> > running.
> > Note that this will /not/ work if the PR was re-opened, until at least 1
> > new build was triggered by a push.
> >
>

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

[Zili Chen]

Thanks Andrey for driving the discussion. Just for clarification,
what we conclude here are several guidelines without automatic
checker/tool guard them, right?

Best,
tison.


Andrey Zagrebin  于2019年8月21日周三 下午8:18写道：

> Hi All,
>
> I suggest we also conclude this discussion now.
>
> Breaking the line of too long statements (line longness is yet to be fully
> defined) to improve code readability in case of
>
>- Long function argument lists (declaration or call): void func(type1
>arg1, type2 arg2, ...)
>- Long sequence of chained calls:
>list.stream().map(...).reduce(...).collect(...)...
>
> Rules:
>
>- Break the list of arguments/calls if the line exceeds limit or earlier
>if you believe that the breaking would improve the code readability
>- If you break the line then each argument/call should have a separate
>line, including the first one
>- Each new line argument/call should have one extra indentation relative
>to the line of the parent function name or called entity
>- The opening brace always stays on the line of the parent function name
>- The closing brace of the function argument list and the possible
>thrown exception list always stay on the line of the last argument
>- The dot of a chained call is always on the line of that chained call
>proceeding the call at the beginning
>
> Examples of breaking:
>
>- Function arguments
>
> *public **void func(*
> *int arg1,*
> *int arg2,*
> *...)** throws E1, E2, E3 {*
> *...*
> *}*
>
>
>- Chained method calls:
>
> *values*
> *.stream()*
> *.map(*...*)*
> *.collect(...);*
>
>
> I suggest we spawn separate discussion threads (can do as a follow-up)
> about:
>
>- the hard line length limit in Java, possibly to confirm it also for
>Scala (cc @Tison)
>- indentation rules for the broken list of a declared function arguments
>
> If there are no more comments/objections/concerns, I will open a PR to
> capture the discussion outcome.
>
> Best,
> Andrey
>
>
>
> On Wed, Aug 21, 2019 at 8:57 AM Zili Chen  wrote:
>
> > Implement question: how to apply the line length rules?
> >
> > If we just turn on checkstyle rule "LineLength" then a huge
> > effort is required to break lines those break the rule. If
> > we use an auto-formatter here then it possibly break line
> > "just at the position" awfully.
> >
> > Is it possible we require only to fit the rule on the fly
> > a pull request touch files?
> >
> > Best,
> > tison.
> >
> >
> > Yu Li  于2019年8月20日周二 下午5:22写道：
> >
> > > I second Stephan's summarize, and to be more explicit, +1 on:
> > > - Set a hard line length limit
> > > - Allow arguments on the same line if below length limit
> > > - With consistent argument breaking when that length is exceeded
> > > - Developers can break before that if they feel it helps with
> readability
> > >
> > > FWIW, hbase project also sets the line length limit to 100 [1]
> > (personally
> > > I don't have any tendency on this, so JFYI).
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/hbase/blob/a59f7d4ffc27ea23b9822c3c26d6aeb76ccdf9aa/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml#L128
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 19 Aug 2019 at 18:22, Stephan Ewen  wrote:
> > >
> > > > I personally prefer not to break lines with few parameters.
> > > > It just feels unnecessarily clumsy to parse the breaks if there are
> > only
> > > > two or three arguments with short names.
> > > >
> > > > So +1
> > > >   - for a hard line length limit
> > > >   - allowing arguments on the same line if below that limit
> > > >   - with consistent argument breaking when that length is exceeded
> > > >   - developers can break before that if they feel it helps with
> > > > readability.
> > > >
> > > > This should be similar to what we have, except for enforcing the line
> > > > length limit.
> > > >
> > > > I think our Java guide originally suggested 120 characters line
> length,
> > > but
> > > > we can reduce that to 100 if a majority argues for that, but it is
> > > separate
> > > > discussion.
> > > > We uses shorter lines in Scala (100 chars) because Scala code becomes
> > > > harder to read faster with long lines.
> > > >
> > > >
> > > > On Mon, Aug 19, 2019 at 10:45 AM Andrey Zagrebin <
> and...@ververica.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Everybody,
> > > > >
> > > > > Thanks for your feedback guys and sorry for not getting back to the
> > > > > discussion for some time.
> > > > >
> > > > > @SHI Xiaogang
> > > > > About breaking lines for thrown exceptions:
> > > > > Indeed that would prevent growing the throw clause indefinitely.
> > > > > I am a bit concerned about putting the right parenthesis and/or
> throw
> > > > > clause on the next line
> > > > > because in general we do not it and there are a lot of variations
> of
> > > how
> > > > > and what to put to the next line so it needs explicit memorising.
> > > > > Also, we do 

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Zili Chen]

Thanks for driving this thread Andrey. Conclusion looks good to me.

Best,
tison.


Andrey Zagrebin  于2019年8月21日周三 下午7:25写道：

> FYI the PR: https://github.com/apache/flink-web/pull/251
> A review from the discussion participants would be appreciated.
>
> On Wed, Aug 21, 2019 at 1:23 PM Timo Walther  wrote:
>
> > Thanks for summarizing the discussion Andrey, +1 to this style.
> >
> > Regards,
> > Timo
> >
> >
> > Am 21.08.19 um 11:57 schrieb Andrey Zagrebin:
> > > Hi All,
> > >
> > > It looks like we have reached a consensus regarding the last left
> > question.
> > >
> > > I suggest the following final summary:
> > >
> > > - Use @Nullable annotation where you do not use Optional for the
> > > nullable values
> > > - If you can prove that Optional usage would lead to a performance
> > > degradation in critical code then fallback to @Nullable
> > > - Always use Optional to return nullable values in the API/public
> > > methods except the case of a proven performance concern
> > > - Do not use Optional as a function argument, instead either
> overload
> > > the method or use the Builder pattern for the set of function
> > arguments
> > >- Note: an Optional argument can be allowed in a *private*
> helper
> > >method if you believe that it simplifies the code, example is in
> > [1]
> > >- Do not use Optional for class fields
> > >
> > > If there are no more comments/concerns/objections I will open a PR to
> > > reflect this in the code style guide.
> > >
> > > Bets,
> > > Andrey
> > >
> > > [1]
> > >
> >
> https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroFactory.java#L95
> > >
> > > On Tue, Aug 20, 2019 at 10:35 AM Yu Li  wrote:
> > >
> > >> Thanks for the summarize Andrey!
> > >>
> > >> I'd also like to adjust my -1 to +0 on using Optional as parameter for
> > >> private methods due to the existence of the very first rule - "Avoid
> > using
> > >> Optional in any performance critical code". I'd regard the "possible
> GC
> > >> burden while using Optional as parameter" also one performance related
> > >> factor.
> > >>
> > >> And besides the code convention itself, I believe it's even more
> > important
> > >> to make us contributors know the reason behind.
> > >>
> > >> Thanks.
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >> On Tue, 20 Aug 2019 at 10:14, Stephan Ewen  wrote:
> > >>
> > >>> I think Dawid raised a very good point here.
> > >>> One of the outcomes should be that we are consistent in our
> > >> recommendations
> > >>> and requests during PR reviews. Otherwise we'll just confuse
> > >> contributors.
> > >>> So I would be
> > >>>+1 for someone to use Optional in a private method if they believe
> > it
> > >> is
> > >>> helpful
> > >>>-1 to push contributors during reviews to do that
> > >>>
> > >>>
> > >>> On Tue, Aug 20, 2019 at 9:42 AM Dawid Wysakowicz <
> > dwysakow...@apache.org
> > >>>
> > >>> wrote:
> > >>>
> >  Hi Andrey,
> > 
> >  Just wanted to quickly elaborate on my opinion. I wouldn't say I am
> > -1,
> >  just -0 for the Optionals in private methods. I am ok with not
> >  forbidding them there. I just think in all cases there is a better
> >  solution than passing the Optionals around, even in private
> methods. I
> >  just hope the outcome of the discussion won't be that it is no
> longer
> >  allowed to suggest those during review.
> > 
> >  Best,
> > 
> >  Dawid
> > 
> >  On 19/08/2019 17:53, Andrey Zagrebin wrote:
> > > Hi all,
> > >
> > > Sorry for not getting back to this discussion for some time.
> > > It looks like in general we agree on the initially suggested
> points:
> > >
> > > - Always use Optional only to return nullable values in the
> > >>> API/public
> > > methods
> > >- Only if you can prove that Optional usage would lead to a
> > >performance degradation in critical code then return
> nullable
> > >>> value
> > >directly and annotate it with @Nullable
> > > - Passing an Optional argument to a method can be allowed if it
> > is
> > > within a private helper method and simplifies the code
> > > - Optional should not be used for class fields
> > >
> > > The first point can be also elaborated by explicitly forbiding
> > > Optional/Nullable parameters in public methods.
> > > In general we can always avoid Optional parameters by either
> > >>> overloading
> > > the method or using a pojo with a builder to pass a set of
> > >> parameters.
> > > The third point does not prevent from using @Nullable/@Nonnull for
> > >>> class
> > > fields.
> > > If we agree to not use Optional for fields then not sure I see any
> > >> use
> >  case
> > > for SerializableOptional (please comment on that if you have more
> >  details).
> > > 

[jira] [Created] (FLINK-13813) metrics is different between overview ui and metric response

[lidesheng (Jira)]

lidesheng created FLINK-13813:
-

 Summary: metrics is different between overview ui and metric 
response
 Key: FLINK-13813
 URL: https://issues.apache.org/jira/browse/FLINK-13813
 Project: Flink
  Issue Type: Bug
 Environment: Flink 1.8.1 with hdfs
Reporter: lidesheng
 Attachments: metrics.png, overview.png

After a flink task is over, I get metrics by http request. The record count in 
response is different with job overview.After a flink task is over, I get 
metrics by http request. The record count in response is different with job 
overview.
get http://x.x.x.x:8081/jobs/57969f3978edf3115354fab1a72fd0c8 returns:
{ "jid": "57969f3978edf3115354fab1a72fd0c8", "name": "3349_cjrw_1566177018152", 
"isStoppable": false, "state": "FINISHED", ... "vertices": [ \{ "id": 
"d1cdde18b91ef6ce7c6a1cfdfa9e968d", "name": "CHAIN DataSource 
(3349_cjrw_1566177018152/SOURCE/0) -> Map (3349_cjrw_1566177018152/AUDIT/0)", 
"parallelism": 1, "status": "FINISHED", ... "metrics": { "read-bytes": 0, 
"read-bytes-complete": true, "write-bytes": 555941888, "write-bytes-complete": 
true, "read-records": 0, "read-records-complete": true, "write-records": 
500, "write-records-complete": true } ... } ... ] }}
But, the metrics by 
http://x.x.x.x:8081/jobs/57969f3978edf3115354fab1a72fd0c8/vertices/d1cdde18b91ef6ce7c6a1cfdfa9e968d/metrics?get=0.numRecordsOut
 returns
[\{"id":"0.numRecordsOut","value":"4084803"}]
The overview record count is different with task metrics, please view the 
apppendix.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: CiBot Update

[vino yang]

+1 for "@flinkbot run travis", it is very convenient.

Chesnay Schepler  于2019年8月21日周三 下午9:12写道：

> Hi everyone,
>
> this is an update on recent changes to the CI bot.
>
>
> The bot now cancels builds if a new commit was added to a PR, and
> cancels all builds if the PR was closed.
> (This was implemented a while ago; I'm just mentioning it again for
> discoverability)
>
>
> Additionally, starting today you can now re-trigger a Travis run by
> writing a comment "@flinkbot run travis"; this means you no longer have
> to commit an empty commit or do other shenanigans to get another build
> running.
> Note that this will /not/ work if the PR was re-opened, until at least 1
> new build was triggered by a push.
>

[RESULT] [VOTE] Apache Flink 1.9.0, release candidate #3

[Tzu-Li (Gordon) Tai]

I'm happy to announce that we have unanimously approved this candidate as
the 1.9.0 release.

There are 12 approving votes, 5 of which are binding:
- Yu Li
- Zili Chen
- Gordon Tai
- Stephan Ewen
- Jark Wu
- Vino Yang
- Gary Yao
- Bowen Li
- Chesnay Schepler
- Till Rohrmann
- Aljoscha Krettek
- David Anderson

There are no disapproving votes.

Thanks everyone who has contributed to this release!

I will wait until tomorrow morning for the artifacts to be available in
Maven central before announcing the release in a separate thread.

The release blog post will also be merged tomorrow along with the official
announcement.

Cheers,
Gordon

On Wed, Aug 21, 2019, 5:37 PM David Anderson  wrote:

> +1 (non-binding)
>
> I upgraded the flink-training-exercises project.
>
> I encountered a few rough edges, including problems in the docs, but
> nothing serious.
>
> I had to make some modifications to deal with changes in the Table API:
>
> ExternalCatalogTable.builder became new ExternalCatalogTableBuilder
> TableEnvironment.getTableEnvironment became StreamTableEnvironment.create
> StreamTableDescriptorValidator.UPDATE_MODE() became
> StreamTableDescriptorValidator.UPDATE_MODE
> org.apache.flink.table.api.java.Slide moved to
> org.apache.flink.table.api.Slide
>
> I also found myself forced to change a CoProcessFunction to a
> KeyedCoProcessFunction (which it should have been).
>
> I also tried a few complex queries in the SQL console, and wrote a
> simple job using the State Processor API. Everything worked.
>
> David
>
>
> David Anderson | Training Coordinator
>
> Follow us @VervericaData
>
> --
> Join Flink Forward - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
>
>
> On Wed, Aug 21, 2019 at 1:45 PM Aljoscha Krettek 
> wrote:
> >
> > +1
> >
> > I checked the last RC on a GCE cluster and was satisfied with the
> testing. The cherry-picked commits didn’t change anything related, so I’m
> forwarding my vote from there.
> >
> > Aljoscha
> >
> > > On 21. Aug 2019, at 13:34, Chesnay Schepler 
> wrote:
> > >
> > > +1 (binding)
> > >
> > > On 21/08/2019 08:09, Bowen Li wrote:
> > >> +1 non-binding
> > >>
> > >> - built from source with default profile
> > >> - manually ran SQL and Table API tests for Flink's metadata
> integration
> > >> with Hive Metastore in local cluster
> > >> - manually ran SQL tests for batch capability with Blink planner and
> Hive
> > >> integration (source/sink/udf) in local cluster
> > >> - file formats include: csv, orc, parquet
> > >>
> > >>
> > >> On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> Reran Jepsen tests 10 times.
> > >>>
> > >>> On Wed, Aug 21, 2019 at 5:35 AM vino yang 
> wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  - checkout source code and build successfully
> >  - started a local cluster and ran some example jobs successfully
> >  - verified signatures and hashes
> >  - checked release notes and post
> > 
> >  Best,
> >  Vino
> > 
> >  Stephan Ewen  于2019年8月21日周三 上午4:20写道：
> > 
> > > +1 (binding)
> > >
> > >  - Downloaded the binary release tarball
> > >  - started a standalone cluster with four nodes
> > >  - ran some examples through the Web UI
> > >  - checked the logs
> > >  - created a project from the Java quickstarts maven archetype
> > >  - ran a multi-stage DataSet job in batch mode
> > >  - killed as TaskManager and verified correct restart behavior,
> > >>> including
> > > failover region backtracking
> > >
> > >
> > > I found a few issues, and a common theme here is confusing error
> >  reporting
> > > and logging.
> > >
> > > (1) When testing batch failover and killing a TaskManager, the job
> >  reports
> > > as the failure cause "org.apache.flink.util.FlinkException: The
> > >>> assigned
> > > slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
> > > I think that is a pretty bad error message, as a user I don't
> know
> >  what
> > > that means. Some internal book keeping thing?
> > > You need to know a lot about Flink to understand that this
> means
> > > "TaskManager failure".
> > > https://issues.apache.org/jira/browse/FLINK-13805
> > > I would not block the release on this, but think this should
> get
> >  pretty
> > > urgent attention.
> > >
> > > (2) The Metric Fetcher floods the log with error messages when a
> > > TaskManager is lost.
> > >  There are many exceptions being logged by the Metrics Fetcher
> due
> > >>> to
> > > not reaching the TM any more.
> > >  This pollutes the log and drowns out the original exception
> and
> > >>> the
> > > meaningful logs from the scheduler/execution graph.
> > >  https://issues.apache.org/jira/browse/FLINK-13806
> > >  Again, I would not block the release on this, but think this
> > >>> should
> > > 

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[David Anderson]

+1 (non-binding)

I upgraded the flink-training-exercises project.

I encountered a few rough edges, including problems in the docs, but
nothing serious.

I had to make some modifications to deal with changes in the Table API:

ExternalCatalogTable.builder became new ExternalCatalogTableBuilder
TableEnvironment.getTableEnvironment became StreamTableEnvironment.create
StreamTableDescriptorValidator.UPDATE_MODE() became
StreamTableDescriptorValidator.UPDATE_MODE
org.apache.flink.table.api.java.Slide moved to org.apache.flink.table.api.Slide

I also found myself forced to change a CoProcessFunction to a
KeyedCoProcessFunction (which it should have been).

I also tried a few complex queries in the SQL console, and wrote a
simple job using the State Processor API. Everything worked.

David


David Anderson | Training Coordinator

Follow us @VervericaData

--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time


On Wed, Aug 21, 2019 at 1:45 PM Aljoscha Krettek  wrote:
>
> +1
>
> I checked the last RC on a GCE cluster and was satisfied with the testing. 
> The cherry-picked commits didn’t change anything related, so I’m forwarding 
> my vote from there.
>
> Aljoscha
>
> > On 21. Aug 2019, at 13:34, Chesnay Schepler  wrote:
> >
> > +1 (binding)
> >
> > On 21/08/2019 08:09, Bowen Li wrote:
> >> +1 non-binding
> >>
> >> - built from source with default profile
> >> - manually ran SQL and Table API tests for Flink's metadata integration
> >> with Hive Metastore in local cluster
> >> - manually ran SQL tests for batch capability with Blink planner and Hive
> >> integration (source/sink/udf) in local cluster
> >> - file formats include: csv, orc, parquet
> >>
> >>
> >> On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> Reran Jepsen tests 10 times.
> >>>
> >>> On Wed, Aug 21, 2019 at 5:35 AM vino yang  wrote:
> >>>
>  +1 (non-binding)
> 
>  - checkout source code and build successfully
>  - started a local cluster and ran some example jobs successfully
>  - verified signatures and hashes
>  - checked release notes and post
> 
>  Best,
>  Vino
> 
>  Stephan Ewen  于2019年8月21日周三 上午4:20写道：
> 
> > +1 (binding)
> >
> >  - Downloaded the binary release tarball
> >  - started a standalone cluster with four nodes
> >  - ran some examples through the Web UI
> >  - checked the logs
> >  - created a project from the Java quickstarts maven archetype
> >  - ran a multi-stage DataSet job in batch mode
> >  - killed as TaskManager and verified correct restart behavior,
> >>> including
> > failover region backtracking
> >
> >
> > I found a few issues, and a common theme here is confusing error
>  reporting
> > and logging.
> >
> > (1) When testing batch failover and killing a TaskManager, the job
>  reports
> > as the failure cause "org.apache.flink.util.FlinkException: The
> >>> assigned
> > slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
> > I think that is a pretty bad error message, as a user I don't know
>  what
> > that means. Some internal book keeping thing?
> > You need to know a lot about Flink to understand that this means
> > "TaskManager failure".
> > https://issues.apache.org/jira/browse/FLINK-13805
> > I would not block the release on this, but think this should get
>  pretty
> > urgent attention.
> >
> > (2) The Metric Fetcher floods the log with error messages when a
> > TaskManager is lost.
> >  There are many exceptions being logged by the Metrics Fetcher due
> >>> to
> > not reaching the TM any more.
> >  This pollutes the log and drowns out the original exception and
> >>> the
> > meaningful logs from the scheduler/execution graph.
> >  https://issues.apache.org/jira/browse/FLINK-13806
> >  Again, I would not block the release on this, but think this
> >>> should
> > get pretty urgent attention.
> >
> > (3) If you put "web.submit.enable: false" into the configuration, the
> >>> web
> > UI will still display the "SubmitJob" page, but errors will
> > continuously pop up, stating "Unable to load requested file /jars."
> > https://issues.apache.org/jira/browse/FLINK-13799
> >
> > (4) REST endpoint logs ERROR level messages when selecting the
> > "Checkpoints" tab for batch jobs. That does not seem correct.
> >  https://issues.apache.org/jira/browse/FLINK-13795
> >
> > Best,
> > Stephan
> >
> >
> >
> >
> > On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai <
>  tzuli...@apache.org>
> > wrote:
> >
> >> +1
> >>
> >> Legal checks:
> >> - verified signatures and hashes
> >> - New bundled Javascript dependencies for flink-runtime-web are
>  correctly
> >> reflected under 

CiBot Update

[Chesnay Schepler]

Hi everyone,

this is an update on recent changes to the CI bot.


The bot now cancels builds if a new commit was added to a PR, and 
cancels all builds if the PR was closed.
(This was implemented a while ago; I'm just mentioning it again for 
discoverability)



Additionally, starting today you can now re-trigger a Travis run by 
writing a comment "@flinkbot run travis"; this means you no longer have 
to commit an empty commit or do other shenanigans to get another build 
running.
Note that this will /not/ work if the PR was re-opened, until at least 1 
new build was triggered by a push.

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

[Andrey Zagrebin]

Hi All,

I suggest we also conclude this discussion now.

Breaking the line of too long statements (line longness is yet to be fully
defined) to improve code readability in case of

   - Long function argument lists (declaration or call): void func(type1
   arg1, type2 arg2, ...)
   - Long sequence of chained calls:
   list.stream().map(...).reduce(...).collect(...)...

Rules:

   - Break the list of arguments/calls if the line exceeds limit or earlier
   if you believe that the breaking would improve the code readability
   - If you break the line then each argument/call should have a separate
   line, including the first one
   - Each new line argument/call should have one extra indentation relative
   to the line of the parent function name or called entity
   - The opening brace always stays on the line of the parent function name
   - The closing brace of the function argument list and the possible
   thrown exception list always stay on the line of the last argument
   - The dot of a chained call is always on the line of that chained call
   proceeding the call at the beginning

Examples of breaking:

   - Function arguments

*public **void func(*
*int arg1,*
*int arg2,*
*...)** throws E1, E2, E3 {*
*...*
*}*


   - Chained method calls:

*values*
*.stream()*
*.map(*...*)*
*.collect(...);*


I suggest we spawn separate discussion threads (can do as a follow-up)
about:

   - the hard line length limit in Java, possibly to confirm it also for
   Scala (cc @Tison)
   - indentation rules for the broken list of a declared function arguments

If there are no more comments/objections/concerns, I will open a PR to
capture the discussion outcome.

Best,
Andrey



On Wed, Aug 21, 2019 at 8:57 AM Zili Chen  wrote:

> Implement question: how to apply the line length rules?
>
> If we just turn on checkstyle rule "LineLength" then a huge
> effort is required to break lines those break the rule. If
> we use an auto-formatter here then it possibly break line
> "just at the position" awfully.
>
> Is it possible we require only to fit the rule on the fly
> a pull request touch files?
>
> Best,
> tison.
>
>
> Yu Li  于2019年8月20日周二 下午5:22写道：
>
> > I second Stephan's summarize, and to be more explicit, +1 on:
> > - Set a hard line length limit
> > - Allow arguments on the same line if below length limit
> > - With consistent argument breaking when that length is exceeded
> > - Developers can break before that if they feel it helps with readability
> >
> > FWIW, hbase project also sets the line length limit to 100 [1]
> (personally
> > I don't have any tendency on this, so JFYI).
> >
> > [1]
> >
> >
> https://github.com/apache/hbase/blob/a59f7d4ffc27ea23b9822c3c26d6aeb76ccdf9aa/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml#L128
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 19 Aug 2019 at 18:22, Stephan Ewen  wrote:
> >
> > > I personally prefer not to break lines with few parameters.
> > > It just feels unnecessarily clumsy to parse the breaks if there are
> only
> > > two or three arguments with short names.
> > >
> > > So +1
> > >   - for a hard line length limit
> > >   - allowing arguments on the same line if below that limit
> > >   - with consistent argument breaking when that length is exceeded
> > >   - developers can break before that if they feel it helps with
> > > readability.
> > >
> > > This should be similar to what we have, except for enforcing the line
> > > length limit.
> > >
> > > I think our Java guide originally suggested 120 characters line length,
> > but
> > > we can reduce that to 100 if a majority argues for that, but it is
> > separate
> > > discussion.
> > > We uses shorter lines in Scala (100 chars) because Scala code becomes
> > > harder to read faster with long lines.
> > >
> > >
> > > On Mon, Aug 19, 2019 at 10:45 AM Andrey Zagrebin  >
> > > wrote:
> > >
> > > > Hi Everybody,
> > > >
> > > > Thanks for your feedback guys and sorry for not getting back to the
> > > > discussion for some time.
> > > >
> > > > @SHI Xiaogang
> > > > About breaking lines for thrown exceptions:
> > > > Indeed that would prevent growing the throw clause indefinitely.
> > > > I am a bit concerned about putting the right parenthesis and/or throw
> > > > clause on the next line
> > > > because in general we do not it and there are a lot of variations of
> > how
> > > > and what to put to the next line so it needs explicit memorising.
> > > > Also, we do not have many checked exceptions and usually avoid them.
> > > > Although I am not a big fan of many function arguments either but
> this
> > > > seems to be a bigger problem in the code base.
> > > > I would be ok to not enforce anything for exceptions atm.
> > > >
> > > > @Chesnay Schepler 
> > > > Thanks for mentioning automatic checks.
> > > > Indeed, pointing out this kind of style issues during PR reviews is
> > very
> > > > tedious
> > > > and cannot really force it without automated tools.
> > > > I would 

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Kostas Kloudas]

Thanks a lot!
We have the 3 +1's so we can move forward.

Cheers,
Kostas

On Wed, Aug 21, 2019 at 1:30 PM Chesnay Schepler  wrote:
>
> +1
>
> On 21/08/2019 13:23, Timo Walther wrote:
> > +1
> >
> > Am 21.08.19 um 13:21 schrieb Stephan Ewen:
> >> +1
> >>
> >> On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas 
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Following the FLIP process, this is a voting thread dedicated to the
> >>> FLIP-52.
> >>> As shown from the corresponding discussion thread [1], we seem to agree
> >>> that
> >>> the Program interface can be removed, so let's make it also official
> >>> with a vote.
> >>>
> >>> Cheers,
> >>> Kostas
> >>>
> >>>
> >>> [1]
> >>> https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E
> >>>
> >>>
> >
> >
>

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Aljoscha Krettek]

+1

> On 21. Aug 2019, at 13:30, Chesnay Schepler  wrote:
> 
> +1
> 
> On 21/08/2019 13:23, Timo Walther wrote:
>> +1
>> 
>> Am 21.08.19 um 13:21 schrieb Stephan Ewen:
>>> +1
>>> 
>>> On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas  wrote:
>>> 
 Hi all,
 
 Following the FLIP process, this is a voting thread dedicated to the
 FLIP-52.
 As shown from the corresponding discussion thread [1], we seem to agree
 that
 the Program interface can be removed, so let's make it also official
 with a vote.
 
 Cheers,
 Kostas
 
 
 [1]
 https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E
  
 
>> 
>> 
> 

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[Till Rohrmann]

+1 (binding)

- tested Flink's Java quickstarts
- tested Yarn deployment
- tested new web UI
- run quickstart job on Flink cluster
- reviewed release announcement and release notes

Cheers,
Till

On Wed, Aug 21, 2019 at 1:34 PM Chesnay Schepler  wrote:

> +1 (binding)
>
> On 21/08/2019 08:09, Bowen Li wrote:
> > +1 non-binding
> >
> > - built from source with default profile
> > - manually ran SQL and Table API tests for Flink's metadata integration
> > with Hive Metastore in local cluster
> > - manually ran SQL tests for batch capability with Blink planner and Hive
> > integration (source/sink/udf) in local cluster
> >  - file formats include: csv, orc, parquet
> >
> >
> > On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:
> >
> >> +1 (non-binding)
> >>
> >> Reran Jepsen tests 10 times.
> >>
> >> On Wed, Aug 21, 2019 at 5:35 AM vino yang 
> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> - checkout source code and build successfully
> >>> - started a local cluster and ran some example jobs successfully
> >>> - verified signatures and hashes
> >>> - checked release notes and post
> >>>
> >>> Best,
> >>> Vino
> >>>
> >>> Stephan Ewen  于2019年8月21日周三 上午4:20写道：
> >>>
>  +1 (binding)
> 
>    - Downloaded the binary release tarball
>    - started a standalone cluster with four nodes
>    - ran some examples through the Web UI
>    - checked the logs
>    - created a project from the Java quickstarts maven archetype
>    - ran a multi-stage DataSet job in batch mode
>    - killed as TaskManager and verified correct restart behavior,
> >> including
>  failover region backtracking
> 
> 
>  I found a few issues, and a common theme here is confusing error
> >>> reporting
>  and logging.
> 
>  (1) When testing batch failover and killing a TaskManager, the job
> >>> reports
>  as the failure cause "org.apache.flink.util.FlinkException: The
> >> assigned
>  slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
>   I think that is a pretty bad error message, as a user I don't
> know
> >>> what
>  that means. Some internal book keeping thing?
>   You need to know a lot about Flink to understand that this means
>  "TaskManager failure".
>   https://issues.apache.org/jira/browse/FLINK-13805
>   I would not block the release on this, but think this should get
> >>> pretty
>  urgent attention.
> 
>  (2) The Metric Fetcher floods the log with error messages when a
>  TaskManager is lost.
>    There are many exceptions being logged by the Metrics Fetcher
> due
> >> to
>  not reaching the TM any more.
>    This pollutes the log and drowns out the original exception and
> >> the
>  meaningful logs from the scheduler/execution graph.
>    https://issues.apache.org/jira/browse/FLINK-13806
>    Again, I would not block the release on this, but think this
> >> should
>  get pretty urgent attention.
> 
>  (3) If you put "web.submit.enable: false" into the configuration, the
> >> web
>  UI will still display the "SubmitJob" page, but errors will
>   continuously pop up, stating "Unable to load requested file
> /jars."
>   https://issues.apache.org/jira/browse/FLINK-13799
> 
>  (4) REST endpoint logs ERROR level messages when selecting the
>  "Checkpoints" tab for batch jobs. That does not seem correct.
>    https://issues.apache.org/jira/browse/FLINK-13795
> 
>  Best,
>  Stephan
> 
> 
> 
> 
>  On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai <
> >>> tzuli...@apache.org>
>  wrote:
> 
> > +1
> >
> > Legal checks:
> > - verified signatures and hashes
> > - New bundled Javascript dependencies for flink-runtime-web are
> >>> correctly
> > reflected under licenses-binary and NOTICE file.
> > - locally built from source (Scala 2.12, without Hadoop)
> > - No missing artifacts in staging repo
> > - No binaries in source release
> >
> > Functional checks:
> > - Quickstart working (both in IDE + job submission)
> > - Simple State Processor API program that performs offline key schema
> > migration (RocksDB backend). Generated savepoint is valid to restore
>  from.
> > - All E2E tests pass locally
> > - Didn’t notice any issues with the new WebUI
> >
> > Cheers,
> > Gordon
> >
> > On Tue, Aug 20, 2019 at 3:53 AM Zili Chen 
> >>> wrote:
> >> +1 (non-binding)
> >>
> >> - build from source: OK(8u212)
> >> - check local setup tutorial works as expected
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> Yu Li  于2019年8月20日周二 上午8:24写道：
> >>
> >>> +1 (non-binding)
> >>>
> >>> - checked release notes: OK
> >>> - checked sums and signatures: OK
> >>> - repository appears to contain all expected artifacts
> >>> - source release
> >>>   - 

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[Aljoscha Krettek]

+1

I checked the last RC on a GCE cluster and was satisfied with the testing. The 
cherry-picked commits didn’t change anything related, so I’m forwarding my vote 
from there.

Aljoscha

> On 21. Aug 2019, at 13:34, Chesnay Schepler  wrote:
> 
> +1 (binding)
> 
> On 21/08/2019 08:09, Bowen Li wrote:
>> +1 non-binding
>> 
>> - built from source with default profile
>> - manually ran SQL and Table API tests for Flink's metadata integration
>> with Hive Metastore in local cluster
>> - manually ran SQL tests for batch capability with Blink planner and Hive
>> integration (source/sink/udf) in local cluster
>> - file formats include: csv, orc, parquet
>> 
>> 
>> On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Reran Jepsen tests 10 times.
>>> 
>>> On Wed, Aug 21, 2019 at 5:35 AM vino yang  wrote:
>>> 
 +1 (non-binding)
 
 - checkout source code and build successfully
 - started a local cluster and ran some example jobs successfully
 - verified signatures and hashes
 - checked release notes and post
 
 Best,
 Vino
 
 Stephan Ewen  于2019年8月21日周三 上午4:20写道：
 
> +1 (binding)
> 
>  - Downloaded the binary release tarball
>  - started a standalone cluster with four nodes
>  - ran some examples through the Web UI
>  - checked the logs
>  - created a project from the Java quickstarts maven archetype
>  - ran a multi-stage DataSet job in batch mode
>  - killed as TaskManager and verified correct restart behavior,
>>> including
> failover region backtracking
> 
> 
> I found a few issues, and a common theme here is confusing error
 reporting
> and logging.
> 
> (1) When testing batch failover and killing a TaskManager, the job
 reports
> as the failure cause "org.apache.flink.util.FlinkException: The
>>> assigned
> slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
> I think that is a pretty bad error message, as a user I don't know
 what
> that means. Some internal book keeping thing?
> You need to know a lot about Flink to understand that this means
> "TaskManager failure".
> https://issues.apache.org/jira/browse/FLINK-13805
> I would not block the release on this, but think this should get
 pretty
> urgent attention.
> 
> (2) The Metric Fetcher floods the log with error messages when a
> TaskManager is lost.
>  There are many exceptions being logged by the Metrics Fetcher due
>>> to
> not reaching the TM any more.
>  This pollutes the log and drowns out the original exception and
>>> the
> meaningful logs from the scheduler/execution graph.
>  https://issues.apache.org/jira/browse/FLINK-13806
>  Again, I would not block the release on this, but think this
>>> should
> get pretty urgent attention.
> 
> (3) If you put "web.submit.enable: false" into the configuration, the
>>> web
> UI will still display the "SubmitJob" page, but errors will
> continuously pop up, stating "Unable to load requested file /jars."
> https://issues.apache.org/jira/browse/FLINK-13799
> 
> (4) REST endpoint logs ERROR level messages when selecting the
> "Checkpoints" tab for batch jobs. That does not seem correct.
>  https://issues.apache.org/jira/browse/FLINK-13795
> 
> Best,
> Stephan
> 
> 
> 
> 
> On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai <
 tzuli...@apache.org>
> wrote:
> 
>> +1
>> 
>> Legal checks:
>> - verified signatures and hashes
>> - New bundled Javascript dependencies for flink-runtime-web are
 correctly
>> reflected under licenses-binary and NOTICE file.
>> - locally built from source (Scala 2.12, without Hadoop)
>> - No missing artifacts in staging repo
>> - No binaries in source release
>> 
>> Functional checks:
>> - Quickstart working (both in IDE + job submission)
>> - Simple State Processor API program that performs offline key schema
>> migration (RocksDB backend). Generated savepoint is valid to restore
> from.
>> - All E2E tests pass locally
>> - Didn’t notice any issues with the new WebUI
>> 
>> Cheers,
>> Gordon
>> 
>> On Tue, Aug 20, 2019 at 3:53 AM Zili Chen 
 wrote:
>>> +1 (non-binding)
>>> 
>>> - build from source: OK(8u212)
>>> - check local setup tutorial works as expected
>>> 
>>> Best,
>>> tison.
>>> 
>>> 
>>> Yu Li  于2019年8月20日周二 上午8:24写道：
>>> 
 +1 (non-binding)
 
 - checked release notes: OK
 - checked sums and signatures: OK
 - repository appears to contain all expected artifacts
 - source release
  - contains no binaries: OK
  - contains no 1.9-SNAPSHOT references: OK
  - build from source: OK (8u102)
 - binary 

Re: [DISCUSS] Release flink-shaded 8.0

[Chesnay Schepler]

Nico has opened a PR for bumping netty; we plan to have this merged by 
tomorrow.


Unless anyone has concerns I will kick off the release on Friday.

On 19/08/2019 12:11, Nico Kruber wrote:

I quickly went through all the changelogs for Netty 4.1.32 (which we
currently use) to the latest Netty 4.1.39.Final. Below, you will find a
list of bug fixes and performance improvements that may affect us. Nice
changes we could benefit from, also for the Java > 8 efforts. The most
important ones fixing leaks etc are #8921, #9167, #9274, #9394, and the
various CompositeByteBuf fixes. The rest are mostly performance
improvements.

Since we are still early in the dev cycle for Flink 1.10, it would maybe
nice to update and verify that the new version works correctly. I'll
create a ticket and PR.


FYI (1): My own patches to bring dynamically-linked openSSL to more
distributions, namely SUSE and Arch, have not made it into a release yet.

FYI (2): We are currently using the latest version of netty-tcnative,
i.e. 2.0.25.


Nico

--
Netty 4.1.33.Final
- Fix ClassCastException and native crash when using kqueue transport
(#8665)
- Provide a way to cache the internal nioBuffer of the PooledByteBuffer
to reduce GC (#8603)

Netty 4.1.34.Final
- Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs
related to GCLocker (#8921)
- Correctly monkey-patch id also in whe os / arch is used within library
name (#8913)
- Further reduce ensureAccessible() overhead (#8895)
- Support using an Executor to offload blocking / long-running tasks
when processing TLS / SSL via the SslHandler (#8847)
- Minimize memory footprint for AbstractChannelHandlerContext for
handlers that execute in the EventExecutor (#8786)
- Fix three bugs in CompositeByteBuf (#8773)

Netty 4.1.35.Final
- Fix possible ByteBuf leak when CompositeByteBuf is resized (#8946)
- Correctly produce ssl alert when certificate validation fails on the
client-side when using native SSL implementation (#8949)

Netty 4.1.37.Final
- Don't filter out TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (#9274)
- Try to mark child channel writable again once the parent channel
becomes writable (#9254)
- Properly debounce wakeups (#9191)
- Don't read from timerfd and eventfd on each EventLoop tick (#9192)
- Correctly detect that KeyManagerFactory is not supported when using
OpenSSL 1.1.0+ (#9170)
- Fix possible unsafe sharing of internal NIO buffer in CompositeByteBuf
(#9169)
- KQueueEventLoop won't unregister active channels reusing a file
descriptor (#9149)
- Prefer direct io buffers if direct buffers pooled (#9167)

Netty 4.1.38.Final
- Prevent ByteToMessageDecoder from overreading when !isAutoRead (#9252)
- Correctly take length of ByteBufInputStream into account for
readLine() / readByte() (#9310)
- availableSharedCapacity will be slowly exhausted (#9394)
--

On 18/08/2019 16:47, Stephan Ewen wrote:

Are we fine with the current Netty version, or would be want to bump it?

On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler mailto:ches...@apache.org>> wrote:

 Hello,

 I would like to kick off the next flink-shaded release next week. There
 are 2 ongoing efforts that are blocked on this release:

   * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
     handle Java 11 bytecode
   * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
     requires asm-commons to be added to flink-shaded-asm

 Are there any other changes on anyone's radar that we will have to make
 for 1.10? (will bumping calcite require anything, for example)

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[Chesnay Schepler]

+1 (binding)

On 21/08/2019 08:09, Bowen Li wrote:

+1 non-binding

- built from source with default profile
- manually ran SQL and Table API tests for Flink's metadata integration
with Hive Metastore in local cluster
- manually ran SQL tests for batch capability with Blink planner and Hive
integration (source/sink/udf) in local cluster
 - file formats include: csv, orc, parquet


On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:


+1 (non-binding)

Reran Jepsen tests 10 times.

On Wed, Aug 21, 2019 at 5:35 AM vino yang  wrote:


+1 (non-binding)

- checkout source code and build successfully
- started a local cluster and ran some example jobs successfully
- verified signatures and hashes
- checked release notes and post

Best,
Vino

Stephan Ewen  于2019年8月21日周三 上午4:20写道：


+1 (binding)

  - Downloaded the binary release tarball
  - started a standalone cluster with four nodes
  - ran some examples through the Web UI
  - checked the logs
  - created a project from the Java quickstarts maven archetype
  - ran a multi-stage DataSet job in batch mode
  - killed as TaskManager and verified correct restart behavior,

including

failover region backtracking


I found a few issues, and a common theme here is confusing error

reporting

and logging.

(1) When testing batch failover and killing a TaskManager, the job

reports

as the failure cause "org.apache.flink.util.FlinkException: The

assigned

slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
 I think that is a pretty bad error message, as a user I don't know

what

that means. Some internal book keeping thing?
 You need to know a lot about Flink to understand that this means
"TaskManager failure".
 https://issues.apache.org/jira/browse/FLINK-13805
 I would not block the release on this, but think this should get

pretty

urgent attention.

(2) The Metric Fetcher floods the log with error messages when a
TaskManager is lost.
  There are many exceptions being logged by the Metrics Fetcher due

to

not reaching the TM any more.
  This pollutes the log and drowns out the original exception and

the

meaningful logs from the scheduler/execution graph.
  https://issues.apache.org/jira/browse/FLINK-13806
  Again, I would not block the release on this, but think this

should

get pretty urgent attention.

(3) If you put "web.submit.enable: false" into the configuration, the

web

UI will still display the "SubmitJob" page, but errors will
 continuously pop up, stating "Unable to load requested file /jars."
 https://issues.apache.org/jira/browse/FLINK-13799

(4) REST endpoint logs ERROR level messages when selecting the
"Checkpoints" tab for batch jobs. That does not seem correct.
  https://issues.apache.org/jira/browse/FLINK-13795

Best,
Stephan




On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai <

tzuli...@apache.org>

wrote:


+1

Legal checks:
- verified signatures and hashes
- New bundled Javascript dependencies for flink-runtime-web are

correctly

reflected under licenses-binary and NOTICE file.
- locally built from source (Scala 2.12, without Hadoop)
- No missing artifacts in staging repo
- No binaries in source release

Functional checks:
- Quickstart working (both in IDE + job submission)
- Simple State Processor API program that performs offline key schema
migration (RocksDB backend). Generated savepoint is valid to restore

from.

- All E2E tests pass locally
- Didn’t notice any issues with the new WebUI

Cheers,
Gordon

On Tue, Aug 20, 2019 at 3:53 AM Zili Chen 

wrote:

+1 (non-binding)

- build from source: OK(8u212)
- check local setup tutorial works as expected

Best,
tison.


Yu Li  于2019年8月20日周二 上午8:24写道：


+1 (non-binding)

- checked release notes: OK
- checked sums and signatures: OK
- repository appears to contain all expected artifacts
- source release
  - contains no binaries: OK
  - contains no 1.9-SNAPSHOT references: OK
  - build from source: OK (8u102)
- binary release
  - no examples appear to be missing
  - started a cluster; WebUI reachable, example ran

successfully

- checked README.md file and found nothing unexpected

Best Regards,
Yu


On Tue, 20 Aug 2019 at 01:16, Tzu-Li (Gordon) Tai <

tzuli...@apache.org

wrote:


Hi all,

Release candidate #3 for Apache Flink 1.9.0 is now ready for

your

review.

Please review and vote on release candidate #3 for version

1.9.0,

as

follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific

comments)

The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source release and binary convenience

releases

to

be

deployed to dist.apache.org [2], which are signed with the key

with

fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
* all artifacts to be deployed to the Maven Central Repository

[4],

* source code tag “release-1.9.0-rc3” [5].
* pull requests for the release note documentation [6] and

[DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

[Timo Walther]

Hi everyone,

some of you might remember the discussion I started end of March [1] 
about introducing a new Java DSL for Table API that is not embedded in a 
string.


In particular, it solves the following issues:

- No possibility of deprecating functions

- Missing documentation for users

- Missing auto-completion for users

- Need to port the ExpressionParser from Scala to Java

- Scala symbols are deprecated! A Java DSL can also enable the Scala DSL 
one.


Due to shift of priorities, we could not work on it in Flink 1.9 but the 
feedback at that time was positive and we should aim for 1.10 to 
simplify the API with this change.


We propose the following FLIP-55:

https://docs.google.com/document/d/1CfaaD3j8APJDKwzIT4YsX7QD2huKTB4xlA3vnMUFJmA/edit?usp=sharing 



Thanks for any feedback,

Timo

[1] 
https://lists.apache.org/thread.html/e6f31d7fa53890b91be0991c2da64556a91ef0fc9ab3ffa889dacc23@%3Cdev.flink.apache.org%3E

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Chesnay Schepler]

+1

On 21/08/2019 13:23, Timo Walther wrote:

+1

Am 21.08.19 um 13:21 schrieb Stephan Ewen:

+1

On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas  
wrote:



Hi all,

Following the FLIP process, this is a voting thread dedicated to the
FLIP-52.
As shown from the corresponding discussion thread [1], we seem to agree
that
the Program interface can be removed, so let's make it also official
with a vote.

Cheers,
Kostas


[1]
https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E 

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Andrey Zagrebin]

FYI the PR: https://github.com/apache/flink-web/pull/251
A review from the discussion participants would be appreciated.

On Wed, Aug 21, 2019 at 1:23 PM Timo Walther  wrote:

> Thanks for summarizing the discussion Andrey, +1 to this style.
>
> Regards,
> Timo
>
>
> Am 21.08.19 um 11:57 schrieb Andrey Zagrebin:
> > Hi All,
> >
> > It looks like we have reached a consensus regarding the last left
> question.
> >
> > I suggest the following final summary:
> >
> > - Use @Nullable annotation where you do not use Optional for the
> > nullable values
> > - If you can prove that Optional usage would lead to a performance
> > degradation in critical code then fallback to @Nullable
> > - Always use Optional to return nullable values in the API/public
> > methods except the case of a proven performance concern
> > - Do not use Optional as a function argument, instead either overload
> > the method or use the Builder pattern for the set of function
> arguments
> >- Note: an Optional argument can be allowed in a *private* helper
> >method if you believe that it simplifies the code, example is in
> [1]
> >- Do not use Optional for class fields
> >
> > If there are no more comments/concerns/objections I will open a PR to
> > reflect this in the code style guide.
> >
> > Bets,
> > Andrey
> >
> > [1]
> >
> https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroFactory.java#L95
> >
> > On Tue, Aug 20, 2019 at 10:35 AM Yu Li  wrote:
> >
> >> Thanks for the summarize Andrey!
> >>
> >> I'd also like to adjust my -1 to +0 on using Optional as parameter for
> >> private methods due to the existence of the very first rule - "Avoid
> using
> >> Optional in any performance critical code". I'd regard the "possible GC
> >> burden while using Optional as parameter" also one performance related
> >> factor.
> >>
> >> And besides the code convention itself, I believe it's even more
> important
> >> to make us contributors know the reason behind.
> >>
> >> Thanks.
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Tue, 20 Aug 2019 at 10:14, Stephan Ewen  wrote:
> >>
> >>> I think Dawid raised a very good point here.
> >>> One of the outcomes should be that we are consistent in our
> >> recommendations
> >>> and requests during PR reviews. Otherwise we'll just confuse
> >> contributors.
> >>> So I would be
> >>>+1 for someone to use Optional in a private method if they believe
> it
> >> is
> >>> helpful
> >>>-1 to push contributors during reviews to do that
> >>>
> >>>
> >>> On Tue, Aug 20, 2019 at 9:42 AM Dawid Wysakowicz <
> dwysakow...@apache.org
> >>>
> >>> wrote:
> >>>
>  Hi Andrey,
> 
>  Just wanted to quickly elaborate on my opinion. I wouldn't say I am
> -1,
>  just -0 for the Optionals in private methods. I am ok with not
>  forbidding them there. I just think in all cases there is a better
>  solution than passing the Optionals around, even in private methods. I
>  just hope the outcome of the discussion won't be that it is no longer
>  allowed to suggest those during review.
> 
>  Best,
> 
>  Dawid
> 
>  On 19/08/2019 17:53, Andrey Zagrebin wrote:
> > Hi all,
> >
> > Sorry for not getting back to this discussion for some time.
> > It looks like in general we agree on the initially suggested points:
> >
> > - Always use Optional only to return nullable values in the
> >>> API/public
> > methods
> >- Only if you can prove that Optional usage would lead to a
> >performance degradation in critical code then return nullable
> >>> value
> >directly and annotate it with @Nullable
> > - Passing an Optional argument to a method can be allowed if it
> is
> > within a private helper method and simplifies the code
> > - Optional should not be used for class fields
> >
> > The first point can be also elaborated by explicitly forbiding
> > Optional/Nullable parameters in public methods.
> > In general we can always avoid Optional parameters by either
> >>> overloading
> > the method or using a pojo with a builder to pass a set of
> >> parameters.
> > The third point does not prevent from using @Nullable/@Nonnull for
> >>> class
> > fields.
> > If we agree to not use Optional for fields then not sure I see any
> >> use
>  case
> > for SerializableOptional (please comment on that if you have more
>  details).
> > @Jingsong Lee
> > Using Optional in Maps.
> > I can see this as a possible use case.
> > I would leave this decision to the developer/reviewer to reason about
> >>> it
> > and keep the scope of this discussion to the variables/parameters as
> >> it
> > seems to be the biggest point of friction atm.
> >
> > Finally, I see a split regarding the second point:  >> Optional
> 

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Timo Walther]

+1

Am 21.08.19 um 13:21 schrieb Stephan Ewen:

+1

On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas  wrote:


Hi all,

Following the FLIP process, this is a voting thread dedicated to the
FLIP-52.
As shown from the corresponding discussion thread [1], we seem to agree
that
the Program interface can be removed, so let's make it also official
with a vote.

Cheers,
Kostas


[1]
https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Timo Walther]

Thanks for summarizing the discussion Andrey, +1 to this style.

Regards,
Timo


Am 21.08.19 um 11:57 schrieb Andrey Zagrebin:

Hi All,

It looks like we have reached a consensus regarding the last left question.

I suggest the following final summary:

- Use @Nullable annotation where you do not use Optional for the
nullable values
- If you can prove that Optional usage would lead to a performance
degradation in critical code then fallback to @Nullable
- Always use Optional to return nullable values in the API/public
methods except the case of a proven performance concern
- Do not use Optional as a function argument, instead either overload
the method or use the Builder pattern for the set of function arguments
   - Note: an Optional argument can be allowed in a *private* helper
   method if you believe that it simplifies the code, example is in [1]
   - Do not use Optional for class fields

If there are no more comments/concerns/objections I will open a PR to
reflect this in the code style guide.

Bets,
Andrey

[1]
https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroFactory.java#L95

On Tue, Aug 20, 2019 at 10:35 AM Yu Li  wrote:


Thanks for the summarize Andrey!

I'd also like to adjust my -1 to +0 on using Optional as parameter for
private methods due to the existence of the very first rule - "Avoid using
Optional in any performance critical code". I'd regard the "possible GC
burden while using Optional as parameter" also one performance related
factor.

And besides the code convention itself, I believe it's even more important
to make us contributors know the reason behind.

Thanks.

Best Regards,
Yu


On Tue, 20 Aug 2019 at 10:14, Stephan Ewen  wrote:


I think Dawid raised a very good point here.
One of the outcomes should be that we are consistent in our

recommendations

and requests during PR reviews. Otherwise we'll just confuse

contributors.

So I would be
   +1 for someone to use Optional in a private method if they believe it

is

helpful
   -1 to push contributors during reviews to do that


On Tue, Aug 20, 2019 at 9:42 AM Dawid Wysakowicz 
Hi Andrey,

Just wanted to quickly elaborate on my opinion. I wouldn't say I am -1,
just -0 for the Optionals in private methods. I am ok with not
forbidding them there. I just think in all cases there is a better
solution than passing the Optionals around, even in private methods. I
just hope the outcome of the discussion won't be that it is no longer
allowed to suggest those during review.

Best,

Dawid

On 19/08/2019 17:53, Andrey Zagrebin wrote:

Hi all,

Sorry for not getting back to this discussion for some time.
It looks like in general we agree on the initially suggested points:

- Always use Optional only to return nullable values in the

API/public

methods
   - Only if you can prove that Optional usage would lead to a
   performance degradation in critical code then return nullable

value

   directly and annotate it with @Nullable
- Passing an Optional argument to a method can be allowed if it is
within a private helper method and simplifies the code
- Optional should not be used for class fields

The first point can be also elaborated by explicitly forbiding
Optional/Nullable parameters in public methods.
In general we can always avoid Optional parameters by either

overloading

the method or using a pojo with a builder to pass a set of

parameters.

The third point does not prevent from using @Nullable/@Nonnull for

class

fields.
If we agree to not use Optional for fields then not sure I see any

use

case

for SerializableOptional (please comment on that if you have more

details).

@Jingsong Lee
Using Optional in Maps.
I can see this as a possible use case.
I would leave this decision to the developer/reviewer to reason about

it

and keep the scope of this discussion to the variables/parameters as

it

seems to be the biggest point of friction atm.

Finally, I see a split regarding the second point: 
Optional

argument to a method can be allowed if it is within a private helper

method

and simplifies the code>.
There are people who have a strong opinion against allowing it.
Let's vote then for whether to allow it or not.
So far as I see we have the following votes (correct me if wrong and

add

more if you want):
Piotr+1
Biao+1
Timo   -1
Yu   -1
Aljoscha -1
Till  +1
Igal+1
Dawid-1
Me +1

So far: +5 / -4 (Optional arguments in private methods)

Best,
Andrey


On Tue, Aug 6, 2019 at 8:53 AM Piotr Nowojski 

wrote:

Hi Qi,


For example, SingleInputGate is already creating Optional for every

BufferOrEvent in getNextBufferOrEvent(). How much performance gain

would we

get if it’s replaced by null check?

When I was introducing it there, I have micro-benchmarked this

change,

and

there was no visible throughput change in a pure 

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Stephan Ewen]

+1

On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas  wrote:

> Hi all,
>
> Following the FLIP process, this is a voting thread dedicated to the
> FLIP-52.
> As shown from the corresponding discussion thread [1], we seem to agree
> that
> the Program interface can be removed, so let's make it also official
> with a vote.
>
> Cheers,
> Kostas
>
>
> [1]
> https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E
>

[VOTE] FLIP-52: Remove legacy Program interface.

[Kostas Kloudas]

Hi all,

Following the FLIP process, this is a voting thread dedicated to the FLIP-52.
As shown from the corresponding discussion thread [1], we seem to agree that
the Program interface can be removed, so let's make it also official
with a vote.

Cheers,
Kostas


[1]  
https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E

Re: [DISCUSS][CODE STYLE] Create collections always with initial capacity

[Andrey Zagrebin]

FYI the PR: https://github.com/apache/flink-web/pull/249
A review from the discussion participants would be appreciated.

On Tue, Aug 20, 2019 at 5:29 PM Andrey Zagrebin 
wrote:

> I created an umbrella issue for the code style guide effort and a subtask
> for this discussion:
> https://issues.apache.org/jira/browse/FLINK-13804
> I will also submit a PR to flink-web based on the conclusion.
>
> On Mon, Aug 19, 2019 at 6:15 PM Stephan Ewen  wrote:
>
>> @Andrey Will you open a PR to add this to the code style?
>>
>> On Mon, Aug 19, 2019 at 11:51 AM Andrey Zagrebin 
>> wrote:
>>
>> > Hi All,
>> >
>> > It looks like this proposal has an approval and we can conclude this
>> > discussion.
>> > Additionally, I agree with Piotr we should really force the proven good
>> > reasoning for setting the capacity to avoid confusion, redundancy and
>> other
>> > already mentioned things while reading and maintaining the code.
>> > Ideally the need of setting the capacity should be either immediately
>> clear
>> > (e.g. perf etc) or explained in comments if it is non-trivial.
>> > Although, it can easily enter a grey zone, so I would not demand
>> strictly
>> > performance measurement proof e.g. if the size is known and it is "per
>> > record" code.
>> > At the end of the day it is a decision of the code developer and
>> reviewer.
>> >
>> > The conclusion is then:
>> > Set the initial capacity only if there is a good proven reason to do it.
>> > Otherwise do not clutter the code with it.
>> >
>> > Best,
>> > Andrey
>> >
>> > On Thu, Aug 1, 2019 at 5:10 PM Piotr Nowojski 
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > > - a bit more code, increases maintenance burden.
>> > >
>> > > I think there is even more to that. It’s almost like a code
>> duplication,
>> > > albeit expressed in very different way, with all of the drawbacks of
>> > > duplicated code: initial capacity can drift out of sync, causing
>> > confusion.
>> > > Also it’s not “a bit more code”, it might be non trivial
>> > > reasoning/calculation how to set the initial value. Whenever we change
>> > > something/refactor the code, "maintenance burden” will mostly come
>> from
>> > > that.
>> > >
>> > > Also I think this just usually falls under a premature optimisation
>> rule.
>> > >
>> > > Besides:
>> > >
>> > > > The conclusion is the following at the moment:
>> > > > Only set the initial capacity if you have a good idea about the
>> > expected
>> > > size.
>> > >
>> > > I would add a clause to set the initial capacity “only for good proven
>> > > reasons”. It’s not about whether we can set it, but whether it makes
>> > sense
>> > > to do so (to avoid the before mentioned "maintenance burden”).
>> > >
>> > > Piotrek
>> > >
>> > > > On 1 Aug 2019, at 14:41, Xintong Song 
>> wrote:
>> > > >
>> > > > +1 on setting initial capacity only when have good expectation on
>> the
>> > > > collection size.
>> > > >
>> > > > Thank you~
>> > > >
>> > > > Xintong Song
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Aug 1, 2019 at 2:32 PM Andrey Zagrebin <
>> and...@ververica.com>
>> > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> As you probably already noticed, Stephan has triggered a discussion
>> > > thread
>> > > >> about code style guide for Flink [1]. Recently we were discussing
>> > > >> internally some smaller concerns and I would like start separate
>> > threads
>> > > >> for them.
>> > > >>
>> > > >> This thread is about creating collections always with initial
>> > capacity.
>> > > As
>> > > >> you might have seen, some parts of our code base always initialise
>> > > >> collections with some non-default capacity. You can even activate a
>> > > check
>> > > >> in IntelliJ Idea that can monitor and highlight creation of
>> collection
>> > > >> without initial capacity.
>> > > >>
>> > > >> Pros:
>> > > >> - performance gain if there is a good reasoning about initial
>> capacity
>> > > >> - the capacity is always deterministic and does not depend on any
>> > > changes
>> > > >> of its default value in Java
>> > > >> - easy to follow: always initialise, has IDE support for detection
>> > > >>
>> > > >> Cons (for initialising w/o good reasoning):
>> > > >> - We are trying to outsmart JVM. When there is no good reasoning
>> about
>> > > >> initial capacity, we can rely on JVM default value.
>> > > >> - It is even confusing e.g. for hash maps as the real size depends
>> on
>> > > the
>> > > >> load factor.
>> > > >> - It would only add minor performance gain.
>> > > >> - a bit more code, increases maintenance burden.
>> > > >>
>> > > >> The conclusion is the following at the moment:
>> > > >> Only set the initial capacity if you have a good idea about the
>> > expected
>> > > >> size.
>> > > >>
>> > > >> Please, feel free to share you thoughts.
>> > > >>
>> > > >> Best,
>> > > >> Andrey
>> > > >>
>> > > >> [1]
>> > > >>
>> > > >>
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/flink-dev/201906.mbox/%3ced91df4b-7cab-4547-a430-85bc710fd...@apache.org%3E
>> > > >>
>> 

Re: [VOTE] FLIP-50: Spill-able Heap State Backend

[Yu Li]

With 3 binding +1 votes and no +0/-1, this vote passed.

Thanks all for voting, will start the implementation soon.

Best Regards,
Yu


On Mon, 19 Aug 2019 at 14:07, Tzu-Li (Gordon) Tai 
wrote:

> +1
>
> On Sun, Aug 18, 2019 at 4:30 PM Stephan Ewen  wrote:
>
> > +1
> >
> > On Sun, Aug 18, 2019 at 3:31 PM Till Rohrmann 
> > wrote:
> >
> > > +1
> > >
> > > On Fri, Aug 16, 2019 at 4:54 PM Yu Li  wrote:
> > >
> > > > Hi All,
> > > >
> > > > Since we have reached a consensus in the discussion thread [1], I'd
> > like
> > > to
> > > > start the voting for FLIP-50 [2].
> > > >
> > > > This vote will be open for at least 72 hours. Unless objection I will
> > try
> > > > to close it by end of Tuesday August 20, 2019 if we have sufficient
> > > votes.
> > > > Thanks.
> > > >
> > > > [1] https://s.apache.org/cq358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > >
> >
>

[jira] [Created] (FLINK-13812) Code style for the usage of Java Optional

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-13812:
---

 Summary: Code style for the usage of Java Optional
 Key: FLINK-13812
 URL: https://issues.apache.org/jira/browse/FLINK-13812
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Project Website
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Andrey Zagrebin]

Hi All,

It looks like we have reached a consensus regarding the last left question.

I suggest the following final summary:

   - Use @Nullable annotation where you do not use Optional for the
   nullable values
   - If you can prove that Optional usage would lead to a performance
   degradation in critical code then fallback to @Nullable
   - Always use Optional to return nullable values in the API/public
   methods except the case of a proven performance concern
   - Do not use Optional as a function argument, instead either overload
   the method or use the Builder pattern for the set of function arguments
  - Note: an Optional argument can be allowed in a *private* helper
  method if you believe that it simplifies the code, example is in [1]
  - Do not use Optional for class fields

If there are no more comments/concerns/objections I will open a PR to
reflect this in the code style guide.

Bets,
Andrey

[1]
https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroFactory.java#L95

On Tue, Aug 20, 2019 at 10:35 AM Yu Li  wrote:

> Thanks for the summarize Andrey!
>
> I'd also like to adjust my -1 to +0 on using Optional as parameter for
> private methods due to the existence of the very first rule - "Avoid using
> Optional in any performance critical code". I'd regard the "possible GC
> burden while using Optional as parameter" also one performance related
> factor.
>
> And besides the code convention itself, I believe it's even more important
> to make us contributors know the reason behind.
>
> Thanks.
>
> Best Regards,
> Yu
>
>
> On Tue, 20 Aug 2019 at 10:14, Stephan Ewen  wrote:
>
> > I think Dawid raised a very good point here.
> > One of the outcomes should be that we are consistent in our
> recommendations
> > and requests during PR reviews. Otherwise we'll just confuse
> contributors.
> >
> > So I would be
> >   +1 for someone to use Optional in a private method if they believe it
> is
> > helpful
> >   -1 to push contributors during reviews to do that
> >
> >
> > On Tue, Aug 20, 2019 at 9:42 AM Dawid Wysakowicz  >
> > wrote:
> >
> > > Hi Andrey,
> > >
> > > Just wanted to quickly elaborate on my opinion. I wouldn't say I am -1,
> > > just -0 for the Optionals in private methods. I am ok with not
> > > forbidding them there. I just think in all cases there is a better
> > > solution than passing the Optionals around, even in private methods. I
> > > just hope the outcome of the discussion won't be that it is no longer
> > > allowed to suggest those during review.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 19/08/2019 17:53, Andrey Zagrebin wrote:
> > > > Hi all,
> > > >
> > > > Sorry for not getting back to this discussion for some time.
> > > > It looks like in general we agree on the initially suggested points:
> > > >
> > > >- Always use Optional only to return nullable values in the
> > API/public
> > > >methods
> > > >   - Only if you can prove that Optional usage would lead to a
> > > >   performance degradation in critical code then return nullable
> > value
> > > >   directly and annotate it with @Nullable
> > > >- Passing an Optional argument to a method can be allowed if it is
> > > >within a private helper method and simplifies the code
> > > >- Optional should not be used for class fields
> > > >
> > > > The first point can be also elaborated by explicitly forbiding
> > > > Optional/Nullable parameters in public methods.
> > > > In general we can always avoid Optional parameters by either
> > overloading
> > > > the method or using a pojo with a builder to pass a set of
> parameters.
> > > >
> > > > The third point does not prevent from using @Nullable/@Nonnull for
> > class
> > > > fields.
> > > > If we agree to not use Optional for fields then not sure I see any
> use
> > > case
> > > > for SerializableOptional (please comment on that if you have more
> > > details).
> > > >
> > > > @Jingsong Lee
> > > > Using Optional in Maps.
> > > > I can see this as a possible use case.
> > > > I would leave this decision to the developer/reviewer to reason about
> > it
> > > > and keep the scope of this discussion to the variables/parameters as
> it
> > > > seems to be the biggest point of friction atm.
> > > >
> > > > Finally, I see a split regarding the second point:  Optional
> > > > argument to a method can be allowed if it is within a private helper
> > > method
> > > > and simplifies the code>.
> > > > There are people who have a strong opinion against allowing it.
> > > > Let's vote then for whether to allow it or not.
> > > > So far as I see we have the following votes (correct me if wrong and
> > add
> > > > more if you want):
> > > > Piotr+1
> > > > Biao+1
> > > > Timo   -1
> > > > Yu   -1
> > > > Aljoscha -1
> > > > Till  +1
> > > > Igal+1
> > > > Dawid-1
> > > > Me +1
> > > >
> > > > So far: +5 / -4 

[jira] [Created] (FLINK-13811) Support converting flink table to pandas dataframe

[Jeff Zhang (Jira)]

Jeff Zhang created FLINK-13811:
--

 Summary: Support converting flink table to pandas dataframe
 Key: FLINK-13811
 URL: https://issues.apache.org/jira/browse/FLINK-13811
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.9.0
Reporter: Jeff Zhang


Pandas dataframe is the refactor tableau data format of python community. It 
would be nice to have the ability to convert flink table to pandas dataframe.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-13810) Update `Elasticsearch Connector` label

[hehuiyuan (Jira)]

hehuiyuan created FLINK-13810:
-

 Summary: Update `Elasticsearch Connector` label 
 Key: FLINK-13810
 URL: https://issues.apache.org/jira/browse/FLINK-13810
 Project: Flink
  Issue Type: Wish
  Components: Documentation
Reporter: hehuiyuan


Sink: Streaming Append Mode
Sink: Streaming Upsert Mode
Format: JSON-only

 

The first table should be Source!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-13809) The log directory of Flink Python API is unwritable if it is installed via "sudo"

[Wei Zhong (Jira)]

Wei Zhong created FLINK-13809:
-

 Summary: The log directory of Flink Python API  is unwritable if 
it is installed via "sudo"
 Key: FLINK-13809
 URL: https://issues.apache.org/jira/browse/FLINK-13809
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.9.0
Reporter: Wei Zhong
 Fix For: 1.10.0, 1.9.1


Currently, if the python apache-flink package is installed via "sudo", an 
exception will be thrown when starting the flink python shell:
{code:java}
log4j:ERROR setFile(null,false) call failed. java.io.FileNotFoundException: 
/Library/Python/2.7/site-packages/pyflink/log/flink-zhongwei-python-zhongweideMacBook-Pro.local.log
 (Permission denied) at java.io.FileOutputStream.open0(Native Method) at 
java.io.FileOutputStream.open(FileOutputStream.java:270) at 
java.io.FileOutputStream.(FileOutputStream.java:213) at 
java.io.FileOutputStream.(FileOutputStream.java:133) at 
org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at 
org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at 
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at 
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) 
at 
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) 
at 
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
 at 
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
 at 
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
 at 
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
 at 
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
 at 
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
 at org.apache.log4j.LogManager.(LogManager.java:127) at 
org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81) at 
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at 
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at 
org.apache.flink.api.java.ExecutionEnvironment.(ExecutionEnvironment.java:102)
 at java.lang.Class.forName0(Native Method) at 
java.lang.Class.forName(Class.java:348) at 
org.apache.flink.api.python.shaded.py4j.reflection.CurrentThreadClassLoadingStrategy.classForName(CurrentThreadClassLoadingStrategy.java:40)
 at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionUtil.classForName(ReflectionUtil.java:51)
 at 
org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.forName(TypeUtil.java:243)
 at 
org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:175)
 at 
org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:87)
 at 
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
 at java.lang.Thread.run(Thread.java:748)
{code}
It does not affect the running of flink python shell but it would be better if 
we can fix it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [SURVEY] How do you use high-availability services in Flink?

[Zili Chen]

In addition, FLINK-13750[1] also likely introduce breaking change
on high-availability services. So it is highly encouraged you who
might be affected by the change share your cases :-)

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-13750


Zili Chen  于2019年8月21日周三 下午3:32写道：

> Hi guys,
>
> We want to have an accurate idea of how users actually use
> high-availability services in Flink, especially how you customize
> high-availability services by HighAvailabilityServicesFactory.
>
> Basically there are standalone impl., zookeeper impl., embedded impl.
> used in MiniCluster, YARN impl. not yet implemented, and a gate to
> customized implementations.
>
> Generally I think standalone impl. and zookeeper impl. are the most
> widely used implementations. The embedded impl. is used without
> awareness when users run a MiniCluster.
>
> Besides that, it is helpful to know how you guys customize
> high-availability services using HighAvailabilityServicesFactory
> interface for the ongoing FLINK-10333[1] which would evolve
> high-availability services in Flink. As well as whether there is any
> user take interest in the not yet implemented YARN impl..
>
> Any user case should be helpful. I really appreciate your time and your
> insight.
>
> Best,
> tison.
>
> [1] https://issues.apache.org/jira/browse/FLINK-10333
>

[jira] [Created] (FLINK-13808) Checkpoints expired by timeout may leak RocksDB files

[Julius Michaelis (Jira)]

Julius Michaelis created FLINK-13808:


 Summary: Checkpoints expired by timeout may leak RocksDB files
 Key: FLINK-13808
 URL: https://issues.apache.org/jira/browse/FLINK-13808
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.8.1, 1.8.0
 Environment: Reproduction is difficult, the only place I can reliably 
reproduce is a 4-node cluster with parallelism +>+ 100.

A semi-reliable way of reproducing the issue is provided by [this 
docker-compose|https://github.com/jcaesar/flink-rocksdb-file-leak].
Reporter: Julius Michaelis


A RocksDB state backend with HDFS checkpoints, with or without local recovery, 
may leak files in {{io.tmp.dirs}} on checkpoint expiry by timeout.

If the size of a checkpoint crosses what can be transferred during one 
checkpoint timeout, checkpoints will continue to fail forever. If this is 
combined with a quick rollover of SST files (e.g. due to a high density of 
writes), this may quickly exhaust available disk space (or memory, as /tmp is 
the default location).

As a workaround, the jobmanager's REST API can be frequently queried for failed 
checkpoints, and deleting associated files accordingly.

I've tried investing the cause a little bit, but I'm stuck:
 * {{Checkpoint 19 of job ac7efce3457d9d73b0a4f775a6ef46f8 expired before 
completing.}} and similar gets printed, so
 * [{{abortExpired}} is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L549],
 so
 * [{{dispose}} is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java#L455],
 so
 * [{{cancelCaller}} is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java#L488],
 so
 * [the canceler is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java#L497]
 ([through one more 
layer|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/state/AsyncSnapshotCallable.java#L129]),
 so
 * [{{cleanup}} is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/state/AsyncSnapshotCallable.java#L95],
 (possibly [not from 
{{cancel}}|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/state/AsyncSnapshotCallable.java#L84]),
 so
 * [{{cleanupProvidedResources}} is 
invoked|https://github.com/apache/flink/blob/release-1.8.1/flink-runtime/src/main/java/org/apache/flink/runtime/state/AsyncSnapshotCallable.java#L162]
 (this is the indirection that made me give up), so
 * [this trace 
log|https://github.com/apache/flink/blob/release-1.8.1/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L372]
 should be printed, but it isn't.

I have some time to further investigate, but I'd appreciate help on finding out 
where in this chain things go wrong.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[SURVEY] How do you use high-availability services in Flink?

[Zili Chen]

Hi guys,

We want to have an accurate idea of how users actually use
high-availability services in Flink, especially how you customize
high-availability services by HighAvailabilityServicesFactory.

Basically there are standalone impl., zookeeper impl., embedded impl.
used in MiniCluster, YARN impl. not yet implemented, and a gate to
customized implementations.

Generally I think standalone impl. and zookeeper impl. are the most
widely used implementations. The embedded impl. is used without
awareness when users run a MiniCluster.

Besides that, it is helpful to know how you guys customize
high-availability services using HighAvailabilityServicesFactory
interface for the ongoing FLINK-10333[1] which would evolve
high-availability services in Flink. As well as whether there is any
user take interest in the not yet implemented YARN impl..

Any user case should be helpful. I really appreciate your time and your
insight.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-10333

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

[Zili Chen]

Implement question: how to apply the line length rules?

If we just turn on checkstyle rule "LineLength" then a huge
effort is required to break lines those break the rule. If
we use an auto-formatter here then it possibly break line
"just at the position" awfully.

Is it possible we require only to fit the rule on the fly
a pull request touch files?

Best,
tison.


Yu Li  于2019年8月20日周二 下午5:22写道：

> I second Stephan's summarize, and to be more explicit, +1 on:
> - Set a hard line length limit
> - Allow arguments on the same line if below length limit
> - With consistent argument breaking when that length is exceeded
> - Developers can break before that if they feel it helps with readability
>
> FWIW, hbase project also sets the line length limit to 100 [1] (personally
> I don't have any tendency on this, so JFYI).
>
> [1]
>
> https://github.com/apache/hbase/blob/a59f7d4ffc27ea23b9822c3c26d6aeb76ccdf9aa/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml#L128
>
> Best Regards,
> Yu
>
>
> On Mon, 19 Aug 2019 at 18:22, Stephan Ewen  wrote:
>
> > I personally prefer not to break lines with few parameters.
> > It just feels unnecessarily clumsy to parse the breaks if there are only
> > two or three arguments with short names.
> >
> > So +1
> >   - for a hard line length limit
> >   - allowing arguments on the same line if below that limit
> >   - with consistent argument breaking when that length is exceeded
> >   - developers can break before that if they feel it helps with
> > readability.
> >
> > This should be similar to what we have, except for enforcing the line
> > length limit.
> >
> > I think our Java guide originally suggested 120 characters line length,
> but
> > we can reduce that to 100 if a majority argues for that, but it is
> separate
> > discussion.
> > We uses shorter lines in Scala (100 chars) because Scala code becomes
> > harder to read faster with long lines.
> >
> >
> > On Mon, Aug 19, 2019 at 10:45 AM Andrey Zagrebin 
> > wrote:
> >
> > > Hi Everybody,
> > >
> > > Thanks for your feedback guys and sorry for not getting back to the
> > > discussion for some time.
> > >
> > > @SHI Xiaogang
> > > About breaking lines for thrown exceptions:
> > > Indeed that would prevent growing the throw clause indefinitely.
> > > I am a bit concerned about putting the right parenthesis and/or throw
> > > clause on the next line
> > > because in general we do not it and there are a lot of variations of
> how
> > > and what to put to the next line so it needs explicit memorising.
> > > Also, we do not have many checked exceptions and usually avoid them.
> > > Although I am not a big fan of many function arguments either but this
> > > seems to be a bigger problem in the code base.
> > > I would be ok to not enforce anything for exceptions atm.
> > >
> > > @Chesnay Schepler 
> > > Thanks for mentioning automatic checks.
> > > Indeed, pointing out this kind of style issues during PR reviews is
> very
> > > tedious
> > > and cannot really force it without automated tools.
> > > I would still consider the outcome of this discussion as a soft
> > > recommendation atm (which we also have for some other things in the
> code
> > > style draft).
> > > We need more investigation about how to enforce things. I am not so
> > > knowledgable about code style/IDE checks.
> > > From the first glance I also do not see a simple way. If somebody has
> > more
> > > insight please share your experience.
> > >
> > > @Biao Liu 
> > > Line length limitation:
> > > I do not see anything for Java, only for Scala: 100 (also enforced by
> > build
> > > AFAIK).
> > > From what I heard there has been already some discussion about the hard
> > > limit for the line length.
> > > Although quite some people are in favour of it (including me) and seems
> > to
> > > be a nice limitation,
> > > there are some practical implication about it.
> > > Historically, Flink did not have any code style checks and huge chunks
> of
> > > code base have to be reformatted destroying the commit history.
> > > Another thing is value for the limit. Nowadays, we have wide screens
> and
> > do
> > > not often even need to scroll.
> > > Nevertheless, we can kick off another discussion about the line length
> > > limit and enforcing it.
> > > Atm I see people to adhere to a soft recommendation of 120 line length
> > for
> > > Java because it is usually a bit more verbose comparing to Scala.
> > >
> > > *Question 1*:
> > > I would be ok to always break line if there is more than one chained
> > call.
> > > There are a lot of places where this is only one short call I would not
> > > break line in this case.
> > > If it is too confusing I would be ok to stick to the rule to break
> either
> > > all or none.
> > > Thanks for pointing out this explicitly: For a chained method calls,
> the
> > > new line should be started with the dot.
> > > I think it should be also part of the rule if forced.
> > >
> > > *Question 2:*
> > > The indent of new line should be 1 

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[Bowen Li]

+1 non-binding

- built from source with default profile
- manually ran SQL and Table API tests for Flink's metadata integration
with Hive Metastore in local cluster
- manually ran SQL tests for batch capability with Blink planner and Hive
integration (source/sink/udf) in local cluster
- file formats include: csv, orc, parquet


On Tue, Aug 20, 2019 at 10:23 PM Gary Yao  wrote:

> +1 (non-binding)
>
> Reran Jepsen tests 10 times.
>
> On Wed, Aug 21, 2019 at 5:35 AM vino yang  wrote:
>
> > +1 (non-binding)
> >
> > - checkout source code and build successfully
> > - started a local cluster and ran some example jobs successfully
> > - verified signatures and hashes
> > - checked release notes and post
> >
> > Best,
> > Vino
> >
> > Stephan Ewen  于2019年8月21日周三 上午4:20写道：
> >
> > > +1 (binding)
> > >
> > >  - Downloaded the binary release tarball
> > >  - started a standalone cluster with four nodes
> > >  - ran some examples through the Web UI
> > >  - checked the logs
> > >  - created a project from the Java quickstarts maven archetype
> > >  - ran a multi-stage DataSet job in batch mode
> > >  - killed as TaskManager and verified correct restart behavior,
> including
> > > failover region backtracking
> > >
> > >
> > > I found a few issues, and a common theme here is confusing error
> > reporting
> > > and logging.
> > >
> > > (1) When testing batch failover and killing a TaskManager, the job
> > reports
> > > as the failure cause "org.apache.flink.util.FlinkException: The
> assigned
> > > slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
> > > I think that is a pretty bad error message, as a user I don't know
> > what
> > > that means. Some internal book keeping thing?
> > > You need to know a lot about Flink to understand that this means
> > > "TaskManager failure".
> > > https://issues.apache.org/jira/browse/FLINK-13805
> > > I would not block the release on this, but think this should get
> > pretty
> > > urgent attention.
> > >
> > > (2) The Metric Fetcher floods the log with error messages when a
> > > TaskManager is lost.
> > >  There are many exceptions being logged by the Metrics Fetcher due
> to
> > > not reaching the TM any more.
> > >  This pollutes the log and drowns out the original exception and
> the
> > > meaningful logs from the scheduler/execution graph.
> > >  https://issues.apache.org/jira/browse/FLINK-13806
> > >  Again, I would not block the release on this, but think this
> should
> > > get pretty urgent attention.
> > >
> > > (3) If you put "web.submit.enable: false" into the configuration, the
> web
> > > UI will still display the "SubmitJob" page, but errors will
> > > continuously pop up, stating "Unable to load requested file /jars."
> > > https://issues.apache.org/jira/browse/FLINK-13799
> > >
> > > (4) REST endpoint logs ERROR level messages when selecting the
> > > "Checkpoints" tab for batch jobs. That does not seem correct.
> > >  https://issues.apache.org/jira/browse/FLINK-13795
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > >
> > >
> > > On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Legal checks:
> > > > - verified signatures and hashes
> > > > - New bundled Javascript dependencies for flink-runtime-web are
> > correctly
> > > > reflected under licenses-binary and NOTICE file.
> > > > - locally built from source (Scala 2.12, without Hadoop)
> > > > - No missing artifacts in staging repo
> > > > - No binaries in source release
> > > >
> > > > Functional checks:
> > > > - Quickstart working (both in IDE + job submission)
> > > > - Simple State Processor API program that performs offline key schema
> > > > migration (RocksDB backend). Generated savepoint is valid to restore
> > > from.
> > > > - All E2E tests pass locally
> > > > - Didn’t notice any issues with the new WebUI
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > On Tue, Aug 20, 2019 at 3:53 AM Zili Chen 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - build from source: OK(8u212)
> > > > > - check local setup tutorial works as expected
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > Yu Li  于2019年8月20日周二 上午8:24写道：
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - checked release notes: OK
> > > > > > - checked sums and signatures: OK
> > > > > > - repository appears to contain all expected artifacts
> > > > > > - source release
> > > > > >  - contains no binaries: OK
> > > > > >  - contains no 1.9-SNAPSHOT references: OK
> > > > > >  - build from source: OK (8u102)
> > > > > > - binary release
> > > > > >  - no examples appear to be missing
> > > > > >  - started a cluster; WebUI reachable, example ran
> successfully
> > > > > > - checked README.md file and found nothing unexpected
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 20 Aug 2019 at 01:16,