[jira] [Created] (FLINK-7123) Support timesOrMore in CEP

[Dian Fu (JIRA)]

Dian Fu created FLINK-7123:
--

 Summary: Support timesOrMore in CEP
 Key: FLINK-7123
 URL: https://issues.apache.org/jira/browse/FLINK-7123
 Project: Flink
  Issue Type: Sub-task
  Components: CEP
Reporter: Dian Fu
Assignee: Dian Fu


The CEP API should provide API such as timesOrMore(#ofTimes) for quantifier 
{n,}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Make SubmittedJobGraphStore configurable

[Chen Qin]

Sure,
​ I would imagine ​couple of extra lines within flink.conf
​...​
graphstore.type: customized/zookeeper
graphstore.class:
​org​
.
​apache.flink.contrib​
.MyS3SubmittedJobGraphStoreImp
graphstore.endpoint: s3.amazonaws.com
graphstore.path.root: s3://my root/

which overwrites initiation of

*org.apache.flink.runtime.highavailability​.​HighAvailabilityServices*

/**
* Gets the submitted job graph store for the job manager
*
* @return Submitted job graph store
* @throws Exception if the submitted job graph store could not be created
*/

SubmittedJobGraphStore *getSubmittedJobGraphStore*() throws Exception;

In this case, user implemented their own s3 backed job graph store and
stores job graphs in s3 instead of zookeeper(high availability) or
never(nonha)

​I find [1] is somehow related and more focus on life cycle and dependency
aspect of graph-store and checkpoint-store. FLINK-7106 in this case limited
to enable user implemented their own jobgraphstore instead of hardcoded to
zookeeper.

Thanks,
Chen​


[1] https://issues.apache.org/jira/browse/FLINK-6626


On Thu, Jul 6, 2017 at 2:47 AM, Ted Yu  wrote:

> The sample config entries are broken into multiple lines.
>
> Can you send the config again with one config on one line ?
>
> Cheers
>
> On Wed, Jul 5, 2017 at 10:19 PM, Chen Qin  wrote:
>
> > ​Hi there,
> >
> > ​I would like to propose/discuss median level refactor work to make
> > submittedJobGraphStore configurable and extensible.
> >
> > The rationale behind is to allow users offload those meta data to durable
> > cross dc read after write strong consistency storage and decouple with zk
> > quorum.
> > ​
> >
> > https://issues.apache.org/jira/browse/FLINK-7106
> >
> > 
> > New configurable setting in flink.conf
> > ​ looks like following
> >
> > g​
> > raph
> > ​-s
> > tore:
> > ​customized/zookeeper
> > g​
> > raph
> > ​-s
> > tore​.class: xx.yy.MyS3SubmittedJobGraphStore​Imp
> >
> > g​
> > raph
> > ​-s
> > tore.
> > ​endpoint
> > : s3.amazonaws.com
> > g​
> > raph
> > ​-s
> > tore.path.root:
> > ​s3:/
> > ​
> > /
> > ​my root/​
> >
> > Thanks,
> > Chen
> >
>

Re: How to set jobmanager.rpc.address in TaskManger node in HA cluster

[Mu Kong]

Sorry. I didn't put high-availability: zookeeper in taskmangers'
flink-config.yml.
After I fixed this, everything went well.

On Fri, Jul 7, 2017 at 11:08 AM, Mu Kong  wrote:

> Hi all,
>
> I'm trying setup an HA Flink cluster with 3 job managers and 3 task
> managers.
> After executing start-cluster.sh, 3 job managers are normally started.
> However, the task managers are still down due to an exception showed below:
>
> org.apache.flink.util.ConfigurationException: Config parameter 'Key:
> 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing
> (hostname/address of JobManager to connect to).
>
>
> I managed to setup a non-HA cluster before and I know I should put the Job
> manager's address here, but what should I put here in HA cluster since
> there are three job managers here.
>
>
> Besides, according to the original config file here:
>
> https://github.com/apache/flink/blob/master/flink-dist/
> src/main/resources/flink-conf.yaml#L28
>
> this should be taken care of automatically.
>
> I have already put masters under /conf folder with all the job managers'
> addresses/ports in it. So the exception shouldn't have shown in the first
> place.
>
>
> Is there anything else I have missed?
>
>
> Thanks in advance.
>
> Mu
>
>
>

[jira] [Created] (FLINK-7122) When metric name contains special characters then Graphite will not display them

[wyp (JIRA)]

wyp created FLINK-7122:
--

 Summary: When metric name contains special characters then 
Graphite will not display them
 Key: FLINK-7122
 URL: https://issues.apache.org/jira/browse/FLINK-7122
 Project: Flink
  Issue Type: Bug
  Components: Metrics
Affects Versions: 1.3.1
Reporter: wyp
Priority: Minor


When metric name contains special characters then Graphite will not display 
them, because Graphite(GraphiteUDP) only replace white space to 
{{-}}(See:[GraphiteUDP.java|https://github.com/dropwizard/metrics/blob/3.1-maintenance/metrics-graphite/src/main/java/com/codahale/metrics/graphite/GraphiteUDP.java#L109]).
 But Flink metric variable's associated value like this:
{code}
 -> bcab9cc1c21d8ac83f721ef908391755
 -> cbc357ccb763df2852fee8c4fc7d55f2
 -> 2348e14acc555848fea3f9bc8c06a4f6
 -> l-hdps775
 -> Sink: es
 -> Source: Custom Source -> Flat Map -> Sink: es
 -> 0
 -> BinlogSyn
 -> 49870456e42ebab1a3b4afaf2ad1f5d8
 -> 0
{code}
then {{metrics.scope.task}} 's full name will be
{{flinkjobs.localhost.taskmanager.34d31d54c0031328a6ec8910e571a7f8.MyFlinkJobs.Source:
 Custom Source -> TestFlat -> Map -> Sink: es.0}}, but this will not be display 
in Graphite, because Graphite didn't support {{:}},{{>}} etc. special 
characters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7121) Remove the hardcoded way in core

[mingleizhang (JIRA)]

mingleizhang created FLINK-7121:
---

 Summary: Remove the hardcoded way in core
 Key: FLINK-7121
 URL: https://issues.apache.org/jira/browse/FLINK-7121
 Project: Flink
  Issue Type: Bug
  Components: Core
Reporter: mingleizhang
Assignee: mingleizhang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

How to set jobmanager.rpc.address in TaskManger node in HA cluster

[Mu Kong]

Hi all,

I'm trying setup an HA Flink cluster with 3 job managers and 3 task
managers.
After executing start-cluster.sh, 3 job managers are normally started.
However, the task managers are still down due to an exception showed below:

org.apache.flink.util.ConfigurationException: Config parameter 'Key:
'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing
(hostname/address of JobManager to connect to).


I managed to setup a non-HA cluster before and I know I should put the Job
manager's address here, but what should I put here in HA cluster since
there are three job managers here.


Besides, according to the original config file here:

https://github.com/apache/flink/blob/master/flink-dist/src/main/resources/flink-conf.yaml#L28

this should be taken care of automatically.

I have already put masters under /conf folder with all the job managers'
addresses/ports in it. So the exception shouldn't have shown in the first
place.


Is there anything else I have missed?


Thanks in advance.

Mu

[jira] [Created] (FLINK-7120) Remove the hardcoded way in Utils

[mingleizhang (JIRA)]

mingleizhang created FLINK-7120:
---

 Summary: Remove the hardcoded way in Utils
 Key: FLINK-7120
 URL: https://issues.apache.org/jira/browse/FLINK-7120
 Project: Flink
  Issue Type: Bug
  Components: YARN
Reporter: mingleizhang
Assignee: mingleizhang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7119) Remove the hardcoded way in HadoopUtils

[mingleizhang (JIRA)]

mingleizhang created FLINK-7119:
---

 Summary: Remove the hardcoded way in HadoopUtils
 Key: FLINK-7119
 URL: https://issues.apache.org/jira/browse/FLINK-7119
 Project: Flink
  Issue Type: Bug
  Components: Java API
Reporter: mingleizhang
Assignee: mingleizhang


I can see this kind of hardcoded way in {{HadoopUtils}}.

{code:java}
clazz = Class.forName("org.apache.hadoop.mapred.JobContextImpl", true, 
Thread.currentThread().getContextClassLoader());
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7118) Remove hadoop1.x code in HadoopUtils

[mingleizhang (JIRA)]

mingleizhang created FLINK-7118:
---

 Summary: Remove hadoop1.x code in HadoopUtils
 Key: FLINK-7118
 URL: https://issues.apache.org/jira/browse/FLINK-7118
 Project: Flink
  Issue Type: Bug
  Components: Java API
Reporter: mingleizhang
Assignee: mingleizhang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7117) Add JobManagerLeaderRetrieval method with default JobManager address

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7117:


 Summary: Add JobManagerLeaderRetrieval method with default 
JobManager address
 Key: FLINK-7117
 URL: https://issues.apache.org/jira/browse/FLINK-7117
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Reporter: Till Rohrmann
Assignee: Till Rohrmann


The {{HighAvailabilityServices}} should have a method 
{{getJobManagerLeaderRetriever}} which can be called with a {{JobID}} and a 
default {{JobManager}} address. This is a requirement for Flip-6 where in the 
{{StandaloneHaServices}} case, we need to dynamically create a 
{{LeaderRetrievalService}} with the given {{JobManager}} address.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7116) Add getPort method to RpcService

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7116:


 Summary: Add getPort method to RpcService
 Key: FLINK-7116
 URL: https://issues.apache.org/jira/browse/FLINK-7116
 Project: Flink
  Issue Type: Sub-task
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor


In order to connect to a remote {{RpcService}}, it should expose the port it is 
bound to.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] FLIP proposal for Model Serving over Flink

[Stavros Kontopoulos]

Thanx Fabian! I asked Robert for that already, but I will let you know then
:)

Best,
Stavros

On Thu, Jul 6, 2017 at 10:10 PM, Fabian Hueske  wrote:

> Great, thank you Theodore!
>
> @Stavros Let me know when you need edit permissions to add the FLIP to the
> wiki.
>
> Cheers,
> Fabian
>
> 2017-07-06 2:33 GMT+02:00 Theodore Vasiloudis <
> theodoros.vasilou...@gmail.com>:
>
> > Hello all,
> >
> > I just wanted to indicate that I would be very willing to help out with
> > code reviews
> > for this project and participating in design discussions.
> >
> > But I should note that I don' think I'll have time to contribute code
> until
> > I get back to Stockholm in September.
> >
> > Regards,
> > Theodore
> >
> > On Tue, Jul 4, 2017 at 9:41 AM, Andrea Spina  >
> > wrote:
> >
> > > Hi all,
> > > yes, we did too. We - from Radicalbit - have submitted a talk focused
> > > on the recently released flink-jpmml library about model serving.
> > > Lately, it became part of the FlinkML project.
> > >
> > > Cheers, Andrea
> > >
> > > 2017-07-04 16:14 GMT+02:00 Boris Lublinsky <
> > boris.lublin...@lightbend.com
> > > >:
> > > > Yes,
> > > > I submitted a talk with Stavros on model serving
> > > >
> > > >
> > > > Boris Lublinsky
> > > > FDP Architect
> > > > boris.lublin...@lightbend.com
> > > > https://www.lightbend.com/
> > > >
> > > > On Jul 3, 2017, at 1:18 PM, Robert Metzger 
> > wrote:
> > > >
> > > > Big +1 from my side on getting this effort started.
> > > >
> > > > Users have asked for this and I would like to see some progress
> there.
> > > > Did anybody submit a talk about the ML efforts to Flink Forward
> Berlin
> > > this
> > > > year?
> > > >
> > > > On Fri, Jun 30, 2017 at 6:04 PM, Fabian Hueske 
> > > wrote:
> > > >>
> > > >> Yes, I know that Theo is engaged in the ML efforts but wasn't sure
> how
> > > >> much
> > > >> he is involved in the model serving part (thought he was more into
> the
> > > >> online learning part).
> > > >> It would be great if Theo could help here!
> > > >>
> > > >> I just wanted to make sure that we find somebody to help
> > bootstrapping.
> > > >>
> > > >> Cheers, Fabian
> > > >>
> > > >>
> > > >> 2017-06-30 17:52 GMT+02:00 Stavros Kontopoulos <
> > > st.kontopou...@gmail.com>:
> > > >>
> > > >> > Hi Fabian,
> > > >> >
> > > >> > However, we should keep in mind that we need a committer to
> > bootstrap
> > > >> > the
> > > >> > > new module.
> > > >> >
> > > >> >
> > > >> > Absolutely I thought Theodore Vassiloudis could help, as an
> initial
> > > >> > committer.
> > > >> > Is this known? He is part of the effort btw.
> > > >> >
> > > >> > Best,
> > > >> > Stavros
> > > >> >
> > > >> > On Fri, Jun 30, 2017 at 6:42 PM, Fabian Hueske  >
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks Stavros (and everybody else involved) for starting this
> > > effort
> > > >> > > and
> > > >> > > bringing the discussion back to the mailing list.
> > > >> > >
> > > >> > > As I said before, a model serving module/component would be a
> > great
> > > >> > feature
> > > >> > > for Flink.
> > > >> > > I see the biggest advantage for such a module in the integration
> > > with
> > > >> > > the
> > > >> > > other APIs and libraries, such as DataStream, CEP, SQL.
> > > >> > >
> > > >> > > A FLIP would be a great way to continue your efforts and work
> on a
> > > >> > > design
> > > >> > > for the component.
> > > >> > >
> > > >> > > However, we should keep in mind that we need a committer to
> > > bootstrap
> > > >> > > the
> > > >> > > new module.
> > > >> > > As people are contributing to the model serving module, the
> number
> > > of
> > > >> > > committers should hopefully grow after some time.
> > > >> > >
> > > >> > > Best, Fabian
> > > >> > >
> > > >> > > 2017-06-30 10:58 GMT+02:00 Stavros Kontopoulos
> > > >> > >  > > >> > >:
> > > >> > >
> > > >> > > > Hi all,
> > > >> > > >
> > > >> > > > After coordinating with Theodore Vasiloudis and the guys
> behind
> > > the
> > > >> > Flink
> > > >> > > > Model Serving effort (Eron, Radicalbit people, Boris, Bas
> > (ING)),
> > > we
> > > >> > > > propose to start working on the model serving over Flink in a
> > more
> > > >> > > official
> > > >> > > > way.
> > > >> > > > That translates to capturing design details in a FLIP
> document.
> > > >> > > >
> > > >> > > > Please let's discuss and vote whether you think this FLIP
> would
> > be
> > > >> > > viable.
> > > >> > > >
> > > >> > > > Model Serving as a Flink component might involve a lot of work
> > and
> > > >> > > > we
> > > >> > > need
> > > >> > > > to commit to support it in future Flink releases.
> > > >> > > >
> > > >> > > > In the mean time a lot of people have joined Flink ml slack
> > > channel
> > > >> > > > (
> > > >> > > > https://flinkml.slack.com, https://flinkml-invites.
> > herokuapp.com/
> > > )
> > > >> > and I
> > > >> > > > think its time to try get them gradually on board.
> > > >> > > >
> > > >> > > > So far we have several efforts hosted here:
> > > >> > > > https://github.

Re: Tips to fix IDEA strange problem after updating master code

[Greg Hogan]

I’m wondering if we can remove the ILoopCompat duplication by checking and 
reading the trait properties with reflection … but I have not discovered how to 
do this.


> On Jul 4, 2017, at 3:57 AM, Piotr Nowojski  wrote:
> 
> Besides deactivating “scala-2.10” profile in the Intellij it might be 
> necessary to:
> - reimport maven project:
>   1. Right click on root module: “flink-parent”
>   2. Maven
>   3. reimport
> - invalidate caches and restart: File -> Invalidate caches and restart -> 
> invalidate /restart
> - rebuild whole project
> 
> I suspect that either activation of scala-2.10 by default comes from 
> flink-scala and flick-scala-shell poms or it’s an artifact because you 
> created/imported Intellij project when 2.10 was the default. If the first 
> option is true, this PR: https://github.com/apache/flink/pull/4240 
>  might fix this issue.
> 
> 
> Another quirk that I encauntered is the compile error about  ILoopCompat 
> class being defined twice in Intellij (works fine from console). This comes 
> from flink-scala-shell/pom.xml, which defines two different source paths 
> depending on Scala version:
> 
> src/main/scala-${scala.binary.version}
> 
> Such thing is not supported by Intellij and one have to manually remove one 
> of the source directory (either 2.11 or 2.10) from the project settings.
> 
> Piotrek
> 
>> On Jul 4, 2017, at 9:46 AM, Aljoscha Krettek  wrote:
>> 
>> Thanks for the hint!
>> 
>>> On 4. Jul 2017, at 06:03, Ted Yu  wrote:
>>> 
>>> Looks like the picture didn't go thru.
>>> 
>>> Mind using third party site ?
>>> 
>>> Thanks
>>> 
>>> On Mon, Jul 3, 2017 at 8:56 PM, Jark Wu  wrote:
>>> 
 Hi devs,
 
 Yesterday, I updated the master code which include [FLINK-7030]: Build
 with scala-2.11 by default. After that, I entered a strange problem with
 IDEA that many classes can't be found, and the project can't be
 built/compiled (in IDEA), but maven install worked good.
 
 After a series of attempts, I found that IDEA activate the scala-2.10
 profile by default which result in this problem. After deactivate
 scala-2.10 profile via  sidebar Maven Projects -> Profiles -> deactivate
 "scala-2.10" profile, and every works good again.
 
 [image: 内嵌图片 1]
 
 I share this tip in the dev list, because a lot of my colleagues have the
 same issues, and maybe many other Flink devs have the same problem too.
 
 BTW, I don't know why IDEA activate scala-2.10 by default, not sure it's a
 IDEA bug or the wrong profile setting somewhere.
 
 
 Regards,
 Jark Wu
 
>> 
> 

Re: [DISCUSS] FLIP proposal for Model Serving over Flink

[Fabian Hueske]

Great, thank you Theodore!

@Stavros Let me know when you need edit permissions to add the FLIP to the
wiki.

Cheers,
Fabian

2017-07-06 2:33 GMT+02:00 Theodore Vasiloudis <
theodoros.vasilou...@gmail.com>:

> Hello all,
>
> I just wanted to indicate that I would be very willing to help out with
> code reviews
> for this project and participating in design discussions.
>
> But I should note that I don' think I'll have time to contribute code until
> I get back to Stockholm in September.
>
> Regards,
> Theodore
>
> On Tue, Jul 4, 2017 at 9:41 AM, Andrea Spina 
> wrote:
>
> > Hi all,
> > yes, we did too. We - from Radicalbit - have submitted a talk focused
> > on the recently released flink-jpmml library about model serving.
> > Lately, it became part of the FlinkML project.
> >
> > Cheers, Andrea
> >
> > 2017-07-04 16:14 GMT+02:00 Boris Lublinsky <
> boris.lublin...@lightbend.com
> > >:
> > > Yes,
> > > I submitted a talk with Stavros on model serving
> > >
> > >
> > > Boris Lublinsky
> > > FDP Architect
> > > boris.lublin...@lightbend.com
> > > https://www.lightbend.com/
> > >
> > > On Jul 3, 2017, at 1:18 PM, Robert Metzger 
> wrote:
> > >
> > > Big +1 from my side on getting this effort started.
> > >
> > > Users have asked for this and I would like to see some progress there.
> > > Did anybody submit a talk about the ML efforts to Flink Forward Berlin
> > this
> > > year?
> > >
> > > On Fri, Jun 30, 2017 at 6:04 PM, Fabian Hueske 
> > wrote:
> > >>
> > >> Yes, I know that Theo is engaged in the ML efforts but wasn't sure how
> > >> much
> > >> he is involved in the model serving part (thought he was more into the
> > >> online learning part).
> > >> It would be great if Theo could help here!
> > >>
> > >> I just wanted to make sure that we find somebody to help
> bootstrapping.
> > >>
> > >> Cheers, Fabian
> > >>
> > >>
> > >> 2017-06-30 17:52 GMT+02:00 Stavros Kontopoulos <
> > st.kontopou...@gmail.com>:
> > >>
> > >> > Hi Fabian,
> > >> >
> > >> > However, we should keep in mind that we need a committer to
> bootstrap
> > >> > the
> > >> > > new module.
> > >> >
> > >> >
> > >> > Absolutely I thought Theodore Vassiloudis could help, as an initial
> > >> > committer.
> > >> > Is this known? He is part of the effort btw.
> > >> >
> > >> > Best,
> > >> > Stavros
> > >> >
> > >> > On Fri, Jun 30, 2017 at 6:42 PM, Fabian Hueske 
> > >> > wrote:
> > >> >
> > >> > > Thanks Stavros (and everybody else involved) for starting this
> > effort
> > >> > > and
> > >> > > bringing the discussion back to the mailing list.
> > >> > >
> > >> > > As I said before, a model serving module/component would be a
> great
> > >> > feature
> > >> > > for Flink.
> > >> > > I see the biggest advantage for such a module in the integration
> > with
> > >> > > the
> > >> > > other APIs and libraries, such as DataStream, CEP, SQL.
> > >> > >
> > >> > > A FLIP would be a great way to continue your efforts and work on a
> > >> > > design
> > >> > > for the component.
> > >> > >
> > >> > > However, we should keep in mind that we need a committer to
> > bootstrap
> > >> > > the
> > >> > > new module.
> > >> > > As people are contributing to the model serving module, the number
> > of
> > >> > > committers should hopefully grow after some time.
> > >> > >
> > >> > > Best, Fabian
> > >> > >
> > >> > > 2017-06-30 10:58 GMT+02:00 Stavros Kontopoulos
> > >> > >  > >> > >:
> > >> > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > After coordinating with Theodore Vasiloudis and the guys behind
> > the
> > >> > Flink
> > >> > > > Model Serving effort (Eron, Radicalbit people, Boris, Bas
> (ING)),
> > we
> > >> > > > propose to start working on the model serving over Flink in a
> more
> > >> > > official
> > >> > > > way.
> > >> > > > That translates to capturing design details in a FLIP document.
> > >> > > >
> > >> > > > Please let's discuss and vote whether you think this FLIP would
> be
> > >> > > viable.
> > >> > > >
> > >> > > > Model Serving as a Flink component might involve a lot of work
> and
> > >> > > > we
> > >> > > need
> > >> > > > to commit to support it in future Flink releases.
> > >> > > >
> > >> > > > In the mean time a lot of people have joined Flink ml slack
> > channel
> > >> > > > (
> > >> > > > https://flinkml.slack.com, https://flinkml-invites.
> herokuapp.com/
> > )
> > >> > and I
> > >> > > > think its time to try get them gradually on board.
> > >> > > >
> > >> > > > So far we have several efforts hosted here:
> > >> > > > https://github.com/FlinkML
> > >> > > >
> > >> > > > Related documents for what we are doing:
> > >> > > >
> > >> > > > Flink ML roadmap
> > >> > > >
> > >> > > > https://docs.google.com/document/d/
> 1afQbvZBTV15qF3vobVWUjxQc49h3U
> > >> > > > d06MIRhahtJ6dw/edit
> > >> > > >
> > >> > > > Flink MS
> > >> > > >
> > >> > > > https://docs.google.com/document/d/
> 1CjWL9aLxPrKytKxUF5c3ohs0ickp0
> > >> > > > fdEXPsPYPEywsE/edit#
> > >> > > >
> > >> > > > PS. I will work on the last document the next few

[jira] [Created] (FLINK-7115) test instability in MiniClusterITCase.runJobWithMultipleJobManagers

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-7115:
--

 Summary: test instability in 
MiniClusterITCase.runJobWithMultipleJobManagers
 Key: FLINK-7115
 URL: https://issues.apache.org/jira/browse/FLINK-7115
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.4.0
Reporter: Nico Kruber


In a test run with unrelated changes, I to have one case of 
{{MiniClusterITCase}} hanging:

https://s3.amazonaws.com/archive.travis-ci.org/jobs/250775808/log.txt?X-Amz-Expires=30&X-Amz-Date=20170706T151556Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170706/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5b7c512137c7cbd82dcb77a98aeadc3d761b7055bea6d8f07ad6b786dc196f37

{code}
==
Maven produced no output for 300 seconds.
==
==
The following Java processes are running (JPS)
==
12852 Jps
9166 surefirebooter1705381973603203163.jar
4966 Launcher
==
Printing stack trace of Java process 12865
==
12865: No such process
==
Printing stack trace of Java process 9166
==
2017-07-06 15:05:52
Full thread dump OpenJDK 64-Bit Server VM (24.131-b00 mixed mode):

"Attach Listener" daemon prio=10 tid=0x7fc520b1a000 nid=0x3266 waiting on 
condition [0x]
   java.lang.Thread.State: RUNNABLE

"flink-akka.actor.default-dispatcher-9" daemon prio=10 tid=0x7fc520b0e800 
nid=0x23fd waiting on condition [0x7fc51abcb000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007a0ca2c78> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akka.actor.default-dispatcher-8" daemon prio=10 tid=0x7fc520bb9800 
nid=0x23fc waiting on condition [0x7fc51aaca000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007a0ca2c78> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"Flink-MetricRegistry-1" prio=10 tid=0x7fc520ba7800 nid=0x23f9 waiting on 
condition [0x7fc51a4c4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007a09699c8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1092)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"flink-akka.actor.default-dispatcher-7" daemon prio=10 tid=0x7fc520b9d800 
nid=0x23f7 waiting on condition [0x7fc51a6c6000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007a0ca2c78> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"flink-akk

[jira] [Created] (FLINK-7114) Remove the hardcoded way in flink

[mingleizhang (JIRA)]

mingleizhang created FLINK-7114:
---

 Summary: Remove the hardcoded way in flink
 Key: FLINK-7114
 URL: https://issues.apache.org/jira/browse/FLINK-7114
 Project: Flink
  Issue Type: Bug
Reporter: mingleizhang
Assignee: mingleizhang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re:[DISCUSS]refactor StreamConfig

[Xu Pingyong]

Hi Aljoscha:
  
   Great. I cannot agree with you more. So I introduce OperatorSettings and 
OperatorProperties.


   StreamTaskConfig relys on the underlying configuration and is provided for 
the streamTask to use. It contains:
 1) in.physical.edges
 2) out.physical.edges
 3) chained OperatorSettings
 4) execution environment: checkpoint, state.backend and so on... 


   OperatorSettings is serialisable and stores things that are tied to one 
operator within the chain. It is provided for the streamTask to build an 
operator. It contains:
   1)  operator information: name, id
   2)  streamOperator
   3)  input serializer.
   4)  output edges and serializers.
   5)  s.chain.start, is.chain.end
   6)  state.key.serializer


   OperatorProperties can be an interface to view few things of 
OperatorSettings, it is provided for an operator to setup, it contains:
   1)  operator information: name, id
   2)  input serializer.
   3)  is.chain.start, is.chain.end  (existed now, maybe moved later)
   4)  state.key.serializer


What do you think?


 Best Regards!
Xu Pingyong

Re: Tips to fix IDEA strange problem after updating master code

[Piotr Nowojski]

First of all by activating scala-2.10/2.11 profile we change two properties to 
some predefined values:

2.10.6

2.10

To drop profile for that, one would have to overwrite both of those properties 
simultaneously. 

Secondly yes, we have different dependencies between Scala-2.10 and Scala-2.11 
(flink-scala-shell and flink-scala modules). 

One more thing is that once we add Kafka 0.11 connector, changing Scala profile 
will also have to add/remove flink-connector-kafka-0.11 module. I’m not sure if 
that could be done without profiles.

Piotrek

> On Jul 6, 2017, at 4:01 PM, Stephan Ewen  wrote:
> 
> Is it possible to not use a profile for that, but only an actual property
> variable?
> Or do we use different dependencies with Scala 2.10 vs Scala 2.11 ?
> 
> 
> On Tue, Jul 4, 2017 at 11:23 AM, 郭健  wrote:
> 
>> After deactivating scala-2.10 profile in IntelliJ, this issue is gone.
>> Thank you all.
>> 
>> On 7/4/17, 17:11, "Piotr Nowojski"  wrote:
>> 
>>Maybe try
>> 
>>$ mvn clean
>> 
>>Before reimporting and restarting/invalidating caches in IntelliJ? Did
>> you deactivate scala-2.10 profile in the IntelliJ?
>> 
>>Piotrek
>> 
>> 
>> 
>>> On Jul 4, 2017, at 11:05 AM, 郭健  wrote:
>>> 
>>> I have done all these but still got some issue in IDEA, especially
>> in the flink-connector project.
>>> 
>>> 
>>> On 7/4/17, 15:57, "Piotr Nowojski"  wrote:
>>> 
>>>   Besides deactivating “scala-2.10” profile in the Intellij it
>> might be necessary to:
>>>   - reimport maven project:
>>>  1. Right click on root module: “flink-parent”
>>>  2. Maven
>>>  3. reimport
>>>   - invalidate caches and restart: File -> Invalidate caches and
>> restart -> invalidate /restart
>>>   - rebuild whole project
>>> 
>>>   I suspect that either activation of scala-2.10 by default comes
>> from flink-scala and flick-scala-shell poms or it’s an artifact because you
>> created/imported Intellij project when 2.10 was the default. If the first
>> option is true, this PR: https://github.com/apache/flink/pull/4240 <
>> https://github.com/apache/flink/pull/4240> might fix this issue.
>>> 
>>> 
>>>   Another quirk that I encauntered is the compile error about
>> ILoopCompat class being defined twice in Intellij (works fine from
>> console). This comes from flink-scala-shell/pom.xml, which defines two
>> different source paths depending on Scala version:
>>> 
>>>   src/main/scala-${scala.binary.version}
>>> 
>>>   Such thing is not supported by Intellij and one have to manually
>> remove one of the source directory (either 2.11 or 2.10) from the project
>> settings.
>>> 
>>>   Piotrek
>>> 
 On Jul 4, 2017, at 9:46 AM, Aljoscha Krettek 
>> wrote:
 
 Thanks for the hint!
 
> On 4. Jul 2017, at 06:03, Ted Yu  wrote:
> 
> Looks like the picture didn't go thru.
> 
> Mind using third party site ?
> 
> Thanks
> 
> On Mon, Jul 3, 2017 at 8:56 PM, Jark Wu  wrote:
> 
>> Hi devs,
>> 
>> Yesterday, I updated the master code which include [FLINK-7030]:
>> Build
>> with scala-2.11 by default. After that, I entered a strange
>> problem with
>> IDEA that many classes can't be found, and the project can't be
>> built/compiled (in IDEA), but maven install worked good.
>> 
>> After a series of attempts, I found that IDEA activate the
>> scala-2.10
>> profile by default which result in this problem. After deactivate
>> scala-2.10 profile via  sidebar Maven Projects -> Profiles ->
>> deactivate
>> "scala-2.10" profile, and every works good again.
>> 
>> [image: 内嵌图片 1]
>> 
>> I share this tip in the dev list, because a lot of my colleagues
>> have the
>> same issues, and maybe many other Flink devs have the same
>> problem too.
>> 
>> BTW, I don't know why IDEA activate scala-2.10 by default, not
>> sure it's a
>> IDEA bug or the wrong profile setting somewhere.
>> 
>> 
>> Regards,
>> Jark Wu
>> 
 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 

Re: [DISCUSS] Managing announcements of breaking API / state compatibility changes in major releases

[Tzu-Li (Gordon) Tai]

@Ted
Good idea. Using the “Release Note” field would definitely be better than some 
dangling comment.


On 6 July 2017 at 6:35:24 PM, Ted Yu (yuzhih...@gmail.com) wrote:

Thru INFRA-14519, Release Note field has been added to JIRA. 
We can use this field to record API/ compatibility changes. 
 Original message From: Aljoscha Krettek  
Date: 7/6/17 2:01 AM (GMT-08:00) To: dev@flink.apache.org Subject: Re: 
[DISCUSS] Managing announcements of breaking API / state
  compatibility changes in major releases  
+1

This sounds very good! I just hope that everyone is aware enough and doesn’t 
forget adding these tags.

> On 6. Jul 2017, at 09:59, Tzu-Li (Gordon) Tai  wrote:
>  
>  
> Hi devs,
>  
> I would like to follow up my proposal in [1] regarding how we can more 
> systematically and easily collect breaking changes, so that major release 
> announcements can officially include a list of such changes.
>  
> Originally the idea was to collect these in the Wiki whenever a breaking 
> change is merged, but the extra step to go to the Wiki after closing the JIRA 
> ticket seems to be a bit too tedious.
>  
> There were other suggestions by simply doing this:
> 1. when closing the ticket, label the JIRA ticket as either 
> `state-compat-breaking` / `api-breaking` / `api-deprecated`
> 2. leave a comment on the JIRA that describes the change. Ideally, this 
> comment can be directly copy-and-pasted as an announcement for the change.
>  
> When releasing a major release, for example 1.4, the release manager can then 
> search for such changes by simply filtering the fix version and labels.
>  
> For `api-breaking` / `api-deprecated` changes, it would be straightforward: 
> public API was broken / deprecated starting from the fix version set on the 
> ticket.
> For `state-compat-breaking` changes: please describe clearly in the added 
> comment which older versions the state is no longer compatible with. For 
> example, "State compatibility is broken for versions before Flink XX. To 
> restore savepoints prior to Flink XX in the latest release Flink YY, please 
> first restore the savepoint with a version > XX and < YY.”
>  
> Note that the contracts of `@Public` / `@PublicEvolving` [2] should still 
> remain the same; we’re not discussing altering the API stability contracts 
> here.
> Also note that the discussion for our state backwards compatibility policy is 
> being discussed here [3].
>  
> What do you think? Ideally this would be minimal extra effort for committers, 
> and would make it easier to let our release announcements and API migration 
> guide [4] be much more informative of these changes.
> If you agree, I'll add this guideline to [5], and suggest that we start doing 
> this immediately starting from Flink 1.4.0.
>  
> Cheers,
> Gordon
>  
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/RESULT-VOTE-Release-Apache-Flink-1-3-0-RC3-td17841.html
> [2] https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations
> [3] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backwards-compatibility-policy-td17640.html
>   
> [4] 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/migration.html
> [5] 
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+development+guidelines

Re: Re: Tips to fix IDEA strange problem after updating master code

[Stephan Ewen]

Is it possible to not use a profile for that, but only an actual property
variable?
Or do we use different dependencies with Scala 2.10 vs Scala 2.11 ?


On Tue, Jul 4, 2017 at 11:23 AM, 郭健  wrote:

> After deactivating scala-2.10 profile in IntelliJ, this issue is gone.
> Thank you all.
>
> On 7/4/17, 17:11, "Piotr Nowojski"  wrote:
>
> Maybe try
>
> $ mvn clean
>
> Before reimporting and restarting/invalidating caches in IntelliJ? Did
> you deactivate scala-2.10 profile in the IntelliJ?
>
> Piotrek
>
>
>
> > On Jul 4, 2017, at 11:05 AM, 郭健  wrote:
> >
> > I have done all these but still got some issue in IDEA, especially
> in the flink-connector project.
> > 
> >
> > On 7/4/17, 15:57, "Piotr Nowojski"  wrote:
> >
> >Besides deactivating “scala-2.10” profile in the Intellij it
> might be necessary to:
> >- reimport maven project:
> >   1. Right click on root module: “flink-parent”
> >   2. Maven
> >   3. reimport
> >- invalidate caches and restart: File -> Invalidate caches and
> restart -> invalidate /restart
> >- rebuild whole project
> >
> >I suspect that either activation of scala-2.10 by default comes
> from flink-scala and flick-scala-shell poms or it’s an artifact because you
> created/imported Intellij project when 2.10 was the default. If the first
> option is true, this PR: https://github.com/apache/flink/pull/4240 <
> https://github.com/apache/flink/pull/4240> might fix this issue.
> >
> >
> >Another quirk that I encauntered is the compile error about
> ILoopCompat class being defined twice in Intellij (works fine from
> console). This comes from flink-scala-shell/pom.xml, which defines two
> different source paths depending on Scala version:
> >
> >src/main/scala-${scala.binary.version}
> >
> >Such thing is not supported by Intellij and one have to manually
> remove one of the source directory (either 2.11 or 2.10) from the project
> settings.
> >
> >Piotrek
> >
> >> On Jul 4, 2017, at 9:46 AM, Aljoscha Krettek 
> wrote:
> >>
> >> Thanks for the hint!
> >>
> >>> On 4. Jul 2017, at 06:03, Ted Yu  wrote:
> >>>
> >>> Looks like the picture didn't go thru.
> >>>
> >>> Mind using third party site ?
> >>>
> >>> Thanks
> >>>
> >>> On Mon, Jul 3, 2017 at 8:56 PM, Jark Wu  wrote:
> >>>
>  Hi devs,
> 
>  Yesterday, I updated the master code which include [FLINK-7030]:
> Build
>  with scala-2.11 by default. After that, I entered a strange
> problem with
>  IDEA that many classes can't be found, and the project can't be
>  built/compiled (in IDEA), but maven install worked good.
> 
>  After a series of attempts, I found that IDEA activate the
> scala-2.10
>  profile by default which result in this problem. After deactivate
>  scala-2.10 profile via  sidebar Maven Projects -> Profiles ->
> deactivate
>  "scala-2.10" profile, and every works good again.
> 
>  [image: 内嵌图片 1]
> 
>  I share this tip in the dev list, because a lot of my colleagues
> have the
>  same issues, and maybe many other Flink devs have the same
> problem too.
> 
>  BTW, I don't know why IDEA activate scala-2.10 by default, not
> sure it's a
>  IDEA bug or the wrong profile setting somewhere.
> 
> 
>  Regards,
>  Jark Wu
> 
> >>
> >
> >
> >
>
>
>
>

[jira] [Created] (FLINK-7113) Make ClusterDescriptor independent of Flink cluster size

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7113:


 Summary: Make ClusterDescriptor independent of Flink cluster size
 Key: FLINK-7113
 URL: https://issues.apache.org/jira/browse/FLINK-7113
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


The {{ClusterDescriptor}} needs to know the size of the Flink cluster it is 
supposed to deploy. As a consequence we have the 
{{AbstractYarnClusterDescriptor}} which is configured with this information via 
setters. I think it would be better to give the cluster size to the 
{{ClusterDescriptor}} via the {{deploySession(ClusterSpecification)}} call. 
That way we better decouple the deployment from the cluster configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Managing announcements of breaking API / state compatibility changes in major releases

[Ted Yu]

Thru INFRA-14519, Release Note field has been added to JIRA. 
We can use this field to record API/ compatibility changes. 
 Original message From: Aljoscha Krettek  
Date: 7/6/17  2:01 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: 
[DISCUSS] Managing announcements of breaking API / state
  compatibility changes in major releases 
+1

This sounds very good! I just hope that everyone is aware enough and doesn’t 
forget adding these tags.

> On 6. Jul 2017, at 09:59, Tzu-Li (Gordon) Tai  wrote:
> 
> 
> Hi devs,
> 
> I would like to follow up my proposal in [1] regarding how we can more 
> systematically and easily collect breaking changes, so that major release 
> announcements can officially include a list of such changes.
> 
> Originally the idea was to collect these in the Wiki whenever a breaking 
> change is merged, but the extra step to go to the Wiki after closing the JIRA 
> ticket seems to be a bit too tedious.
> 
> There were other suggestions by simply doing this:
> 1. when closing the ticket, label the JIRA ticket as either 
> `state-compat-breaking` / `api-breaking` / `api-deprecated`
> 2. leave a comment on the JIRA that describes the change. Ideally, this 
> comment can be directly copy-and-pasted as an announcement for the change.
> 
> When releasing a major release, for example 1.4, the release manager can then 
> search for such changes by simply filtering the fix version and labels.
> 
> For `api-breaking` / `api-deprecated` changes, it would be straightforward: 
> public API was broken / deprecated starting from the fix version set on the 
> ticket.
> For `state-compat-breaking` changes: please describe clearly in the added 
> comment which older versions the state is no longer compatible with. For 
> example, "State compatibility is broken for versions before Flink XX. To 
> restore savepoints prior to Flink XX in the latest release Flink YY, please 
> first restore the savepoint with a version > XX and < YY.”
> 
> Note that the contracts of `@Public` / `@PublicEvolving` [2] should still 
> remain the same; we’re not discussing altering the API stability contracts 
> here.
> Also note that the discussion for our state backwards compatibility policy is 
> being discussed here [3].
> 
> What do you think? Ideally this would be minimal extra effort for committers, 
> and would make it easier to let our release announcements and API migration 
> guide [4] be much more informative of these changes.
> If you agree, I'll add this guideline to [5], and suggest that we start doing 
> this immediately starting from Flink 1.4.0.
> 
> Cheers,
> Gordon
> 
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/RESULT-VOTE-Release-Apache-Flink-1-3-0-RC3-td17841.html
> [2] https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations
> [3] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backwards-compatibility-policy-td17640.html
>  
> [4] 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/migration.html
> [5] 
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+development+guidelines

Re: Make SubmittedJobGraphStore configurable

[Ted Yu]

The sample config entries are broken into multiple lines.

Can you send the config again with one config on one line ?

Cheers

On Wed, Jul 5, 2017 at 10:19 PM, Chen Qin  wrote:

> ​Hi there,
>
> ​I would like to propose/discuss median level refactor work to make
> submittedJobGraphStore configurable and extensible.
>
> The rationale behind is to allow users offload those meta data to durable
> cross dc read after write strong consistency storage and decouple with zk
> quorum.
> ​
>
> https://issues.apache.org/jira/browse/FLINK-7106
>
> 
> New configurable setting in flink.conf
> ​ looks like following
>
> g​
> raph
> ​-s
> tore:
> ​customized/zookeeper
> g​
> raph
> ​-s
> tore​.class: xx.yy.MyS3SubmittedJobGraphStore​Imp
>
> g​
> raph
> ​-s
> tore.
> ​endpoint
> : s3.amazonaws.com
> g​
> raph
> ​-s
> tore.path.root:
> ​s3:/
> ​
> /
> ​my root/​
>
> Thanks,
> Chen
>

[jira] [Created] (FLINK-7112) YARNSessionCapacitySchedulerITCase uses example jar without dependency

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7112:
---

 Summary: YARNSessionCapacitySchedulerITCase uses example jar 
without dependency
 Key: FLINK-7112
 URL: https://issues.apache.org/jira/browse/FLINK-7112
 Project: Flink
  Issue Type: Bug
  Components: Examples, Tests, YARN
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Priority: Critical
 Fix For: 1.4.0


The following tests in {{YARNSessionCapacitySchedulerITCase}} make use of an 
example jar without having specified any dependency on the example modules:
* perJobYarnCluster
* perJobYarnClusterWithParallelism
* testDetachedPerJobYarnCluster
* testDetachedPerJobYarnClusterWithStreamingJob



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: refactor StreamConfig

[Aljoscha Krettek]

Hi,

Yes, the fact that the operator can see isChainStart() and isChainEnd() is not 
good, in my opinion. These seems to be an implementation detail that an 
operator should not be aware of. For now it’s ok but maybe we can fix that 
later.

Regarding output edges and serialisers, I think it might be necessary to 
differentiate between an operator config that the operator “can see”, this 
would be very minimal, and an operator config that the task uses to setup the 
chain and other stuff. This would store things that are tied to one operator 
within the chain but that the operator itself must not be concerned with. What 
do you think?

Best,
Aljoscha

> On 5. Jul 2017, at 07:39, Xu Pingyong  wrote:
> 
> Hi Aljoscha:
> 
> 
>I sum up my thoughts now.
>1. rename StreamConfig to StreamTaskConfig.
>2. OperatorConig can be changed to be serialisable. If StreamTaskConfig is 
> also serialisable, it cannot be deserialized when it is passed to the 
> jobManager, which do not depend on "flink-streaming-java".
>3. The call getChainIndex() is used only in OperatorConfig.toString(), it 
> can be removed. However, isChainStart() and isChainEnd() is used in 
> AbstractStreamOperator.setup(...).
> 
>However I am not sure whether to put some properties in StreamTaskConfig 
> or OperatorConfig, for example input serializer is used not only in Operator 
> but also in OpeatorChain. Linkewise output edges and serialisers are only 
> used in OpeatorChain now, but whether the operator can see and use them later?
>2)  streamOperator
>4)  output edges and serializers.
> 
>   What do you think?
> 
> 
>Best Regards!
> Xu Pingyong
> 
> 
> 
> 
> 
> 
> 
> 
> 
> At 2017-07-05 11:02:56, "Xu Pingyong"  wrote:
>> Hi Aljoscha:
>> 
>> 
>> Ye, I agree with you that an operator should not see output edges and 
>> serialisers. The call getChainIndex() is used only in 
>> OperatorConfig.toString(), it can be removed. However, isChainStart() and 
>> isChainEnd() is used in AbstractStreamOperator.setup(...).
>> 
>> 
>> But I think what Stephan meant is only that changing OperatorConfig to be 
>> serialisable. If StreamConfig is also serialisable, it need to be serialized 
>> into the Configuration, which is underlying before and flows across modules.
>> 
>> 
>> Do you agree what I understand?
>> 
>> 
>> Best Regards!
>> 
>> 
>> Xu Pingyong
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> At 2017-07-05 00:01:34, "Aljoscha Krettek"  wrote:
>>> Hi,
>>> 
>>> Yes, but I think what Stephan was hinting at was to change both of them to 
>>> be serialisable when already working on this.
>>> 
>>> I think input serialiser is fine to have in OperatorConfig, you’re right! I 
>>> don’t see getChainIndex() used anywhere in the code, though. And the output 
>>> edges and serialisers also look like they should not be visible to the 
>>> operator.
>>> 
>>> What do you think?
>>> 
>>> Best,
>>> Aljoscha
>>> 
 On 4. Jul 2017, at 17:52, xu  wrote:
 
 Hi Aljoscha:
   Thanks a lot for your advice.
 
 
   I think I have not need to separate steps, because what I do is only 
 that introducing OperatorConfig and moving the fields. StreamConfig  still 
  relys on an underlying Configuration which flows from client to the 
 jobmanager and then to the task.
 
 
   The following configs are used in an operator now:
   2) input serializer is used in AsyncWaitOperator.class
   5) chain.index is used in AbstractStreamOperator.setup(...)
 
 
   However, What I put in the OperatorConfig is all configs belong to the 
 operator, contains not only the operator uses now, but also the streamTask 
 uses to build an operator. By OperatorConfig, an operator can not see 
 configs belong to others.
 
 
  Best Regards!
  JiPing
>>> 

Re: [DISCUSS] Managing announcements of breaking API / state compatibility changes in major releases

[Aljoscha Krettek]

+1

This sounds very good! I just hope that everyone is aware enough and doesn’t 
forget adding these tags.

> On 6. Jul 2017, at 09:59, Tzu-Li (Gordon) Tai  wrote:
> 
> 
> Hi devs,
> 
> I would like to follow up my proposal in [1] regarding how we can more 
> systematically and easily collect breaking changes, so that major release 
> announcements can officially include a list of such changes.
> 
> Originally the idea was to collect these in the Wiki whenever a breaking 
> change is merged, but the extra step to go to the Wiki after closing the JIRA 
> ticket seems to be a bit too tedious.
> 
> There were other suggestions by simply doing this:
> 1. when closing the ticket, label the JIRA ticket as either 
> `state-compat-breaking` / `api-breaking` / `api-deprecated`
> 2. leave a comment on the JIRA that describes the change. Ideally, this 
> comment can be directly copy-and-pasted as an announcement for the change.
> 
> When releasing a major release, for example 1.4, the release manager can then 
> search for such changes by simply filtering the fix version and labels.
> 
> For `api-breaking` / `api-deprecated` changes, it would be straightforward: 
> public API was broken / deprecated starting from the fix version set on the 
> ticket.
> For `state-compat-breaking` changes: please describe clearly in the added 
> comment which older versions the state is no longer compatible with. For 
> example, "State compatibility is broken for versions before Flink XX. To 
> restore savepoints prior to Flink XX in the latest release Flink YY, please 
> first restore the savepoint with a version > XX and < YY.”
> 
> Note that the contracts of `@Public` / `@PublicEvolving` [2] should still 
> remain the same; we’re not discussing altering the API stability contracts 
> here.
> Also note that the discussion for our state backwards compatibility policy is 
> being discussed here [3].
> 
> What do you think? Ideally this would be minimal extra effort for committers, 
> and would make it easier to let our release announcements and API migration 
> guide [4] be much more informative of these changes.
> If you agree, I'll add this guideline to [5], and suggest that we start doing 
> this immediately starting from Flink 1.4.0.
> 
> Cheers,
> Gordon
> 
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/RESULT-VOTE-Release-Apache-Flink-1-3-0-RC3-td17841.html
> [2] https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations
> [3] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backwards-compatibility-policy-td17640.html
>  
> [4] 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/migration.html
> [5] 
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+development+guidelines

[jira] [Created] (FLINK-7111) flink-scala-shell fails on mvn verify

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7111:
---

 Summary: flink-scala-shell fails on mvn verify
 Key: FLINK-7111
 URL: https://issues.apache.org/jira/browse/FLINK-7111
 Project: Flink
  Issue Type: Bug
  Components: Scala Shell, Tests
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
 Fix For: 1.4.0


Running {{mvn verify}} after {{mvn clean install -DskipTests}} causes the build 
to fail in {{flink-scala-shell}} with

{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.4:single 
(create-library-loading-jar) on project flink-scala-shell_2.11: Failed to 
create assembly: Error creating assembly archive test-jar: You must set at 
least one file. -> [Help 1]
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Release 1.3.2 planning

[Aljoscha Krettek]

I’m seeing these remaining blockers: 
https://issues.apache.org/jira/browse/FLINK-7069?filter=12334772&jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20resolution%20%3D%20Unresolved
 


Could everyone please correctly mark as “blocking” those issues that they 
consider blocking for 1.3.2 so that we get an accurate overview of where we are.

@Chesnay, could you maybe check if this one should in fact be considered a 
blocker: https://issues.apache.org/jira/browse/FLINK-7034? 


Best,
Aljoscha
> On 6. Jul 2017, at 07:19, Tzu-Li (Gordon) Tai  wrote:
> 
> FLINK-7041 has been merged.
> I’d also like to raise another blocker for 1.3.2: 
> https://issues.apache.org/jira/browse/FLINK-6996.
> 
> Cheers,
> Gordon
> On 30 June 2017 at 12:46:07 AM, Aljoscha Krettek (aljos...@apache.org) wrote:
> 
> Gordon and I found this (in my opinion) blocking issue: 
> https://issues.apache.org/jira/browse/FLINK-7041 
>   
> 
> I’m trying to quickly provide a fix.  
> 
>> On 26. Jun 2017, at 15:30, Timo Walther  wrote:  
>> 
>> I just opened a PR which should be included in the next bug fix release for 
>> the Table API:  
>> https://issues.apache.org/jira/browse/FLINK-7005  
>> 
>> Timo  
>> 
>> Am 23.06.17 um 14:09 schrieb Robert Metzger:  
>>> Thanks Haohui.  
>>> 
>>> The first main task for the release management is to come up with a  
>>> timeline :)  
>>> Lets just wait and see which issues get reported. There are currently no  
>>> blockers set for 1.3.1 in JIRA.  
>>> 
>>> On Thu, Jun 22, 2017 at 6:47 PM, Haohui Mai  wrote:  
>>> 
 Hi,  
 
 Release management is though, I'm happy to help. Are there any timelines  
 you have in mind?  
 
 Haohui  
 On Fri, Jun 23, 2017 at 12:01 AM Robert Metzger   
 wrote:  
 
> Hi all,  
> 
> with the 1.3.1 release on the way, we can start thinking about the 1.3.2  
> release.  
> 
> We have already one issue that should go in there:  
> - https://issues.apache.org/jira/browse/FLINK-6964  
> 
> If there are any other blockers, let us know here :)  
> 
> I'm wondering if there's somebody from the community who's willing to  
 take  
> care of the release management of 1.3.2 :)  
> 
>> 
> 

[DISCUSS] Managing announcements of breaking API / state compatibility changes in major releases

[Tzu-Li (Gordon) Tai]

Hi devs,

I would like to follow up my proposal in [1] regarding how we can more 
systematically and easily collect breaking changes, so that major release 
announcements can officially include a list of such changes.

Originally the idea was to collect these in the Wiki whenever a breaking change 
is merged, but the extra step to go to the Wiki after closing the JIRA ticket 
seems to be a bit too tedious.

There were other suggestions by simply doing this:
1. when closing the ticket, label the JIRA ticket as either 
`state-compat-breaking` / `api-breaking` / `api-deprecated`
2. leave a comment on the JIRA that describes the change. Ideally, this comment 
can be directly copy-and-pasted as an announcement for the change.

When releasing a major release, for example 1.4, the release manager can then 
search for such changes by simply filtering the fix version and labels.

For `api-breaking` / `api-deprecated` changes, it would be straightforward: 
public API was broken / deprecated starting from the fix version set on the 
ticket.
For `state-compat-breaking` changes: please describe clearly in the added 
comment which older versions the state is no longer compatible with. For 
example, "State compatibility is broken for versions before Flink XX. To 
restore savepoints prior to Flink XX in the latest release Flink YY, please 
first restore the savepoint with a version > XX and < YY.”

Note that the contracts of `@Public` / `@PublicEvolving` [2] should still 
remain the same; we’re not discussing altering the API stability contracts here.
Also note that the discussion for our state backwards compatibility policy is 
being discussed here [3].

What do you think? Ideally this would be minimal extra effort for committers, 
and would make it easier to let our release announcements and API migration 
guide [4] be much more informative of these changes.
If you agree, I'll add this guideline to [5], and suggest that we start doing 
this immediately starting from Flink 1.4.0.

Cheers,
Gordon

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/RESULT-VOTE-Release-Apache-Flink-1-3-0-RC3-td17841.html
[2] https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations
[3] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backwards-compatibility-policy-td17640.html
 
[4] 
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/migration.html
[5] 
https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+development+guidelines

[jira] [Created] (FLINK-7110) Add deploy with job to ClusterDescriptor

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7110:


 Summary: Add deploy with job to ClusterDescriptor
 Key: FLINK-7110
 URL: https://issues.apache.org/jira/browse/FLINK-7110
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Reporter: Till Rohrmann
Assignee: Till Rohrmann
 Fix For: 1.4.0


The {{ClusterDescriptor's}} interface has to be improved to support deploying a 
per-job cluster where you directly provide the {{JobGraph}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)