Re: [ANNOUNCE] Apache Flink 1.3.0 released

[Ismaël Mejía]

Ted, The artifacts are already in maven central.
https://search.maven.org/#search|ga|1|g%3A%22org.apache.flink%22

Notice that google somehow puts always that mvnrepository.com website
mirror first in the results, even if it is almost always outdated. I
had the same problem when checking for jars before.

Ismaël

On Fri, Jun 2, 2017 at 3:59 AM, Ted Yu  wrote:
> Robert:
> Do you know when maven artifacts would be populated ?
>
> Currently I don't see 1.3.0 here:
>
> https://mvnrepository.com/artifact/org.apache.flink/flink-core
>
> Thanks
>
> On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger  wrote:
>
>> The Apache Flink community is pleased to announce the release of Apache
>> Flink 1.3.0.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>>
>> The release is available for download at:
>>
>> https://flink.apache.org/downloads.html
>>
>> There is a blog post available on the Flink site explaining most of the
>> new features:
>>
>>https://flink.apache.org/news/2017/06/01/release-1.3.0.html
>>
>> The full changelog is available at: https://flink.apache.org/
>> blog/release_1.3.0-changelog.html
>>
>>
>>
>> We would like to thank all contributors who made this release possible!
>>

[jira] [Created] (FLINK-6808) Stream join fails when checkpointing is enabled

[Francisco Rosa (JIRA)]

Francisco Rosa created FLINK-6808:
-

 Summary: Stream join fails when checkpointing is enabled
 Key: FLINK-6808
 URL: https://issues.apache.org/jira/browse/FLINK-6808
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Affects Versions: 1.3.0
Reporter: Francisco Rosa
 Fix For: 1.3.1


The combination of joining streams and checkpointing fails in 1.3.0. It used to 
work with the previous 1.2 version. Code example for failure:

public static void main(String[] args) throws Exception {

final StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
// enable checkpoints
env.enableCheckpointing(5000);

// create two streams
DataStreamSource one = env.generateSequence(0, 5000);
DataStreamSource two = env.generateSequence(2000, 15000);

// process both, provide a delay to make sure checkpoint will happen
DataStream oneProcessed = one.
map(oneValue -> {
Thread.sleep(1000);
return "val-" + oneValue;
});
DataStream twoProcessed = two.
map(oneValue -> {
Thread.sleep(1000);
return "val-" + oneValue;
});

// join the two streams, join on string match
DataStream joinedStreams = oneProcessed.
join(twoProcessed).
where(String::toString).
equalTo(String::toString).
window(TumblingProcessingTimeWindows.of(Time.seconds(5))).
apply(new JoinFunction() {
@Override
public String join(String oneValue, String twoValue) {
// nothing really relevant, just concatenate string
return oneValue + "+" + twoValue;
}
});

// output results
joinedStreams.print();

env.execute("Issue with stream join and checkpoints");
}

Stack trace:

java.lang.Exception: Could not perform checkpoint 1 for operator 
TriggerWindow(TumblingProcessingTimeWindows(5000), 
ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
 ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
Sink: Unnamed (1/1).
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:550)
at 
org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:378)
at 
org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:281)
at 
org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:183)
at 
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:213)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Could not complete snapshot 1 for operator 
TriggerWindow(TumblingProcessingTimeWindows(5000), 
ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
 ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
Sink: Unnamed (1/1).
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1089)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:653)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:589)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:542)
... 8 more
Caused by: java.lang.UnsupportedOperationException: This serializer is not 
registered for managed state.
at 
org.apache.flink.streaming.api.datastream.CoGroupedStreams$UnionSerializer.snapshotConfiguration(CoGroupedStreams.java:555)
at 
org.apache.flink.api.common.typeutils.CompositeTypeSerializerConfigSnapshot.(CompositeTypeSerializerConfigSnapshot.java:53)
at 
org.apache.flink.api.common.typeutils.base.CollectionSerializerConfigSnapshot.(CollectionSerializerConfigSnapshot.java:39)
at 

Re: [ANNOUNCE] Apache Flink 1.3.0 released

[Ted Yu]

Robert:
Do you know when maven artifacts would be populated ?

Currently I don't see 1.3.0 here:

https://mvnrepository.com/artifact/org.apache.flink/flink-core

Thanks

On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger  wrote:

> The Apache Flink community is pleased to announce the release of Apache
> Flink 1.3.0.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
>
> The release is available for download at:
>
> https://flink.apache.org/downloads.html
>
> There is a blog post available on the Flink site explaining most of the
> new features:
>
>https://flink.apache.org/news/2017/06/01/release-1.3.0.html
>
> The full changelog is available at: https://flink.apache.org/
> blog/release_1.3.0-changelog.html
>
>
>
> We would like to thank all contributors who made this release possible!
>

[jira] [Created] (FLINK-6807) Elasticsearch 5 connector artifact not published to maven

[Francisco Rosa (JIRA)]

Francisco Rosa created FLINK-6807:
-

 Summary: Elasticsearch 5 connector artifact not published to maven 
 Key: FLINK-6807
 URL: https://issues.apache.org/jira/browse/FLINK-6807
 Project: Flink
  Issue Type: Bug
  Components: ElasticSearch Connector
Affects Versions: 1.3.0
Reporter: Francisco Rosa
 Fix For: 1.3.0


Downloaded Flink 1.3.0, want to use the Elasticsearch 5.X connector.
The maven artifact does not seem to be published to maven central repositories 
yet. Looking for:
  
org.apache.flink:flink-connector-elasticsearch5_2.10:1.3.0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6805) Flink Cassandra connector dependency on Netty disagrees with Flink

[Shannon Carey (JIRA)]

Shannon Carey created FLINK-6805:


 Summary: Flink Cassandra connector dependency on Netty disagrees 
with Flink
 Key: FLINK-6805
 URL: https://issues.apache.org/jira/browse/FLINK-6805
 Project: Flink
  Issue Type: Bug
  Components: Cassandra Connector
Affects Versions: 1.2.1, 1.3.0
Reporter: Shannon Carey


The Flink Cassandra connector has a dependency on Netty libraries (via 
promotion of transitive dependencies by the Maven shade plugin) at version 
4.0.33.Final, which disagrees with the version included in Flink of 
4.0.27.Final which is included & managed by the parent POM via dependency on 
netty-all.

Due to use of netty-all, the dependency management doesn't take effect on the 
individual libraries such as netty-handler, netty-codec, etc.

I suggest that dependency management of Netty should be added for all Netty 
libraries individually (netty-handler, etc.) so that all Flink modules use the 
same version, and similarly I suggest that exclusions be added to the 
quickstart example POM for the individual Netty libraries so that fat JARs 
don't include conflicting versions of Netty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] Apache Flink 1.3.0 released

[Patrick Lucas]

And here's the PR for the new Docker images:
https://github.com/docker-library/official-images/pull/3006

(It was merged 10 min ago; the new images should become available by
tomorrow)

--
Patrick Lucas

On Thu, Jun 1, 2017 at 7:36 PM, Eron Wright  wrote:

> Opened a PR for Homebrew update to `apache-flink` formula:
> https://github.com/Homebrew/homebrew-core/pull/14151
>
> On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger 
> wrote:
>
> > The Apache Flink community is pleased to announce the release of Apache
> > Flink 1.3.0.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> >
> >
> > The release is available for download at:
> >
> > https://flink.apache.org/downloads.html
> >
> > There is a blog post available on the Flink site explaining most of the
> > new features:
> >
> >https://flink.apache.org/news/2017/06/01/release-1.3.0.html
> >
> > The full changelog is available at: https://flink.apache.org/
> > blog/release_1.3.0-changelog.html
> >
> >
> >
> > We would like to thank all contributors who made this release possible!
> >
>

flink-connector-elasticsearch5_2.11 version 1.3.0 is missing

[Fritz Budiyanto]

Hi All,

I updated my pom file to use the newly release 1.3.0, and my build failed.
Looks like there is no flink connector for ES5 with version 1.3.0.
Could you someone release it ?
Am I using the right artifactId connector for ES5/1.3.0 ?

Thanks,
Fritz

  
org.apache.flink

flink-connector-elasticsearch5_2.11
+ 1.3.0
- 1.3-SNAPSHOT

Re: Building only flink-java8 module

[Dawid Wysakowicz]

I tried the second approach before and it results in the error with
scala.binary.version I attached, which is the same Ted got. I use it though
for other modules and it works.

I will try the first approach soon.

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder



2017-06-01 18:14 GMT+02:00 Ted Yu :

> That removes the error.
> However, looks like tests from other module(s) are run as well.
> Just an example:
>
> 
> 
> 16:12:53,103 INFO
>  org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest  -
> 
> 
>
> On Thu, Jun 1, 2017 at 9:08 AM, Aljoscha Krettek 
> wrote:
>
> > Ah, I forgot that you also have to add “-Pjdk8” to activate the Java 8
> > profile. Otherwise the flink-java8 module will not be referenced in the
> > main pom.
> >
> > > On 1. Jun 2017, at 17:42, Ted Yu  wrote:
> > >
> > > When using the second approach (install followed by 'mvn verify'), I
> got
> > > the following:
> > >
> > > [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> > > resolve dependencies for project
> > > org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
> > > dependencies at
> > > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to
> > read
> > > artifact descriptor for
> > > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure
> to
> > > find
> > > org.apache.flink:flink-examples_${scala.binary.
> version}:pom:1.4-SNAPSHOT
> > in
> > > https://repository.apache.org/snapshots was cached in the local
> > repository,
> > > resolution will not be reattempted until the update interval of
> > > apache.snapshots has elapsed or updates are forced -> [Help 1]
> > >
> > > Looks like ${scala.binary.version} was not substituted for correctly.
> > >
> > > On Thu, Jun 1, 2017 at 8:24 AM, Ted Yu  wrote:
> > >
> > >> When I used the command given by Aljoscha, I got:
> > >>
> > >> https://pastebin.com/8WTGvdFQ
> > >>
> > >> FYI
> > >>
> > >> On Thu, Jun 1, 2017 at 8:17 AM, Aljoscha Krettek  >
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I think you can use something like
> > >>>
> > >>> mvn verify -am -pl flink-java8
> > >>>
> > >>> (From the base directory)
> > >>>
> > >>> The -pl flag will tell maven to only do that module while -am tells
> it
> > to
> > >>> also builds its dependencies. This might or might not also run the
> > tests on
> > >>> the dependent-upon projects, I’m not sure.
> > >>>
> > >>> As an alternative you can do “mvn clean install …” (skipping tests
> and
> > >>> everything) and then switch into the flink-java8 directory and run
> “mvn
> > >>> verify” there.
> > >>>
> > >>> Best,
> > >>> Aljoscha
> > >>>
> > >>>
> >  On 1. Jun 2017, at 16:04, Dawid Wysakowicz <
> > wysakowicz.da...@gmail.com>
> > >>> wrote:
> > 
> >  Hi devs!
> > 
> >  Recently I tried running* mvn verify* just for the *flink-java8*
> > module
> > >>> (to
> >  run those tests locally) and it fails with the following error:
> > 
> >  [ERROR] Failed to execute goal on project flink-java8_2.10: Could
> not
> > > resolve dependencies for project
> > > org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to
> > collect
> > > dependencies at
> > > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT:
> Failed
> > >>> to read
> > > artifact descriptor for
> > > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT:
> Failure
> > >>> to
> > > find
> > > org.apache.flink:flink-examples_${scala.binary.
> > version}:pom:1.4-SNAPSHOT
> > >>> in
> > > https://repository.apache.org/snapshots was cached in the local
> > > repository, resolution will not be reattempted until the update
> > >>> interval of
> > > apache.snapshots has elapsed or updates are forced -> [Help 1]
> > >
> > 
> >  That strategy works for any other module I tried. I would be
> grateful
> > >>> for
> >  any tips how can I run just tests for flink-java8 locally.
> > 
> >  Thanks in advance.
> > 
> >  Z pozdrowieniami! / Cheers!
> > 
> >  Dawid Wysakowicz
> > 
> >  *Data/Software Engineer*
> > 
> >  Skype: dawid_wys | Twitter: @OneMoreCoder
> > 
> >  
> > >>>
> > >>>
> > >>
> >
> >
>

Re: Building only flink-java8 module

[Ted Yu]

That removes the error.
However, looks like tests from other module(s) are run as well.
Just an example:


16:12:53,103 INFO
 org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest  -


On Thu, Jun 1, 2017 at 9:08 AM, Aljoscha Krettek 
wrote:

> Ah, I forgot that you also have to add “-Pjdk8” to activate the Java 8
> profile. Otherwise the flink-java8 module will not be referenced in the
> main pom.
>
> > On 1. Jun 2017, at 17:42, Ted Yu  wrote:
> >
> > When using the second approach (install followed by 'mvn verify'), I got
> > the following:
> >
> > [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> > resolve dependencies for project
> > org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
> > dependencies at
> > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to
> read
> > artifact descriptor for
> > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure to
> > find
> > org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT
> in
> > https://repository.apache.org/snapshots was cached in the local
> repository,
> > resolution will not be reattempted until the update interval of
> > apache.snapshots has elapsed or updates are forced -> [Help 1]
> >
> > Looks like ${scala.binary.version} was not substituted for correctly.
> >
> > On Thu, Jun 1, 2017 at 8:24 AM, Ted Yu  wrote:
> >
> >> When I used the command given by Aljoscha, I got:
> >>
> >> https://pastebin.com/8WTGvdFQ
> >>
> >> FYI
> >>
> >> On Thu, Jun 1, 2017 at 8:17 AM, Aljoscha Krettek 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I think you can use something like
> >>>
> >>> mvn verify -am -pl flink-java8
> >>>
> >>> (From the base directory)
> >>>
> >>> The -pl flag will tell maven to only do that module while -am tells it
> to
> >>> also builds its dependencies. This might or might not also run the
> tests on
> >>> the dependent-upon projects, I’m not sure.
> >>>
> >>> As an alternative you can do “mvn clean install …” (skipping tests and
> >>> everything) and then switch into the flink-java8 directory and run “mvn
> >>> verify” there.
> >>>
> >>> Best,
> >>> Aljoscha
> >>>
> >>>
>  On 1. Jun 2017, at 16:04, Dawid Wysakowicz <
> wysakowicz.da...@gmail.com>
> >>> wrote:
> 
>  Hi devs!
> 
>  Recently I tried running* mvn verify* just for the *flink-java8*
> module
> >>> (to
>  run those tests locally) and it fails with the following error:
> 
>  [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> > resolve dependencies for project
> > org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to
> collect
> > dependencies at
> > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed
> >>> to read
> > artifact descriptor for
> > org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure
> >>> to
> > find
> > org.apache.flink:flink-examples_${scala.binary.
> version}:pom:1.4-SNAPSHOT
> >>> in
> > https://repository.apache.org/snapshots was cached in the local
> > repository, resolution will not be reattempted until the update
> >>> interval of
> > apache.snapshots has elapsed or updates are forced -> [Help 1]
> >
> 
>  That strategy works for any other module I tried. I would be grateful
> >>> for
>  any tips how can I run just tests for flink-java8 locally.
> 
>  Thanks in advance.
> 
>  Z pozdrowieniami! / Cheers!
> 
>  Dawid Wysakowicz
> 
>  *Data/Software Engineer*
> 
>  Skype: dawid_wys | Twitter: @OneMoreCoder
> 
>  
> >>>
> >>>
> >>
>
>

Re: Building only flink-java8 module

[Aljoscha Krettek]

Ah, I forgot that you also have to add “-Pjdk8” to activate the Java 8 profile. 
Otherwise the flink-java8 module will not be referenced in the main pom.

> On 1. Jun 2017, at 17:42, Ted Yu  wrote:
> 
> When using the second approach (install followed by 'mvn verify'), I got
> the following:
> 
> [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> resolve dependencies for project
> org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
> dependencies at
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure to
> find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://repository.apache.org/snapshots was cached in the local repository,
> resolution will not be reattempted until the update interval of
> apache.snapshots has elapsed or updates are forced -> [Help 1]
> 
> Looks like ${scala.binary.version} was not substituted for correctly.
> 
> On Thu, Jun 1, 2017 at 8:24 AM, Ted Yu  wrote:
> 
>> When I used the command given by Aljoscha, I got:
>> 
>> https://pastebin.com/8WTGvdFQ
>> 
>> FYI
>> 
>> On Thu, Jun 1, 2017 at 8:17 AM, Aljoscha Krettek 
>> wrote:
>> 
>>> Hi,
>>> 
>>> I think you can use something like
>>> 
>>> mvn verify -am -pl flink-java8
>>> 
>>> (From the base directory)
>>> 
>>> The -pl flag will tell maven to only do that module while -am tells it to
>>> also builds its dependencies. This might or might not also run the tests on
>>> the dependent-upon projects, I’m not sure.
>>> 
>>> As an alternative you can do “mvn clean install …” (skipping tests and
>>> everything) and then switch into the flink-java8 directory and run “mvn
>>> verify” there.
>>> 
>>> Best,
>>> Aljoscha
>>> 
>>> 
 On 1. Jun 2017, at 16:04, Dawid Wysakowicz 
>>> wrote:
 
 Hi devs!
 
 Recently I tried running* mvn verify* just for the *flink-java8* module
>>> (to
 run those tests locally) and it fails with the following error:
 
 [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> resolve dependencies for project
> org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
> dependencies at
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed
>>> to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure
>>> to
> find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT
>>> in
> https://repository.apache.org/snapshots was cached in the local
> repository, resolution will not be reattempted until the update
>>> interval of
> apache.snapshots has elapsed or updates are forced -> [Help 1]
> 
 
 That strategy works for any other module I tried. I would be grateful
>>> for
 any tips how can I run just tests for flink-java8 locally.
 
 Thanks in advance.
 
 Z pozdrowieniami! / Cheers!
 
 Dawid Wysakowicz
 
 *Data/Software Engineer*
 
 Skype: dawid_wys | Twitter: @OneMoreCoder
 
 
>>> 
>>> 
>> 

Re: [DISCUSS] Planning Release 1.4

[Greg Hogan]

I’d like to propose keeping the same schedule but move branch forking from the 
feature freeze to the code freeze. The early fork required duplicate 
verification and commits for numerous bug fixes and minor features which had 
been reviewed but were still queued. There did not look to be much new 
development merged to master between the freezes.

Greg


> On Jun 1, 2017, at 11:26 AM, Robert Metzger  wrote:
> 
> Hi all,
> 
> Flink 1.2 was released on February 2, Flink 1.3 on June 1, which means
> we've managed to release Flink 1.3 in almost exactly 4 months!
> 
> For the 1.4 release, I've put the following deadlines into the wiki [1]:
> 
> *Next scheduled major release*: 1.4.0
> *Feature freeze (branch forking)*:  4. September 2017
> *Code freeze (first voting RC)*:  18 September 2017
> *Release date*: 29 September 2017
> 
> I'll try to send a message every month into this thread to have a countdown
> to the next feature freeze.
> 
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+and+Feature+Plan

Re: Building only flink-java8 module

[Ted Yu]

When using the second approach (install followed by 'mvn verify'), I got
the following:

[ERROR] Failed to execute goal on project flink-java8_2.10: Could not
resolve dependencies for project
org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
dependencies at
org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to read
artifact descriptor for
org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure to
find
org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
https://repository.apache.org/snapshots was cached in the local repository,
resolution will not be reattempted until the update interval of
apache.snapshots has elapsed or updates are forced -> [Help 1]

Looks like ${scala.binary.version} was not substituted for correctly.

On Thu, Jun 1, 2017 at 8:24 AM, Ted Yu  wrote:

> When I used the command given by Aljoscha, I got:
>
> https://pastebin.com/8WTGvdFQ
>
> FYI
>
> On Thu, Jun 1, 2017 at 8:17 AM, Aljoscha Krettek 
> wrote:
>
>> Hi,
>>
>> I think you can use something like
>>
>> mvn verify -am -pl flink-java8
>>
>> (From the base directory)
>>
>> The -pl flag will tell maven to only do that module while -am tells it to
>> also builds its dependencies. This might or might not also run the tests on
>> the dependent-upon projects, I’m not sure.
>>
>> As an alternative you can do “mvn clean install …” (skipping tests and
>> everything) and then switch into the flink-java8 directory and run “mvn
>> verify” there.
>>
>> Best,
>> Aljoscha
>>
>>
>> > On 1. Jun 2017, at 16:04, Dawid Wysakowicz 
>> wrote:
>> >
>> > Hi devs!
>> >
>> > Recently I tried running* mvn verify* just for the *flink-java8* module
>> (to
>> > run those tests locally) and it fails with the following error:
>> >
>> > [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
>> >> resolve dependencies for project
>> >> org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
>> >> dependencies at
>> >> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed
>> to read
>> >> artifact descriptor for
>> >> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure
>> to
>> >> find
>> >> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT
>> in
>> >> https://repository.apache.org/snapshots was cached in the local
>> >> repository, resolution will not be reattempted until the update
>> interval of
>> >> apache.snapshots has elapsed or updates are forced -> [Help 1]
>> >>
>> >
>> > That strategy works for any other module I tried. I would be grateful
>> for
>> > any tips how can I run just tests for flink-java8 locally.
>> >
>> > Thanks in advance.
>> >
>> > Z pozdrowieniami! / Cheers!
>> >
>> > Dawid Wysakowicz
>> >
>> > *Data/Software Engineer*
>> >
>> > Skype: dawid_wys | Twitter: @OneMoreCoder
>> >
>> > 
>>
>>
>

[DISCUSS] Planning Release 1.4

[Robert Metzger]

Hi all,

Flink 1.2 was released on February 2, Flink 1.3 on June 1, which means
we've managed to release Flink 1.3 in almost exactly 4 months!

For the 1.4 release, I've put the following deadlines into the wiki [1]:

*Next scheduled major release*: 1.4.0
*Feature freeze (branch forking)*:  4. September 2017
*Code freeze (first voting RC)*:  18 September 2017
*Release date*: 29 September 2017

I'll try to send a message every month into this thread to have a countdown
to the next feature freeze.


[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+and+Feature+Plan

Re: Building only flink-java8 module

[Aljoscha Krettek]

Hi,

I think you can use something like

mvn verify -am -pl flink-java8

(From the base directory)

The -pl flag will tell maven to only do that module while -am tells it to also 
builds its dependencies. This might or might not also run the tests on the 
dependent-upon projects, I’m not sure.

As an alternative you can do “mvn clean install …” (skipping tests and 
everything) and then switch into the flink-java8 directory and run “mvn verify” 
there.

Best,
Aljoscha


> On 1. Jun 2017, at 16:04, Dawid Wysakowicz  wrote:
> 
> Hi devs!
> 
> Recently I tried running* mvn verify* just for the *flink-java8* module (to
> run those tests locally) and it fails with the following error:
> 
> [ERROR] Failed to execute goal on project flink-java8_2.10: Could not
>> resolve dependencies for project
>> org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
>> dependencies at
>> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to read
>> artifact descriptor for
>> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure to
>> find
>> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
>> https://repository.apache.org/snapshots was cached in the local
>> repository, resolution will not be reattempted until the update interval of
>> apache.snapshots has elapsed or updates are forced -> [Help 1]
>> 
> 
> That strategy works for any other module I tried. I would be grateful for
> any tips how can I run just tests for flink-java8 locally.
> 
> Thanks in advance.
> 
> Z pozdrowieniami! / Cheers!
> 
> Dawid Wysakowicz
> 
> *Data/Software Engineer*
> 
> Skype: dawid_wys | Twitter: @OneMoreCoder
> 
> 

[jira] [Created] (FLINK-6804) Inconsistent state migration behaviour between different state backends

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6804:


 Summary: Inconsistent state migration behaviour between different 
state backends
 Key: FLINK-6804
 URL: https://issues.apache.org/jira/browse/FLINK-6804
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing, Type Serialization System
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann
Priority: Critical


The {{MemoryStateBackend}}, {{FsStateBackend}} and {{RocksDBStateBackend}} show 
a different behaviour when it comes to recovery from old state and state 
migration. For example, using the {{MemoryStateBackend}} it is possible to 
recover pojos which now have additional fields (at recovery time). The only 
caveat is that the recovered {{PojoSerializer}} will silently drop the added 
fields when writing the new Pojo. In contrast, the {{RocksDBStateBackend}} 
correctly recognizes that a state migration is necessary and thus fails with a 
{{StateMigrationException}}. The same applies to the case where Pojo field 
types change. The {{MemoryStateBackend}} and the {{FsStateBackend}} accept such 
a change as long as the fields still have the same length. The 
{{RocksDBStateBackend}} correctly fails with a {{StateMigrationException}}.

I think that all state backends should behave similarly and give the user the 
same recovery and state migration guarantees. Otherwise, it could happen that 
jobs run with one state backend but not with another (wrt semantic behaviour).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[ANNOUNCE] Apache Flink 1.3.0 released

[Robert Metzger]

The Apache Flink community is pleased to announce the release of Apache
Flink 1.3.0.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.


The release is available for download at:

https://flink.apache.org/downloads.html

There is a blog post available on the Flink site explaining most of the new
features:

   https://flink.apache.org/news/2017/06/01/release-1.3.0.html

The full changelog is available at:
https://flink.apache.org/blog/release_1.3.0-changelog.html



We would like to thank all contributors who made this release possible!

[jira] [Created] (FLINK-6803) Add test for PojoSerializer when Pojo changes

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6803:


 Summary: Add test for PojoSerializer when Pojo changes
 Key: FLINK-6803
 URL: https://issues.apache.org/jira/browse/FLINK-6803
 Project: Flink
  Issue Type: Improvement
  Components: Type Serialization System
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


We should add test cases for the {{PojoSerializer}} when the underlying Pojo 
type changes in order to test the proper behaviour of the serializer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6802) PojoSerializer does not create ConvertDeserializer for removed/added fields

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6802:


 Summary: PojoSerializer does not create ConvertDeserializer for 
removed/added fields
 Key: FLINK-6802
 URL: https://issues.apache.org/jira/browse/FLINK-6802
 Project: Flink
  Issue Type: Improvement
  Components: Type Serialization System
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann


When calling {{PojoSerializer#ensureCompatibility}}, the PojoSerializer checks 
for compatibility. Currently, the method only construct a ConvertDeserializer 
if the number of old and new pojo fields is exactly the same. However, given 
the {{TypeSerializerConfigurationSnapshots}} and the current set of fields, it 
should also be possible to construct a ConvertDeserializer if new fields were 
added or old fields removed from the Pojo. I think that we should add this 
functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Building only flink-java8 module

[Dawid Wysakowicz]

Hi devs!

Recently I tried running* mvn verify* just for the *flink-java8* module (to
run those tests locally) and it fails with the following error:

[ERROR] Failed to execute goal on project flink-java8_2.10: Could not
> resolve dependencies for project
> org.apache.flink:flink-java8_2.10:jar:1.4-SNAPSHOT: Failed to collect
> dependencies at
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.10:jar:1.4-SNAPSHOT: Failure to
> find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://repository.apache.org/snapshots was cached in the local
> repository, resolution will not be reattempted until the update interval of
> apache.snapshots has elapsed or updates are forced -> [Help 1]
>

That strategy works for any other module I tried. I would be grateful for
any tips how can I run just tests for flink-java8 locally.

Thanks in advance.

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder

[jira] [Created] (FLINK-6801) PojoSerializerConfigSnapshot cannot deal with missing Pojo fields

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6801:


 Summary: PojoSerializerConfigSnapshot cannot deal with missing 
Pojo fields
 Key: FLINK-6801
 URL: https://issues.apache.org/jira/browse/FLINK-6801
 Project: Flink
  Issue Type: Bug
  Components: Type Serialization System
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann


The {{PojoSerializerConfigSnapshot}} tries to deserialize Pojo {{Fields}}. If 
the underlying Pojo class changed such that a formerly present field is not 
longer part of the Pojo, then the deserialization of the 
{{PojoSerializerConfigSnapshot}} will fail. As a consequence of this failure, 
there is no possibility to do state migration (e.g. skipping over this field 
when reading old data).

I think it is better to simply store the field name in the 
{{PojoSerializerConfigSnapshot}} and try to resolve the fields in the 
{{PojoSerializer#ensureCompatibility}} method. That way it is possible to 
construct a ConvertDeserializer allowing for state migration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6800) PojoSerializer ignores added pojo fields

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6800:


 Summary: PojoSerializer ignores added pojo fields
 Key: FLINK-6800
 URL: https://issues.apache.org/jira/browse/FLINK-6800
 Project: Flink
  Issue Type: Bug
  Components: Type Serialization System
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann


The {{PojoSerializer}} contains a list of pojo fields which are represented as 
{{Field}} instances. Upon serialization the names of these fields are 
serialized. When being deserialized these names are used to look up the 
respective {{Fields}} of a dynamically loaded class. If the dynamically loaded 
class has additional fields (compared to when the serializer was serialized), 
then these fields will be ignored (for the read and for the write path). While 
this is necessary to read stored data, it is dangerous when writing new data, 
because all newly added fields won't be serialized. This subtleness is really 
hard to detect for the user. Therefore, I think we should eagerly fail if the 
newly loaded type contains new fields which haven't been present before.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6799) PojoSerializer does not check field types when being deserialized

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6799:


 Summary: PojoSerializer does not check field types when being 
deserialized
 Key: FLINK-6799
 URL: https://issues.apache.org/jira/browse/FLINK-6799
 Project: Flink
  Issue Type: Bug
  Components: Type Serialization System
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann


The {{PojoSerializer}} stores internally the fields of a pojo as {{Field}} 
instances. These field instances are serialized via the {{FieldSerializer}}. 
The {{FieldSerializer}} simply writes out the name of the field. When being 
deserialized, the serializer reads the name and then looks up the field using 
the dynamically loaded class. If the underlying class changes, e.g. type of 
fields change, then this won't be noticed, because we don't check the field 
types. In order to catch errors early and to give meaningful error messages, I 
think we should add type checks for the reloaded fields.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6798) Remove/update documentation about network buffer tuning

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-6798:
-

 Summary: Remove/update documentation about network buffer tuning
 Key: FLINK-6798
 URL: https://issues.apache.org/jira/browse/FLINK-6798
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Network
Reporter: Robert Metzger


{quote}The number of network buffers is a parameter that can currently have an 
effect on checkpointing at large scale. The Flink community is working on 
eliminating that parameter in the next versions of Flink.
{quote} 
in 
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/large_state_tuning.html#tuning-network-buffers




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6797) building docs fails with bundler 1.15

[David Anderson (JIRA)]

David Anderson created FLINK-6797:
-

 Summary: building docs fails with bundler 1.15
 Key: FLINK-6797
 URL: https://issues.apache.org/jira/browse/FLINK-6797
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: David Anderson
Assignee: David Anderson
Priority: Critical
 Fix For: 1.3.1, 1.4.0


The script for building the docs installs the latest version of the bundler 
ruby gem (if it can't find the bundle command, which is always the case on the 
build-bots, for example). Since the release of bundler 1.15 this fails because 
it is now pickier about dependency checking, and we somehow ended up with an 
invalid dependency rule in Gemfile.lock.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Willing to contribute to Flink!

[Chesnay Schepler]

I've given you contributor permissions.

On 01.06.2017 15:10, Robert Metzger wrote:

Hi,

you don't need to file an ICLA. Can you please subscribe to the
dev@flink.apache.org mailing list by sending an empty email to
dev-subscr...@flink.apache.org?
So far, your messages were manually accepted.

On Thu, Jun 1, 2017 at 2:27 PM, jinchul  wrote:


Hello,

Thanks for your kind answer. My JIRA's username is "jinchul". By the way,
should I fill out and send a ICLA document before contributing?

Best regards,
Jinchul



--
View this message in context: http://apache-flink-mailing-
list-archive.1008284.n3.nabble.com/Willing-to-contribute-to-Flink-
tp17855p17864.html
Sent from the Apache Flink Mailing List archive. mailing list archive at
Nabble.com.

Re: Willing to contribute to Flink!

[Robert Metzger]

Hi,

you don't need to file an ICLA. Can you please subscribe to the
dev@flink.apache.org mailing list by sending an empty email to
dev-subscr...@flink.apache.org?
So far, your messages were manually accepted.

On Thu, Jun 1, 2017 at 2:27 PM, jinchul  wrote:

> Hello,
>
> Thanks for your kind answer. My JIRA's username is "jinchul". By the way,
> should I fill out and send a ICLA document before contributing?
>
> Best regards,
> Jinchul
>
>
>
> --
> View this message in context: http://apache-flink-mailing-
> list-archive.1008284.n3.nabble.com/Willing-to-contribute-to-Flink-
> tp17855p17864.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: Willing to contribute to Flink!

[jinchul]

Hello,

Thanks for your kind answer. My JIRA's username is "jinchul". By the way,
should I fill out and send a ICLA document before contributing?

Best regards,
Jinchul



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Willing-to-contribute-to-Flink-tp17855p17864.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

[jira] [Created] (FLINK-6796) Allow setting the user code class loader for AbstractStreamOperatorTestHarness

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-6796:


 Summary: Allow setting the user code class loader for 
AbstractStreamOperatorTestHarness
 Key: FLINK-6796
 URL: https://issues.apache.org/jira/browse/FLINK-6796
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Priority: Minor


For testing purposes it would be good to be able to specify a {{ClassLoader}} 
for the {{AbstractStreamOperatorTestHarness}}. At the moment, the application 
ClassLoader will be used. I propose to take the {{ClassLoader}} provided by the 
{{Environment}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Release Apache Flink 1.3.0 (RC3)

[Robert Metzger]

Hi Timo,

I agree that not having the documentation available for the new features is
not good.
I'll do a pass over all the open documentation PRs and try to merge as many
as possible. If I have the feeling, that we have enough docs, I'll release
1.3.0.
For the table API, I can put a note into the release announcement.



On Wed, May 31, 2017 at 4:52 PM, Timo Walther  wrote:

> What do you think about waiting with the release announcement for Flink
> 1.3.1 until next week.
>
> IMHO the documentation is not in a good shape for a release annoucement
> right now anyway.
>
> Most of the new features of the Table API are not documented. Docs for
> other features are missing as well or exist in open PR [1].
>
> Regards,
> Timo
>
> [1] https://issues.apache.org/jira/browse/FLINK-6674
>
>
> Am 31.05.17 um 15:03 schrieb Aljoscha Krettek:
>
> Yes, FLINK-6783 might even have been a release blocker…. It’s a new
>> feature that simply doesn’t work in most cases.
>>
>> On 31. May 2017, at 14:51, Timo Walther  wrote:
>>>
>>> We should also include FLINK-6783. It seems that
>>> WindowedStream::aggregate is broken right now.
>>>
>>>
>>> Am 31.05.17 um 14:31 schrieb Timo Walther:
>>>
 I merged all Table API related PRs.

 I'm also fine with a 1.3.1 release this or next week.


 Am 31.05.17 um 14:08 schrieb Till Rohrmann:

> I would be ok to quickly release 1.3.1 once the the respective PRs have
> been merged.
>
> Just for your information, I'm not yet through with the testing of the
> type
> serializer upgrade feature, though.
>
> Cheers,
> Till
>
> On Wed, May 31, 2017 at 12:14 PM, Stefan Richter <
> s.rich...@data-artisans.com> wrote:
>
> +1 for releasing now and providing a 1.3.1 release soon.
>>
>> Am 31.05.2017 um 11:02 schrieb Gyula Fóra :
>>>
>>> Hi All,
>>>
>>> I also lean towards getting the release out as soon as possible given
>>>
>> that
>>
>>> it had been delayed quite a bit and there is no major issue without a
>>> straightforward workaround (agreeing with Nico and Kostas). I am sure
>>>
>> once
>>
>>> people will start using the new features we will see more issues that
>>> should be fixed asap in 1.3.1.
>>>
>>> Regarding the critical bug Till had found, we could add a line about
>>> it
>>>
>> to
>>
>>> the release notes so that people don't get blocked by it as there is
>>> a
>>> workaround possible.
>>>
>>> Regards,
>>> Gyula
>>>
>>>
>>> Kostas Kloudas  ezt írta (időpont:
>>> 2017.
>>>
>> máj.
>>
>>> 31., Sze, 10:53):
>>>
>>> Hi all,

 I also tend to agree with the argument that says a release should
 be out
 as soon as possible, given that 1) it improves
 usability/functionality

>>> and
>>
>>> 2) at a minimum, it does not include new known bugs. The arguments
 are
 more or less aligned with Nico’s response on the matter.

 Focusing on the bug that spiked the current discussion, I agree with

>>> Till
>>
>>> that this is alarming, as it passed all previous testing efforts,
 but I
 have to
 add that if nobody so far encountered it, we could release 1.3 now
 and

>>> fix
>>
>>> it in the upcoming 1.3.1.

 Kostas

 On May 31, 2017, at 10:20 AM, Nico Kruber 
>
 wrote:

> IMHO, any release that improves things and does not break anything
> is
>
 worth

> releasing and should not be blocked on bugs that it did not cause.
> There will always be a next (minor/major) release that may fix
> this at
>
 a
>>
>>> later

> time, given that the time between releases is not too high.
>
> Consider someone waiting for a bugfix/feature that made it into
> 1.3.0
>
 who--if

> delayed--would have to wait even longer for "his" bugfix/feature.
> Any
>
 new
>>
>>> bugfixes (and there will always be more) can wait a few more days or
>
 even a few

> weeks and may be fixed in 1.3.1 or so.
>
>
> Nico
>
> On Tuesday, 30 May 2017 20:21:41 CEST Till Rohrmann wrote:
>
>> - Not sure whether it's a good argument to defer fixing major bugs
>>
> because

> they have not been introduced with 1.3.0. It's actually alarming
>> that
>>
> these

> things have not been found earlier given that we test our releases
>> thoroughly.

[jira] [Created] (FLINK-6795) Activate checkstyle for runtime/process

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6795:
---

 Summary: Activate checkstyle for runtime/process
 Key: FLINK-6795
 URL: https://issues.apache.org/jira/browse/FLINK-6795
 Project: Flink
  Issue Type: Improvement
  Components: Local Runtime
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6794) Activate checkstyle for migration/*

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6794:
---

 Summary: Activate checkstyle for  migration/*
 Key: FLINK-6794
 URL: https://issues.apache.org/jira/browse/FLINK-6794
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Trivial
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6793) Activate checkstyle for runtime/metrics

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6793:
---

 Summary: Activate checkstyle for runtime/metrics
 Key: FLINK-6793
 URL: https://issues.apache.org/jira/browse/FLINK-6793
 Project: Flink
  Issue Type: Improvement
  Components: Metrics
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Trivial
 Fix For: 1.4.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6792) flink-yarn-tests always fail on travis

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6792:
---

 Summary: flink-yarn-tests always fail on travis
 Key: FLINK-6792
 URL: https://issues.apache.org/jira/browse/FLINK-6792
 Project: Flink
  Issue Type: Bug
  Components: Tests, YARN
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Priority: Critical


{{flink-yarn-tests}} is currently failing all builds:

{code}
Results :



Failed tests: 

  YARNSessionFIFOITCase.testJavaAPI:244 Error while deploying YARN cluster: 
Couldn't deploy Yarn cluster

Tests in error: 

  YARNHighAvailabilityITCase.testMultipleAMKill:140 » Runtime Couldn't deploy 
Ya...

  
YARNSessionCapacitySchedulerITCase.perJobYarnCluster:120->YarnTestBase.runWithArgs:612
 » Runtime

  
YARNSessionCapacitySchedulerITCase.perJobYarnClusterWithParallelism:344->YarnTestBase.runWithArgs:612
 » Runtime

  
YARNSessionCapacitySchedulerITCase.testClientStartup:99->YarnTestBase.runWithArgs:528->YarnTestBase.runWithArgs:612
 » Runtime

  
YARNSessionCapacitySchedulerITCase.testDetachedPerJobYarnCluster:373->testDetachedPerJobYarnClusterInternal:419->YarnTestBase.startWithArgs:515
 » Runtime

  
YARNSessionCapacitySchedulerITCase.testDetachedPerJobYarnClusterWithStreamingJob:390->testDetachedPerJobYarnClusterInternal:419->YarnTestBase.startWithArgs:515
 » Runtime

  
YARNSessionCapacitySchedulerITCase.testTaskManagerFailure:140->YarnTestBase.startWithArgs:515
 » Runtime

  YARNSessionFIFOITCase.testDetachedMode:84->YarnTestBase.startWithArgs:515 » 
Runtime

  
YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.testDetachedMode:84->YarnTestBase.startWithArgs:515
 » Runtime}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6791) Using MemoryStateBackend as checkpoint stream back-end may block checkpoint/savepoint creation

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-6791:
--

 Summary: Using MemoryStateBackend as checkpoint stream back-end 
may block checkpoint/savepoint creation
 Key: FLINK-6791
 URL: https://issues.apache.org/jira/browse/FLINK-6791
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.2.1, 1.3.0
Reporter: Nico Kruber


If the `MemoryStateBackend` is used as the checkpoint stream back-end in e.g. 
RocksDBStateBackend, it will block further checkpoint/savepoint creation if the 
checkpoint data reaches the back-end's max state size. In that case, an error 
message is logged at the task manager but the save-/checkpoint never completes 
and although the job continues, no further checkpoints will be made.

Please see the following example that should be reproducible:

{code:java}
env.setStateBackend(new RocksDBStateBackend(new MemoryStateBackend(1000 * 1024 
* 1024, false), false));

env.enableCheckpointing(100L);

final long numKeys = 100_000L;
DataStreamSource source1 =
env.addSource(new RichParallelSourceFunction() {
private volatile boolean running = true;

@Override
public void run(SourceContext ctx) throws 
Exception {
long counter = 0;

while (running) {
synchronized (ctx.getCheckpointLock()) {
ctx.collect(Tuple1.of(counter % 
numKeys));
counter++;
}

Thread.yield();
}
}

@Override
public void cancel() {
running = false;
}
});

source1.keyBy(0)
.map(new RichMapFunction() {
private transient ValueState val;

@Override
public Tuple1 map(Tuple1 value)
throws Exception {
val.update(Collections.nCopies(100, value.f0));
return value;
}

@Override
public void open(final Configuration parameters) throws 
Exception {
ValueStateDescriptor descriptor =
new ValueStateDescriptor<>(
"data", // the 
state name

TypeInformation.of(new TypeHint() {
}) // type 
information
);
val = getRuntimeContext().getState(descriptor);
}
}).uid("identity-map-with-state")
.addSink(new DiscardingSink());

env.execute("failingsnapshots");
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Willing to contribute to Flink!

[Chesnay Schepler]

Hello,

it's great that you want to contribute!

If you give me your JIRA user name i can give you contributor 
permissions, so you can assign issues to yourself.
Note that parts of JIRA are not /completely/ up to date (especially in 
the older parts); when you're unsure whether
the JIRA is still valid, or if it would take a lot of time to implement, 
it's generally a good idea to ask for someone's

input in the JIRA.

Regards,
Chesnay

On 01.06.2017 01:55, Jin Chul Kim wrote:

Hello all,

I am very interesting in Flink framework, especially development. Since I
joined a new company below, I have been fallen into OSS. I haven't taken a
chance to contribute to OSS, but I am happy if I would take an opportunity
to be a developer for Flicker.

I read "How to contribute" section and here is my question.

- May I take a "starter" issue? I found some unassigned issues on JIRA.
AFAIK, I don't have a privilege to take it. If I interest in an issue, I
just leave a comment on JIRA, taking a look at and push a commit on my
forked repository. Please correct me if my understanding is wrong.

Please read below if you are interested in my background.

Currently I am working at SK telecom in Republic of Korea. In this company,
I am a member of developers to build up real-time analyzing system for
telco and 3rd party data. We are currently using Flink for the real-time
processing.

Before joining the company, I had been worked SAP Labs Korea for about ten
years and I was one of core developers for SAP HANA in-memory data
platform. In SAP, I contributed SQL/SQLScript compiler(i.e. parser to build
AST, semantic checker, design of intermediate representation(IR), code
generator for executable code), SQL plan cache, SQL query optimizer using
cost-based/rule-based, query plan distribution on multi nodes and so on. I
was struggling to resolve complexity problems such as finding performance
bottleneck on distributed landscape, lots of major customers issues/bugs.

Thanks!

Best regards,
Jinchul