Re: Does `flink-connector-filesystem` work with Hadoop-Free Flink?

[Stephan Ewen]

Hi!

You could try to have Hadoop in your application Jar file, but I expect
trouble with s3a, because of its specific way they do connection pooling.

Making the Bucketing Sink work with Flink's file systems (and thus with
Hadoop-free Flink) is super high up on the list as soon as the release is
out.

Stephan


On Fri, Feb 23, 2018 at 2:32 PM, Jamie Grier  wrote:

> Yeah, I meant that latter..  but it sounds like it could be just asking for
> trouble.  I just like the idea of keeping the set of un-shaded JARs in the
> flink/lib directory to a minimum..
>
> Thanks.
>
> On Fri, Feb 23, 2018 at 5:29 AM, Aljoscha Krettek 
> wrote:
>
> > You mean putting the Flink-native S3 filesystem in the user jar or Hadoop
> > in the user jar. The former wouldn't work, I think, because the
> FileSystems
> > are being initialised before the user-jar is loaded. The latter might
> work
> > but only if you don't have Hadoop in the classpath, i.e. not on YARN and
> > only on a Hadoop-free cluster. Maybe...
> >
> > > On 23. Feb 2018, at 13:32, Jamie Grier  wrote:
> > >
> > > Thanks, Aljoscha :)
> > >
> > > So is it possible to continue to use the new "native' fllesystems along
> > > with the BucketingSink by including the Hadoop dependencies only in the
> > > user's uber jar? Or is that asking for trouble?  Has anyone tried that
> > > successfully?
> > >
> > > -Jamie
> > >
> > >
> > > On Fri, Feb 23, 2018 at 12:39 AM, Aljoscha Krettek <
> aljos...@apache.org>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm afraid not, since the BucketingSink uses the Hadoop FileSystem
> > >> directly and not the Flink FileSystem abstraction. The flink-s3-fs-*
> > >> modules only provide Flink FileSystems.
> > >>
> > >> One of the goals for 1.6 is to provide a BucketingSink that uses the
> > Flink
> > >> FileSystem and also works well with eventually consistent file
> systems.
> > >>
> > >> --
> > >> Aljoscha
> > >>
> > >>> On 23. Feb 2018, at 06:31, Jamie Grier  wrote:
> > >>>
> > >>> Is the `flink-connector-filesystem` connector supposed to work with
> the
> > >>> latest hadoop-free Flink releases, say along with the
> > >> `flink-s3-fs-presto`
> > >>> filesystem implementation?
> > >>>
> > >>> -Jamie
> > >>
> > >>
> >
> >
>

Re: Timestamp/watermark support in Kinesis consumer

[Thomas Weise]

Another nice thing is that readers can potentially also read from different
sources (historic/latest). To arrive at a general connector pattern, it
will also be necessary to consider the ordering relationship between
restrictions/splits/blocks/segments when it is important for the processing
logic - which is what Jamie refers to. For Kinesis, the most obvious case
is reading parent before child shards, but also throttling unrelated shards
if they are unbalanced WRT event time.

We are now implementing a stop gap watermark solution in custom code
because the Kinesis consumer really needs a revamp as part of general
connector overhaul.


On Thu, Feb 22, 2018 at 2:28 AM, Aljoscha Krettek 
wrote:

> While we're on this: https://beam.apache.org/blog/
> 2017/08/16/splittable-do-fn.html
>
> This is a concrete way of separating partition/shard/split discovery from
> their reading. The nice thing about this is that you can mix-and-match
> "discovery components" and "reader components". For example, for Kafka we
> would have a TopicReader and I can envision different discovery
> implementations: one very simple, no-frills, but rock solid, another one
> that does automatic discovery of new partitions, regex matching, etc...
>
>
> > On 22. Feb 2018, at 01:49, Jamie Grier  wrote:
> >
> > I know this is a very simplistic idea but...
> >
> > In general the issue Eron is describing occurs whenever two (or more)
> > parallel partitions are assigned to the same Flink sub-task and there is
> > large time delta between them.  This problem exists though largely
> because
> > we are not making any decisions about which of these partitions to read
> and
> > when but rather just treating them all the same.  However, this isn't the
> > only way to approach the problem.
> >
> > Think instead of each partition as a "roughly time sorted" file and the
> > function of the connector as roughly a merge sort type process.  In other
> > words just read the older data first by peeking at each partition and
> > deciding what to read next.  The output of the connector would be a
> roughly
> > time ordered stream that way..
> >
> > However to really solve the whole problem you'd have to carry this idea
> > throughout Flink and be more selective about which data you read and when
> > throughout the whole data flow graph.  Similar problem I think and just
> > something I've been thinking a bit about lately.
> >
> >
> >
> >
> > On Mon, Feb 12, 2018 at 7:12 PM, Eron Wright 
> wrote:
> >
> >> It is valuable to consider the behavior of a consumer in both a
> real-time
> >> processing context, which consists mostly of tail reads, and a
> historical
> >> processing context, where there's an abundance of backlogged data.   In
> the
> >> historical processing context, system internals (e.g. shard selection
> >> logic) have an outsized influence on the order of observation and
> >> potentially the progression of the event time clock.  In a real-time
> >> context, the order of observation is, by definition, mostly determined
> by
> >> the order in which events are produced.
> >>
> >> My point is, it would be good to explore the efficacy of these
> improvements
> >> in both contexts.
> >>
> >>
> >>
> >>
> >> On Mon, Feb 12, 2018 at 5:10 PM, Thomas Weise  wrote:
> >>
> >>> I don't think there is a generic solution to the problem you are
> >>> describing; we don't know how long it will take for resharding to take
> >>> effect and those changes to become visible to the connector. Depending
> on
> >>> how latency sensitive the pipeline is, possibly a configurable
> watermark
> >>> hold period could be used to cushion the event time chaos introduced by
> >>> resharding.
> >>>
> >>> This isn't the primary motivation for the connector customization I'm
> >>> working on though. We face issues with restart from older checkpoints
> >> where
> >>> parent and child shards are consumed in parallel.
> >>>
> >>>
> >>> --
> >>> sent from mobile
> >>>
> >>>
> >>> On Feb 12, 2018 4:36 PM, "Eron Wright"  wrote:
> >>>
> >>> I'd like to know how you envision dealing with resharding in relation
> to
> >>> the watermark state.   Imagine that a given shard S1 has a watermark of
> >> T1,
> >>> and is then split into two shards S2 and S3.   The new shards are
> >> assigned
> >>> to subtasks according to a hash function.  The current watermarks of
> >> those
> >>> subtasks could be far ahead of T1, and thus the events in S2/S3 will be
> >>> considered late.
> >>>
> >>> The problem of a chaotic event time clock is exacerbated by any source
> >> that
> >>> uses dynamic partitioning.  Would a per-shard watermark generator
> really
> >>> solve the problem that is motivating you?
> >>>
> >>> Thanks,
> >>> Eron
> >>>
> >>> On Mon, Feb 12, 2018 at 10:35 AM, Thomas Weise  wrote:
> >>>
>  Based on my draft implementation, the changes that are needed in the
> >>> Flink
> 

[jira] [Created] (FLINK-8771) Upgrade scalastyle to 1.0.0

[Ted Yu (JIRA)]

Ted Yu created FLINK-8771:
-

 Summary: Upgrade scalastyle to 1.0.0
 Key: FLINK-8771
 URL: https://issues.apache.org/jira/browse/FLINK-8771
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: Ted Yu


scalastyle 1.0.0 fixes issue with import order, explicit type for public 
methods, line length limitation and comment validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8770) CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager is restarted it fails to recover the job due to "checkpoint FileNotFound exception"

[Xinyang Gao (JIRA)]

Xinyang Gao created FLINK-8770:
--

 Summary: CompletedCheckPoints stored on ZooKeeper is not 
up-to-date, when JobManager is restarted it fails to recover the job due to 
"checkpoint FileNotFound exception"
 Key: FLINK-8770
 URL: https://issues.apache.org/jira/browse/FLINK-8770
 Project: Flink
  Issue Type: Bug
  Components: Local Runtime
Affects Versions: 1.4.0
Reporter: Xinyang Gao


Hi, I am running a Flink cluster (1 JobManager + 6 TaskManagers) with HA mode 
on OpenShift, I have enabled Chaos Monkey which kills either JobManager or one 
of the TaskManager in every 5 minutes, ZooKeeper quorum is stable with no chaos 
monkey enabled. Flink reads data from one Kafka topic and writes data into 
another Kafka topic. Checkpoint surely is enabled, with 1000ms interval. 
state.checkpoints.num-retained is set to 10. I am using PVC for state backend 
(checkpoint, recovery, etc), so the checkpoints and states are persistent. 

The restart strategy for Flink jobmanager DeploymentConfig is 
{color:#d04437}recreate, {color:#33}which means it will kill the old 
container of jobmanager before it restarts the new one.{color}{color}

I have run the Chaos test for one day at first, however I have seen the 
exception:

{color:#FF}org.apache.flink.util.FlinkException: Could not retrieve 
checkpoint *** from state handle under /***. This indicates that the retrieved 
state handle is broken. Try cleaning the state handle store. {color:#33}and 
the root cause is checkpoint {color:#d04437}FileNotFound{color}. {color}{color}

{color:#FF}{color:#33}then the Flink job keeps restarting for a few 
hours and due to the above error it cannot be restarted successfully. 
{color}{color}

{color:#FF}{color:#33}After further investigation, I have found the 
following facts in my PVC:{color}{color}

 

{color:#d04437}-rw-r--r--. 1 flink root 11379 Feb 23 02:10 
completedCheckpoint0ee95157de00
-rw-r--r--. 1 flink root 11379 Feb 23 01:51 completedCheckpoint498d0952cf00
-rw-r--r--. 1 flink root 11379 Feb 23 02:10 completedCheckpoint650fe5b021fe
-rw-r--r--. 1 flink root 11379 Feb 23 02:10 completedCheckpoint66634149683e
-rw-r--r--. 1 flink root 11379 Feb 23 02:11 completedCheckpoint67f24c3b018e
-rw-r--r--. 1 flink root 11379 Feb 23 02:10 completedCheckpoint6f64ebf0ae64
-rw-r--r--. 1 flink root 11379 Feb 23 02:11 completedCheckpoint906ebe1fb337
-rw-r--r--. 1 flink root 11379 Feb 23 02:11 completedCheckpoint98b79ea14b09
-rw-r--r--. 1 flink root 11379 Feb 23 02:10 completedCheckpointa0d1070e0b6c
-rw-r--r--. 1 flink root 11379 Feb 23 02:11 completedCheckpointbd3a9ba50322
-rw-r--r--. 1 flink root 11355 Feb 22 17:31 completedCheckpointd433b5e108be
-rw-r--r--. 1 flink root 11379 Feb 22 22:56 completedCheckpointdd0183ed092b
-rw-r--r--. 1 flink root 11379 Feb 22 00:00 completedCheckpointe0a5146c3d81
-rw-r--r--. 1 flink root 11331 Feb 22 17:06 completedCheckpointec82f3ebc2ad
-rw-r--r--. 1 flink root 11379 Feb 23 02:11 
completedCheckpointf86e460f6720{color}

 

{color:#33}The latest 10 checkpoints are created at about 02:10, if you 
ignore the old checkpoints which were not deleted successfully (which I do not 
care too much).{color}

 

{color:#33}However when checking on ZooKeeper, I see the followings in 
flink/checkpoints path (I only list one, but the other 9 are similar){color}

{color:#d04437}cZxid = 0x160001ff5d
��sr;org.apache.flink.runtime.state.RetrievableStreamStateHandle�U�+LwrappedStreamStateHandlet2Lorg/apache/flink/runtime/state/StreamStateHandle;xpsr9org.apache.flink.runtime.state.filesystem.FileStateHandle�u�b�▒▒J
 
stateSizefilePathtLorg/apache/flink/core/fs/Path;xp,ssrorg.apache.flink.core.fs.PathLuritLjava/net/URI;xpsr
 
java.net.URI�x.C�I�LstringtLjava/lang/String;xpt=file:/mnt/flink-test/recovery/completedCheckpointd004a3753870x
[zk: localhost:2181(CONNECTED) 7] ctime = Fri Feb 23 02:08:18 UTC 2018
mZxid = 0x160001ff5d
mtime = Fri Feb 23 02:08:18 UTC 2018
pZxid = 0x1d0c6d
cversion = 31
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 492{color}

{color:#FF}{color:#33} {color}{color}

so the latest completedCheckpoints status stored on ZooKeeper is at about 
{color:#d04437}02:08, {color:#33}which implies that the completed 
checkpoints at{color}{color} {color:#d04437}02:10 {color:#33}somehow are 
not successfully submitted to ZooKpeer, so when it tries to restart the Flink 
job it is not able to find the latest checkpoint thus being 
failed.{color}{color}

{color:#d04437}{color:#33}I am very suprised by this since seems writing 
checkpoint to zookeeper is [synchronous 
|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ZooKeeperCompletedCheckpointStore.java#L222],
 so I am not sure why this happens. Can anyone help looks at this 
?{color}{color}



--
This message was sent by 

RowSerializer

[Radu Tudoran]

Hi,

I was using the RowSerializer (package 
org.apache.flink.api.java.typeutils.runtime;) recently to serialize Rows to 
file (for reading them back in the future).
I observed a strange behavior that I would like to double check with you in 
case this is a serious problem to be addressed:
When the rowserializer is used to convert data back, there is no check for the 
consistency of the data (e.g., size of the object that was serialized, 
checksum...). This leads to situation that for random reads of bytes it can 
happen that inconsistent objects are deserialized, which of course can lead to 
inconsistent data.

For example if we serialize object of the form (Int, Long, Double, String 
String) - if we have available only 1/3 of the bytes, we can end up reading 
back objects  as (0,0,0,null, null) - this is not the only example of how the 
object can be incorrectly deserialized ...rather than having an error for this 
process.

Hence, I wanted to double check if this is an intended behavior for some reason 
and if we should consider fixing the rowserializer to guarantee integrity of 
the objects that are deserialized.

Best regards,

Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Center
Munich Office
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Qiuen Peng, Shengli Wang
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Qiuen Peng, Shengli Wang
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

Re: Does `flink-connector-filesystem` work with Hadoop-Free Flink?

[Jamie Grier]

Yeah, I meant that latter..  but it sounds like it could be just asking for
trouble.  I just like the idea of keeping the set of un-shaded JARs in the
flink/lib directory to a minimum..

Thanks.

On Fri, Feb 23, 2018 at 5:29 AM, Aljoscha Krettek 
wrote:

> You mean putting the Flink-native S3 filesystem in the user jar or Hadoop
> in the user jar. The former wouldn't work, I think, because the FileSystems
> are being initialised before the user-jar is loaded. The latter might work
> but only if you don't have Hadoop in the classpath, i.e. not on YARN and
> only on a Hadoop-free cluster. Maybe...
>
> > On 23. Feb 2018, at 13:32, Jamie Grier  wrote:
> >
> > Thanks, Aljoscha :)
> >
> > So is it possible to continue to use the new "native' fllesystems along
> > with the BucketingSink by including the Hadoop dependencies only in the
> > user's uber jar? Or is that asking for trouble?  Has anyone tried that
> > successfully?
> >
> > -Jamie
> >
> >
> > On Fri, Feb 23, 2018 at 12:39 AM, Aljoscha Krettek 
> > wrote:
> >
> >> Hi,
> >>
> >> I'm afraid not, since the BucketingSink uses the Hadoop FileSystem
> >> directly and not the Flink FileSystem abstraction. The flink-s3-fs-*
> >> modules only provide Flink FileSystems.
> >>
> >> One of the goals for 1.6 is to provide a BucketingSink that uses the
> Flink
> >> FileSystem and also works well with eventually consistent file systems.
> >>
> >> --
> >> Aljoscha
> >>
> >>> On 23. Feb 2018, at 06:31, Jamie Grier  wrote:
> >>>
> >>> Is the `flink-connector-filesystem` connector supposed to work with the
> >>> latest hadoop-free Flink releases, say along with the
> >> `flink-s3-fs-presto`
> >>> filesystem implementation?
> >>>
> >>> -Jamie
> >>
> >>
>
>

Re: Does `flink-connector-filesystem` work with Hadoop-Free Flink?

[Aljoscha Krettek]

You mean putting the Flink-native S3 filesystem in the user jar or Hadoop in 
the user jar. The former wouldn't work, I think, because the FileSystems are 
being initialised before the user-jar is loaded. The latter might work but only 
if you don't have Hadoop in the classpath, i.e. not on YARN and only on a 
Hadoop-free cluster. Maybe...

> On 23. Feb 2018, at 13:32, Jamie Grier  wrote:
> 
> Thanks, Aljoscha :)
> 
> So is it possible to continue to use the new "native' fllesystems along
> with the BucketingSink by including the Hadoop dependencies only in the
> user's uber jar? Or is that asking for trouble?  Has anyone tried that
> successfully?
> 
> -Jamie
> 
> 
> On Fri, Feb 23, 2018 at 12:39 AM, Aljoscha Krettek 
> wrote:
> 
>> Hi,
>> 
>> I'm afraid not, since the BucketingSink uses the Hadoop FileSystem
>> directly and not the Flink FileSystem abstraction. The flink-s3-fs-*
>> modules only provide Flink FileSystems.
>> 
>> One of the goals for 1.6 is to provide a BucketingSink that uses the Flink
>> FileSystem and also works well with eventually consistent file systems.
>> 
>> --
>> Aljoscha
>> 
>>> On 23. Feb 2018, at 06:31, Jamie Grier  wrote:
>>> 
>>> Is the `flink-connector-filesystem` connector supposed to work with the
>>> latest hadoop-free Flink releases, say along with the
>> `flink-s3-fs-presto`
>>> filesystem implementation?
>>> 
>>> -Jamie
>> 
>> 

[jira] [Created] (FLINK-8769) Quickstart job submission logs contain several exceptions

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-8769:
---

 Summary: Quickstart job submission logs contain several exceptions
 Key: FLINK-8769
 URL: https://issues.apache.org/jira/browse/FLINK-8769
 Project: Flink
  Issue Type: Bug
  Components: Local Runtime
Affects Versions: 1.5.0
Reporter: Chesnay Schepler
 Fix For: 1.5.0


While checking out [the PR for 
FLINK-8761|https://github.com/apache/flink/pull/5569] and running a job in the 
IDE several exceptions are being logged. The job still runs properly though.
{code:java}
...

14:19:52,546 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint   
 - Failed to load web based job submission extension.
org.apache.flink.util.FlinkException: Could not load web submission extension.
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:252)
at 
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.initializeHandlers(DispatcherRestEndpoint.java:111)
at 
org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:124)
at 
org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:320)
at 
org.apache.flink.client.LocalExecutor.createJobExecutorService(LocalExecutor.java:144)
at org.apache.flink.client.LocalExecutor.start(LocalExecutor.java:118)
at 
org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:212)
at 
org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
at 
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:815)
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
at iqst.BatchJob.main(BatchJob.java:39)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.flink.runtime.webmonitor.WebMonitorUtils.loadWebSubmissionExtension(WebMonitorUtils.java:243)
... 11 more
Caused by: org.apache.flink.util.ConfigurationException: Config parameter 'Key: 
'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing 
(hostname/address of JobManager to connect to).
at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getJobManagerAddress(HighAvailabilityServicesUtils.java:137)
at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:79)
at 
org.apache.flink.client.program.ClusterClient.(ClusterClient.java:148)
at 
org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:144)
at 
org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:135)
at 
org.apache.flink.runtime.webmonitor.WebSubmissionExtension.(WebSubmissionExtension.java:61)
... 16 more
14:19:53,140 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint   
 - Rest endpoint listening at 127.0.0.1:64908

...

14:19:56,546 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor   
 - Close ResourceManager connection b8a2cff59ba07813067a64ebaf7d7889.
org.apache.flink.util.FlinkException: New ResourceManager leader found under: 
null(null)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.notifyOfNewResourceManagerLeader(TaskExecutor.java:853)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor.access$900(TaskExecutor.java:127)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutor$ResourceManagerLeaderListener.lambda$notifyLeaderAddress$0(TaskExecutor.java:1359)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:292)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:147)
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$onReceive$0(AkkaRpcActor.java:129)
at 
akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at 

[jira] [Created] (FLINK-8768) Change {{NettyMessageDecoder}} to inherit from {{LengthFieldBasedFrameDecoder}}

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8768:
--

 Summary: Change {{NettyMessageDecoder}} to inherit from 
{{LengthFieldBasedFrameDecoder}}
 Key: FLINK-8768
 URL: https://issues.apache.org/jira/browse/FLINK-8768
 Project: Flink
  Issue Type: Improvement
  Components: Network
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.5.0


Let {{NettyMessageDecoder}} inherit from {{LengthFieldBasedFrameDecoder}} 
instead of being an additional step in the pipeline. This does not only remove 
overhead in the pipeline itself but also allows use to override the 
{{#extractFrame()}} method to restore the old Netty 4.0.27 behaviour for 
non-credit based code paths which had a bug with Netty >= 4.0.28 there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Does `flink-connector-filesystem` work with Hadoop-Free Flink?

[Jamie Grier]

Thanks, Aljoscha :)

So is it possible to continue to use the new "native' fllesystems along
with the BucketingSink by including the Hadoop dependencies only in the
user's uber jar? Or is that asking for trouble?  Has anyone tried that
successfully?

-Jamie


On Fri, Feb 23, 2018 at 12:39 AM, Aljoscha Krettek 
wrote:

> Hi,
>
> I'm afraid not, since the BucketingSink uses the Hadoop FileSystem
> directly and not the Flink FileSystem abstraction. The flink-s3-fs-*
> modules only provide Flink FileSystems.
>
> One of the goals for 1.6 is to provide a BucketingSink that uses the Flink
> FileSystem and also works well with eventually consistent file systems.
>
> --
> Aljoscha
>
> > On 23. Feb 2018, at 06:31, Jamie Grier  wrote:
> >
> > Is the `flink-connector-filesystem` connector supposed to work with the
> > latest hadoop-free Flink releases, say along with the
> `flink-s3-fs-presto`
> > filesystem implementation?
> >
> > -Jamie
>
>

Re: [DISCUSS] Release Flink 1.4.2

[Stephan Ewen]

How about releasing 1.4.2 now, meaning immediately. This can be very
lightweight.

FLINK-8451 looks like it should have more thorough testing and should go
into 1.4.3.
I think there is no harm in more frequent bugfix releases.


On Fri, Feb 23, 2018 at 9:56 AM, Timo Walther  wrote:

> I also almost have a fix ready for FLINK-8451. I think it should also go
> into 1.4.2.
>
> Regards,
> Timo
>
>
> Am 2/22/18 um 11:29 AM schrieb Aljoscha Krettek:
>
> They reason they didn't catch this is that the bug only occurs if users
>> use a custom timestamp/watermark assigner. But yes, we should be able to
>> extend the end-to-end tests to catch this.
>>
>> On 22. Feb 2018, at 11:05, Till Rohrmann  wrote:
>>>
>>> If the Kafka connectors are unusable with 1.4.1, then I would be in favor
>>> of releasing 1.4.2 asap.
>>>
>>> I'm wondering why the end-to-end Kafka tests did not catch this problem.
>>> Maybe we could adapt them such that they guard against it in the future.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Thu, Feb 22, 2018 at 9:46 AM, Tzu-Li (Gordon) Tai <
>>> tzuli...@apache.org>
>>> wrote:
>>>
>>> Hi all,

 Unfortunately, we've discovered a bug in 1.4.1, which suggests that we
 should almost immediately release another bugfix release:
 https://issues.apache.org/jira/browse/FLINK-8741.

 Since this issue was introduced only in 1.4.1, it might make sense to
 release 1.4.2 with only the fix for FLINK-8741 included. What do you
 think?

 Best,
 Gordon

 
 不含病毒。www.avast.com
 
 <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>


>

[jira] [Created] (FLINK-8767) Set the maven.compiler.source and .target properties for Java Quickstart

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8767:
---

 Summary: Set the maven.compiler.source and .target properties for 
Java Quickstart
 Key: FLINK-8767
 URL: https://issues.apache.org/jira/browse/FLINK-8767
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


Setting these properties helps properly pinning the Java version in IntelliJ.
Without these properties, Java version keeps switching back to 1.5 in some 
setups.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8765) Simplify quickstart properties

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8765:
---

 Summary: Simplify quickstart properties
 Key: FLINK-8765
 URL: https://issues.apache.org/jira/browse/FLINK-8765
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


This does not pull out the slf4j and log4j version into properties any more, 
making the quickstarts a bit simpler.

Given that both versions are used only once, and only for the feature to have 
convenience logging in the IDE, the versions might as well be defined directly 
in the dependencies.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8766) Pin scala runtime version for Java Quickstart

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8766:
---

 Summary: Pin scala runtime version for Java Quickstart
 Key: FLINK-8766
 URL: https://issues.apache.org/jira/browse/FLINK-8766
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


 Followup to FLINK-7414, which pinned the scala version for the Scala Quickstart



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8764) Make quickstarts work out of the box for IDE and JAR packaging

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8764:
---

 Summary: Make quickstarts work out of the box for IDE and JAR 
packaging
 Key: FLINK-8764
 URL: https://issues.apache.org/jira/browse/FLINK-8764
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


We can make the quickstarts work for IntelliJ, Eclipse, and Maven Jar packaging 
out of the box, without the need to pass a profile name during jar packaging 
via the following trick:

  - All Flink and Scala dependencies are properly set to provided
  - That way, Maven JAR packaging behaves correctly by default
  - Eclipse adds 'provided' dependencies to the classpath when running 
programs, so works out of the box
  - There is a profile that automatically activates in IntelliJ that adds 
the necessary
dependencies in 'compile' scope to make it run out of the box.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8762) Remove unnecessary examples and make "StreamingJob" the default

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8762:
---

 Summary: Remove unnecessary examples and make "StreamingJob" the 
default
 Key: FLINK-8762
 URL: https://issues.apache.org/jira/browse/FLINK-8762
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


The packaged WordCount example jobs have been reported to not be terribly 
helpful and simply create noise in the initial project setup.

In addition, setting the main class by default to {{StreamingJob}} creates a 
better out of the box experience for the majority of the users. We prominently 
document how to adjust this to use {{BatchJob}} as the main class.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8763) Remove obsolete Dummy.java classes from quickstart projects.

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8763:
---

 Summary: Remove obsolete Dummy.java classes from quickstart 
projects.
 Key: FLINK-8763
 URL: https://issues.apache.org/jira/browse/FLINK-8763
 Project: Flink
  Issue Type: Sub-task
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


These classes seem no longer necessary, the project JavaDocs build properly 
without those classes being present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8761) Various improvements to the Quickstarts

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-8761:
---

 Summary: Various improvements to the Quickstarts
 Key: FLINK-8761
 URL: https://issues.apache.org/jira/browse/FLINK-8761
 Project: Flink
  Issue Type: Improvement
  Components: Quickstarts
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


Various improvements to the Quickstarts to give a smoother out of the box 
experience.

Broken down into the subtasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8760) Correctly set `moreAvailable` flag in SingleInputGate and UnionInputGate

[Piotr Nowojski (JIRA)]

Piotr Nowojski created FLINK-8760:
-

 Summary: Correctly set `moreAvailable` flag in SingleInputGate and 
UnionInputGate
 Key: FLINK-8760
 URL: https://issues.apache.org/jira/browse/FLINK-8760
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8759) Bump Netty to 4.0.56

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8759:
--

 Summary: Bump Netty to 4.0.56
 Key: FLINK-8759
 URL: https://issues.apache.org/jira/browse/FLINK-8759
 Project: Flink
  Issue Type: Improvement
  Components: Network
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.5.0


For a bug in Netty's shutdown sequence and overall improvements in Netty, I'd 
like to bump the version (and stay within the 4.0 series for now). The problem 
we faced in the past should not be relevant for credit-based flow control 
anymore and can be worked around (for the fallback code path) by restoring 
{{LengthFieldBasedFrameDecoder}}'s old behaviour of copying contents to new 
buffers instead of slicing the existing one.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8758) Expose method for non-blocking job submission on ClusterClient

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-8758:
---

 Summary: Expose method for non-blocking job submission on 
ClusterClient
 Key: FLINK-8758
 URL: https://issues.apache.org/jira/browse/FLINK-8758
 Project: Flink
  Issue Type: Improvement
  Components: Client, Tests
Affects Versions: 1.5.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 1.5.0


Some tests that we need to port as part of FLINK-8700 need a way of submitting 
jobs to the testing cluster in a non-blocking fashion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8757) Add MiniClusterResource.getClusterClient()

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-8757:
---

 Summary: Add MiniClusterResource.getClusterClient()
 Key: FLINK-8757
 URL: https://issues.apache.org/jira/browse/FLINK-8757
 Project: Flink
  Issue Type: Improvement
  Components: Client, Tests
Affects Versions: 1.5.0
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 1.5.0


Some of the tests that we need to port as part of FLINK-8700 need a way to 
submit jobs asynchronously to the testing cluster. For this, we need to be able 
to retrieve a {{ClusterClient}} and expose a method for asynchronous job 
submission.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8756) Support ClusterClient.getAccumulators() in RestClusterClient

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-8756:
---

 Summary: Support ClusterClient.getAccumulators() in 
RestClusterClient
 Key: FLINK-8756
 URL: https://issues.apache.org/jira/browse/FLINK-8756
 Project: Flink
  Issue Type: Bug
  Components: Client
Affects Versions: 1.5.0
Reporter: Aljoscha Krettek
 Fix For: 1.5.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: apache beam in spring cloud dataflow

[Fabian Hueske]

Hi,

you wrote to the Apache Flink development mailing list.
I think your question should go to the Apache Beam user mailing list:
u...@beam.apache.org

Best, Fabian

2018-02-22 14:35 GMT+01:00 shankara :

> I am new to apache beam and spring cloud dataflow. I am trying to integrate
> apache beam in spring cloud dataflow. How to get spring-kafka message as a
> source in beam pipeline ?. How to add spring-kafka as a sink in beam
> pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
> can I integrate ?
>
> example wordcount PipelineOptions options = PipelineOptionsFactory.create(
> );
>
> Pipeline p = Pipeline.create(options);
>
> p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*"))
> >
> instead of TextIO.read().from want to trigger from message channel INPUT in
> spring cloud dataflow
>  .apply(FlatMapElements
>  .into(TypeDescriptors.strings())
>  .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"
>  .apply(Filter.by((String word) -> !word.isEmpty()))
>  .apply(Count.perElement())
>  .apply(MapElements
>  .into(TypeDescriptors.strings())
>  .via((KV wordCount) -> wordCount.getKey() + ": " +
> wordCount.getValue()))
> .apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
> > send the result to message channel OUTPUT
>
> p.run().waitUntilFinish();
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

[jira] [Created] (FLINK-8755) SpilledSubpartitionView wrongly relys on the backlog for determining whether more data is available

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-8755:
--

 Summary: SpilledSubpartitionView wrongly relys on the backlog for 
determining whether more data is available
 Key: FLINK-8755
 URL: https://issues.apache.org/jira/browse/FLINK-8755
 Project: Flink
  Issue Type: Sub-task
  Components: Network
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.5.0


{code}
public BufferAndBacklog getNextBuffer() throws IOException, 
InterruptedException {
//...
int newBacklog = parent.decreaseBuffersInBacklog(current);
return new BufferAndBacklog(current, newBacklog > 0, newBacklog, 
nextBufferIsEvent);
{code}
relies on the backlog to signal further data availability. However, if there 
are only events left in the buffer queue, their buffers are not included in the 
backlog counting and therefore, {{isMoreAvailable}} will be wrongly {{false}} 
here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

apache beam in spring cloud dataflow

[shankara]

I am new to apache beam and spring cloud dataflow. I am trying to integrate
apache beam in spring cloud dataflow. How to get spring-kafka message as a
source in beam pipeline ?. How to add spring-kafka as a sink in beam
pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
can I integrate ?

example wordcount PipelineOptions options = PipelineOptionsFactory.create();

Pipeline p = Pipeline.create(options);

p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) >
instead of TextIO.read().from want to trigger from message channel INPUT in
spring cloud dataflow
 .apply(FlatMapElements
 .into(TypeDescriptors.strings())
 .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"
 .apply(Filter.by((String word) -> !word.isEmpty()))
 .apply(Count.perElement())
 .apply(MapElements
 .into(TypeDescriptors.strings())
 .via((KV wordCount) -> wordCount.getKey() + ": " +
wordCount.getValue()))
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
> send the result to message channel OUTPUT

p.run().waitUntilFinish();



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re: [DISCUSS] Release Flink 1.4.2

[Timo Walther]

I also almost have a fix ready for FLINK-8451. I think it should also go 
into 1.4.2.


Regards,
Timo


Am 2/22/18 um 11:29 AM schrieb Aljoscha Krettek:

They reason they didn't catch this is that the bug only occurs if users use a 
custom timestamp/watermark assigner. But yes, we should be able to extend the 
end-to-end tests to catch this.


On 22. Feb 2018, at 11:05, Till Rohrmann  wrote:

If the Kafka connectors are unusable with 1.4.1, then I would be in favor
of releasing 1.4.2 asap.

I'm wondering why the end-to-end Kafka tests did not catch this problem.
Maybe we could adapt them such that they guard against it in the future.

Cheers,
Till

On Thu, Feb 22, 2018 at 9:46 AM, Tzu-Li (Gordon) Tai 
wrote:


Hi all,

Unfortunately, we've discovered a bug in 1.4.1, which suggests that we
should almost immediately release another bugfix release:
https://issues.apache.org/jira/browse/FLINK-8741.

Since this issue was introduced only in 1.4.1, it might make sense to
release 1.4.2 with only the fix for FLINK-8741 included. What do you think?

Best,
Gordon


不含病毒。www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Does `flink-connector-filesystem` work with Hadoop-Free Flink?

[Aljoscha Krettek]

Hi,

I'm afraid not, since the BucketingSink uses the Hadoop FileSystem directly and 
not the Flink FileSystem abstraction. The flink-s3-fs-* modules only provide 
Flink FileSystems.

One of the goals for 1.6 is to provide a BucketingSink that uses the Flink 
FileSystem and also works well with eventually consistent file systems.

--
Aljoscha

> On 23. Feb 2018, at 06:31, Jamie Grier  wrote:
> 
> Is the `flink-connector-filesystem` connector supposed to work with the
> latest hadoop-free Flink releases, say along with the `flink-s3-fs-presto`
> filesystem implementation?
> 
> -Jamie