Re: Streaming Exception

This is the exception before the job went into cancelled state. But when I looked into the task manager node, the flink process is still running. java.lang.Exception: TaskManager was lost/killed: 383f6af3299793ba73eeb7bdbab0ddc7 @ ip-xx.xx.xxx.xx.us-west-2.compute.internal (dataPort=37652)

Re: questions on custom state with flink window

Hi Sai, 3) If you want to make "Managed Keyed State" queryable, you have to set it as queryable through the API, e.g.: final ValueStateDescriptor query1State = new ValueStateDescriptor<>("stateName", Long.class); query1State.setQueryable("queryName");

Re: Any good ideas for online/offline detection of devices that send events?

pro tip for debugging watermarks: They are exposed via a metric in Flink 1.2. On Tue, Mar 7, 2017 at 1:37 PM, Bruno Aranda wrote: > Hi Gordon, > > Many thanks for your helpful ideas. We tried yesterday the CEP approach, > but could not figure it out. The ProcessFunction

Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers

There is currently no way in Flink to define such scheduling constraints. On Wed, Mar 8, 2017 at 5:00 PM, PedroMrChaves wrote: > Thanks for the response. > > I would like to assure that the map operator is not in the same task > manager > as the window/apply operator,

Re: Job completion or failure callback?

Hi Shannon, the web UI runs on the same JVM as the JobManager, so log outputs should go there. There is no way of running user code on the JobManager on job completion. We try to not allow users to execute code on the JobManager...bringing the JM down, will kill the entire cluster :) What you

Re: AWS exception serialization problem

Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally, not using EMR?) I think once we can reproduce the issue its easy to fix. On Thu, Mar 9, 2017 at 1:24 AM, Bruno Aranda wrote: > Hi Stephan, we are running Flink 1.2.0 on Yarn

Re: Issues with Event Time and Kafka

Hi Ethan, how late elements (elements with event time after the watermark) are handled depends on the operator. Flink's window operators will trigger a single event window when they fall into the "allowed lateness" timeframe. Otherwise, they are dropped. On Thu, Mar 9, 2017 at 5:30 PM,

Re: Flink support for DeepLearning4j or other deep learning library

Hi, I'm not aware of any community efforts into that direction. This is the only thing google brought up: https://www.slideshare.net/FlinkForward/suneel-marthi-deep-learning-with-apache-flink-and-dl4j But in general, I don't think that it should be terribly hard to get started with something,

Re: Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

Hi Dave, Let me answer your questions: 1. The RocksDB state backend always stores the data on local disks for speed. The back up is done to HDFS or any other distributed file system. The local data directory is configured automatically by YARN. 2. You need to manually configure zookeeper in the

Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

Hi, Forgive me if parts of this question have been answered before but I'd like help in resolving some bits of confusion from the documentation and the fact that I haven't been able to find a good example anywhere for an enterprise-style setup. If anyone has a sample HA / Yarn / ZK / RocksDB

Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

Hi, Forgive me if parts of this question have been answered before but I'd like help in resolving some bits of confusion from the documentation and the fact that I haven't been able to find a good example anywhere for an enterprise-style setup. If anyone has a sample HA / Yarn / ZK / RocksDB

Re: ProcessFunction example

Hi, i've created a PR to fix scala and java examples and the error suggested by Philippe. Hope it will be helpful!! Mauro Il 09/03/2017 10:30, Kostas Kloudas ha scritto: Hi Philippe, You are right! Thanks for reporting it! We will fix it asap. Kostas On Mar 9, 2017, at 8:38 AM, Philippe

Flink support for DeepLearning4j or other deep learning library

Hi, I am trying to use Deep AutoEncoders for anomaly detection in my data stream using Flink. Any pointers/examples on the same. Regards, Adarsh

Re: Flink Standalone Service

Hi Daniel, if you install the flink RPMs (or DEBs) from Apache Bigtop, they should come with init.d or service files. On Thu, Mar 9, 2017 at 6:10 PM, Daniel Skates wrote: > Hi all, > > Is there a init.d or similar service script for Flink on Redhat (or > Centos) 7? Mostly

Re: Flink - Writing Test Case for the Datastream

Hi Mahesh, In the kafka tests, were using a pattern of killing a job by throwing a "SuccessException" after a certain number of messages have passed. Just check the Kafka tests to see how its done :) On Thu, Mar 9, 2017 at 10:09 PM, MAHESH KUMAR wrote: > Hi Team,

Re: Request jira permission

Thank you very much Robert!!! Il 10/03/2017 14:36, Robert Metzger ha scritto: You'll have permissions in the next two minutes :) On Fri, Mar 10, 2017 at 2:19 PM, Mauro Cortellazzi > wrote: Hello Comunity, i

Re: questions on custom state with flink window

Hi Sai, 1) I think its okay to keep state in a RichWindowFunction. 2) I think it stays forever, yes 3) I'm including Nico, he can probably help you with the queryable state question. 4) I guess that's a queryable state question too. On Fri, Mar 10, 2017 at 1:04 AM, saiprasad mishra

Re: Streaming Exception

Hi, this error is only logged at WARN level. As Kaibo already said, its not a critical issue. Can you send some more messages from your log. Usually the Jobmanager logs why a taskmanager has failed. And the last few log messages of the failed TM itself are also often helpful. On Fri, Mar 10,

Re: Request jira permission

You'll have permissions in the next two minutes :) On Fri, Mar 10, 2017 at 2:19 PM, Mauro Cortellazzi < mauro.cortella...@radicalbit.io> wrote: > Hello Comunity, > > i would help and contribute into Flink and i'm already registered into > jira. > > I've created an issue [1] about documentation

Request jira permission

Hello Comunity, i would help and contribute into Flink and i'm already registered into jira. I've created an issue [1] about documentation and i would work on it, but i haven't permission to assign the task to myself, can somebody give me the right permission please? Thank you very much!!

Re: Integrate Flink with S3 on EMR cluster

Hi Vinay, using the HADOOP_CLASSPATH variable on the client machine is the recommended way to solve this problem. I'll update the documentation accordingly. On Wed, Mar 8, 2017 at 10:26 AM, vinay patil wrote: > Hi , > > @Shannon - I am not facing any issue while

Re: Performance tuning

I think that's independent of the serializer registration. What's important is registering the types at the execution environment. On Fri, Feb 24, 2017 at 7:06 PM, Dmitry Golubets wrote: > Hi Robert, > > The bottleneck operator is working with a state (many hash maps

Re: Streaming Exception

I think this is not the root cause of job failure, this task is caused by other tasks failing. You can check the log of the first failed task. 2017-03-10 12:25 GMT+08:00 Govindarajan Srinivasaraghavan < govindragh...@gmail.com>: > Hi All, > > I see the below error after running my streaming job