Re: StreamCorruptedException

2017-09-26 Thread Sridhar Chellappa
.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:536) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) On Wed, Sep 27, 2017 at 8:34 AM, Sridhar Chellappa wrote:

Re: StreamCorruptedException

2017-09-26 Thread Sridhar Chellappa
thread as he may have a better idea > what is happening here. > > Cheers, > Gordon > > On 22 September 2017 at 4:09:07 PM, Sridhar Chellappa ( > flinken...@gmail.com) wrote: > > Thanks for the reply. Well, tracing back to the root cause, I see the > following:

Re: Cannot deploy Flink on YARN

2017-09-26 Thread Sridhar Chellappa
t failed? > > On Sep 11, 2017 2:17 AM, "Sridhar Chellappa" wrote: > > I am trying to start Flink(Version 1.3.0) on YARN (Hadoop 2.8.1) by > issuing the following command: > > ~/flink-1.3.0/bin/yarn-session.sh -s 4 -n 10 -jm 4096 -tm 4096-d > > I am seeing a

Re: StreamCorruptedException

2017-09-22 Thread Sridhar Chellappa
Thanks for the reply. Well, tracing back to the root cause, I see the following: 1. At the Job manager, the Checkpoint times are getting worse : Jobmanager : Checkpoint times are getting worse progressively. 2017-09-16 05:05:50,813 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator

StreamCorruptedException

2017-09-14 Thread Sridhar Chellappa
I am running Flink 1.3.0 against Kafka 0.10. I managed to bring the flink cluster up and have been running my flink CEP job for more than 3 hours when I see the following exception : The messages consumed from Kafka are protobuf messages and I use a protobuf serializer. i have no clue as to where

Cannot deploy Flink on YARN

2017-09-11 Thread Sridhar Chellappa
I am trying to start Flink(Version 1.3.0) on YARN (Hadoop 2.8.1) by issuing the following command: ~/flink-1.3.0/bin/yarn-session.sh -s 4 -n 10 -jm 4096 -tm 4096-d I am seeing a flurry of these Errors: 2017-09-11 08:17:11,410 INFO org.apache.flink.yarn.YarnClusterDescriptor - D

Re: Exception when using keyby operator

2017-09-11 Thread Sridhar Chellappa
that your setters have a return type. A POJO setter usually > should have a void return type. But I agree that this should be mentioned > in the documentation. > > Regards, > Timo > > > Am 07.09.17 um 05:20 schrieb Sridhar Chellappa: > > I am trying to use the

Exception when using keyby operator

2017-09-06 Thread Sridhar Chellappa
I am trying to use the KeyBy operator as follows : Pattern myEventsCEPPattern = Pattern.begin("FirstEvent") .subtype(MyEvent.class) .next("SecondEvent") .subtype(MyEvent.class)

Re: Null Pointer Exception on Trying to read a message from Kafka

2017-08-28 Thread Sridhar Chellappa
astRecord.getValue())); > > Either serializer or castRecord was null. > > I wonder if this has been fixed in 1.3.2 release. > > On Mon, Aug 28, 2017 at 7:24 PM, Sridhar Chellappa > wrote: > >> Kafka Version is 0.10.0 >> >> On Tue, Aug 29, 2017 at 6:43 AM, Sridhar Ch

Re: Null Pointer Exception on Trying to read a message from Kafka

2017-08-28 Thread Sridhar Chellappa
Kafka Version is 0.10.0 On Tue, Aug 29, 2017 at 6:43 AM, Sridhar Chellappa wrote: > 1.3.0 > > On Mon, Aug 28, 2017 at 10:09 PM, Ted Yu wrote: > >> Which Flink version are you using (so that line numbers can be matched >> with source code) ? >> >> On M

Re: Null Pointer Exception on Trying to read a message from Kafka

2017-08-28 Thread Sridhar Chellappa
1.3.0 On Mon, Aug 28, 2017 at 10:09 PM, Ted Yu wrote: > Which Flink version are you using (so that line numbers can be matched > with source code) ? > > On Mon, Aug 28, 2017 at 9:16 AM, Sridhar Chellappa > wrote: > >> DataStream MyKafkaMessageDa

Re: Null Pointer Exception on Trying to read a message from Kafka

2017-08-28 Thread Sridhar Chellappa
? > > Can you show the snippet of code where you create the DataStream ? > > Cheers > > On Mon, Aug 28, 2017 at 7:38 AM, Sridhar Chellappa > wrote: > >> Folks, >> >> I have a KafkaConsumer that I am trying to read messages from. When I try >> t

Null Pointer Exception on Trying to read a message from Kafka

2017-08-28 Thread Sridhar Chellappa
Folks, I have a KafkaConsumer that I am trying to read messages from. When I try to create a DataStream from the KafkConsumer (env.addSource()) I get the following exception : Any idea on how can this happen? java.lang.NullPointerException at org.apache.flink.streaming.runtime.tasks.Ope

Aggregating metrics from different boxes

2017-08-24 Thread Sridhar Chellappa
Folks, I am using RichMapFunction to generate codahale like metrics from different taskmanagers spread across an N-Node cluster. When I see the visualizations (Grafana on InfluxDB), I see all of the metrics as separate streams ($host.$taskmanager.$uuid.$metricname). I thought I can aggregate thes

Null pointer exception while trying to serialize a protobuf message

2017-08-04 Thread Sridhar Chellappa
Folks, I wrote a custom Data source to test me CEP logic. The custom data source looks like : public class CustomerDataSource extends RichParallelSourceFunction { private boolean running = true; private final Random random; public CustomerDataSource() { this.random = new Rand

Flink Vs Google Cloud Dataflow?

2017-07-30 Thread Sridhar Chellappa
Did anyone study Cloud DataFlow as an alternative to Flink? If yes, can someone summarize their analysis of Dataflow as against Flink?

Re: data loss after implementing checkpoint

2017-07-18 Thread Sridhar Chellappa
>>>> The CEP library runs Correction; the CEP implemented using the CEP library runs . On Wed, Jul 19, 2017 at 10:08 AM, Sridhar Chellappa wrote: > A follow up question on this. I have a Complex Event processor implemented > using the CEP library (1.3.0). The CEP libr

Re: data loss after implementing checkpoint

2017-07-18 Thread Sridhar Chellappa
A follow up question on this. I have a Complex Event processor implemented using the CEP library (1.3.0). The CEP library runs a variety of rules that are configured (enable/disable rule) VIA REST APIs. Now, if my application crashes and recovers (or is cancelled and restarted), will my configurat

Getting Errors when using keyby()

2017-07-12 Thread Sridhar Chellappa
I have a DataStream on which I am applying a CEP pattern and grouping the results using keyby(). The DataStream Object is a pojo : public class DataStreamObject { private String field1; private String field2; public DataStreamObject(String field1, String field2) { this.field1

Should customized Complex Events be Serializable?

2017-07-11 Thread Sridhar Chellappa
Folks, I am using the CEP library to create ComplexEvents. My question is, should the ComplexEvents be serializable?

Re: Not able to read Kafka Messages with FlinkKafkaConsumer010

2017-07-11 Thread Sridhar Chellappa
tudienzentrum Berlin > Theodor-Heuss-Platz 5 > 14052 Berlin > *Ph: +49 176 6587 3343 <%2B49%20176%206587%203343>* > *Mail to*: *mmzi...@gmail.com * > > On Tue, Jul 11, 2017 at 9:12 AM, Sridhar Chellappa > wrote: > >> I am pretty sure I am doing somet

Not able to read Kafka Messages with FlinkKafkaConsumer010

2017-07-11 Thread Sridhar Chellappa
I am pretty sure I am doing something wrong here. Just that I do not understand why? I wrote a small program that reads messages from Kafka and prints it out. public class Main { private static final int CHECKPOINT_INTERVAL = 10; private static Properties getpropsFromEnv() {

Re: Integrating Flink CEP with a Rules Engine

2017-06-23 Thread Sridhar Chellappa
t; It would be worth describing a bit more what these tools are and what >> are >> > your needs. >> > In addition, and to see what the CEP library already offers here you can >> > find the documentation: >> > >> > https://ci.apache.org/projects/flink

Integrating Flink CEP with a Rules Engine

2017-06-23 Thread Sridhar Chellappa
Folks, I am new to Flink. One of the reasons why I am interested in Flink is because of its CEP library. Our CEP logic comprises of a set of complex business rules which will have to be managed (Create, Update, Delete) by a bunch of business analysts. Is there a way I can integrate other third p