Questions about the FlinkCEP

Hi I have tried to learn FlinkCEP [1], but i have yet not found the clear answers for questions: 1) Whether the pattern of CEP is meant only for one data stream at the same time ? 2) If i have many different parallel data streams (or sources), should i combine them into one data stream

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

Thanks for the suggestion .. it works now. Also it works cleanly with 1.4.0 (without the hacks) .. On Wed, Feb 28, 2018 at 7:14 PM, Chesnay Schepler wrote: > Well we just ended up in ClassLoader hell... > > There are 2 config options that could help: > >- add

Slow Flink program

Hi, I'm trying to run a simple benchmark on Flink streaming reduce. It seems it is very slow. Could you let me know if I'm doing something wrong. Here is the program. I'm running this on 32 nodes with 20 tasks in each node. So the parallelism is at 640. public class StreamingReduce { int

Re: Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

nevermind :) PEBKAC error. using the CLI and specifying the switches in the wrong location. Hopefully this will not be followed by an 'actually... there's still this other issue' post On 2018/02/28 15:38:29, dyana.r...@salecycle.com wrote: > 2018-02-28

Re: logging question

I'm a bit curious on how you hand your log4j into the docker image for consumption. On SO you are referring to bin/flink-console.sh but executing Flink in docker is a bit different. Maybe I'm wrong, but looking at the sources of the docker image [1], it will not forward any additional parameters

Configuring hdfs with the MesosApplicationMasterRunner

Hi, I get that this might not be a priority given FLIP6, but I set up a mesos configuration for clustering with Flink today, and one of the painful points was the hadoop configuration. It's impossible to pass the hadoop conf folder artefact (containing files like core-site.xml etc.) to the task

Re: Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

should note this is under 1.4.1 On 2018/02/28 15:38:29, dyana.r...@salecycle.com wrote: > 2018-02-28 15:16:36,675 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - class > org.apache.flink.streaming.connectors.kinesis.model.SequenceNumber does

Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

2018-02-28 15:16:36,675 INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.connectors.kinesis.model.SequenceNumber does not contain a setter for field sequenceNumber 2018-02-28 15:16:36,675 INFO

Re: Which test cluster to use for checkpointing tests?

I was a bit confused about when you said that the "source is done" which is when I realized you must be using the batch API for which checkpointing is not available / needed. Let me quote from [1] which imho has not changed: DataSet: Fault tolerance for the DataSet API works by restarting the

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

Well we just ended up in ClassLoader hell... There are 2 config options that could help: * add "org.apache.kafka." to "classloader.parent-first-patterns" o make sure you include the default as well

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

Thanks for the suggestion. I copied the application jar to lib. The error doesn't come but I get another error related to Kafka .. org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:433) at

Re: Machine Learning: Flink and MOA

Hello Theodore, Glad to hear that there is an interest in plugging MOA with Flink! Which part/type of classifiers of MOA would you want to plug with Flink? Let me know if you want to discuss in more details. I guess some windowing function of MOA would be better implemented as Flink Windows

RE: Reading csv-files

Because I have no time to learn all features of Flink and because there can be some issues in this my case, I am very interested about implementing external “logs replayer” or some batch to stream data converter. Do you have any ideas or suggestions how to build this kind of logs replayer ? Or

Re: ClassNotFoundException with Flink 1.4.1 and Kafka connector

Hello, this is probably caused by a known issue in 1.4.1: https://issues.apache.org/jira/browse/FLINK-8741 This bug is not present in 1.4.0, and it will fixed in 1.4.2 which should be released within the next days. As a temporary workaround you can copy your app-assembly-1.0.jar into the