Flink CEP State Change Pattern

2017-10-14 Thread Philip Limbeck
Hi! I am quite new to Flink CEP and try to define a state change pattern with it. This means that only discrete changes in the event stream should be detected i.e. a a b b - triggers a single change from a to b Considering b the "bad" state, I would like to additionally recognize the state chang

RocksDB usage for broad slow data

2017-10-14 Thread Derek VerLee
We have a data which is broad and slow; hundreds of thousands of keys, a small number will get an event every few seconds, some get an event every few days, and the vast majority will get an event in a few times an hour.  Let's say then that keeping this data

Doubts about parallelism

2017-10-14 Thread AndreaKinn
Hi, I read the doc about parallelism, parallel execution and job scheduling but however I have some doubts about parallelism. 1. In my first try I unset parallelism in my code and commented parallelism.default key in link-conf file. In this case I supposed the parallelism was set by Flink automat

Re: Noisy org.apache.flink.configuration.GlobalConfiguration

2017-10-14 Thread Stephan Ewen
Hi! The problem should be fixed in the latest master. https://github.com/apache/flink/commit/536675b03a5050fda9c3e1fd403818cb50dcc6ff Release should come soon... Best, Stephan On Wed, Oct 4, 2017 at 12:50 PM, Stephan Ewen wrote: > Will be opening a PR for file system configuring which should

Re: Empty directories left over from checkpointing

2017-10-14 Thread Stephan Ewen
Some updates on this: Aside from reworking how the S3 directory handling is done, we also looked into supporting S3 different than we currently do. Currently support goes strictly through Hadoop's S3 file systems, which we need to change, because we want it to be possible to use Flink without Hado

Problems with window function

2017-10-14 Thread AndreaKinn
Hi all, I'm trying to implement a time ordering inside a stream using window function. Then my purposes is to order the element inside a tumbling window. This is my code (written following the doc): DataStream LCxAccStream = env .addSource(new FlinkKafkaConsumer010

Re: Problems with window function

2017-10-14 Thread Tony Wei
Hi Andrea, AFAIK, `keyBy` function you used will wrap all keys you selected into `Tuple`. You can use `Tuple.f0` to get your key, whose type will be `String`. If you want the KeyedStream has String Type for its key, you can use `KeySelector` in keyBy function. [1] Hope this will help you. Best Re

Re: Doubts about parallelism

2017-10-14 Thread Tony Wei
Hi Andrea, For your first question, I think you are right, but the basis is set by the default value for `parallelism.default` in flink-conf.yaml. [1] For your second question, I guess you use `forward` function between "learn" and "select" methods. Am I right? That exception is an expected behav