Re: Checkpointing SIGSEGV

Flink’s version is hosted here: https://github.com/dataArtisans/frocksdb > Am 26.05.2017 um 19:59 schrieb Jason Brelloch : > > Thanks for looking into this Stefan. We are moving forward with a different > strategy for now. If I

Re: New "Powered by Flink" success case

Hi Luca, thanks for sharing this exciting use case! I added KEEDIO to Flink's Powered By list: https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink Thank you and best regards, Fabian 2017-05-25 10:35 GMT+02:00 Rosellini, Luca : > Hello everybody, > I am

Re: Tumbling window expects a time attribute for grouping in a stream environment

Hi Enrico, that's (for now) the right approach. I agree, that the KafkaTableSource should implement both DefinedXTimeAttribute interfaces. Best, Fabian 2017-05-25 3:20 GMT+02:00 enrico canzonieri : > I solved this implementing a new Kafka09TableSource in my application.

Re: Last event in event time window is not output

Hi, the problem might be that your source does not send a watermark this timestamp MAX_LONG after the last record has been sent. So your operators never compute the last window. Best, Fabian 2017-05-24 19:00 GMT+02:00 Sendoh : > Hi Flink users, > > We have a unit

Re: Checkpointing SIGSEGV

Thanks for looking into this Stefan. We are moving forward with a different strategy for now. If I want to take a look at this, where do I go to get the Flink version of RocksDB? On Fri, May 26, 2017 at 1:06 PM, Stefan Richter wrote: > I forgot to mention that

Flink - Iteration and Backpressure

Hi Team, I am trying to build an audit like system where I read messages from "n" Kafka queues, key by a unique key and then reduce them to a single message, if it has passed through all the "n" Kafka queues in a window time of "m" hours/days, the message has succeeded else it has expired. I can

Re: Checkpointing SIGSEGV

I played a bit around with your info and this looks now like a general problem in RocksDB to me. Or more specifically, between RocksDB and the JNI bridge. I could reproduce the issue with the following simple test code: File rocksDir = new File("/tmp/rocks"); final Options options = new

Re: Does RichFilterFunction work on multiple thread?

Your objects will be processed by a single thread. Kostas > On May 26, 2017, at 4:50 PM, luutuan wrote: > > Hi, when I have a set of objects goes through a RichFilterFunction, by > default, will the filter handle all objects in 1 single thread or will > divide the work to

Does RichFilterFunction work on multiple thread?

Hi, when I have a set of objects goes through a RichFilterFunction, by default, will the filter handle all objects in 1 single thread or will divide the work to multiple threads? Thank you. -- View this message in context:

Re: Checkpointing SIGSEGV

~2 GB was the total state in the backend. The total number of keys in the test is 10 with an approximately even distribution of state across keys, and parallelism of 1 so all keys are on the same taskmanager. We are using ListState and the number of elements per list would be about 50. On

Re: No Alerts with FinkCEP

Hi Biplob, For the 1.4 version, the input of the select function has changed to expect a list of matching events (Map map instead of Map map), as we have added quantifiers. Also the FIlterFunction has changed to SimpleCondition. The documentation is lagging a bit

Fwd: invalid type code: 00

I am forwarding Stefan’s reply: Hi, this problem can be caused by https://issues.apache.org/jira/browse/FLINK-6044. It is fixed in 1.2.1 and 1.3. Best, Stefan >> Am 26.05.2017 um 16:16 schrieb Kostas Kloudas : >> >> Hi, >> >> Could you provide some info on when

Fwd: invalid type code: 00

I am forwarding Stefan’s reply here: > Hi, > > this problem can be caused by > https://issues.apache.org/jira/browse/FLINK-6044 > . It is fixed in 1.2.1 and > 1.3. > > Best, > Stefan > >> Am 26.05.2017 um 16:16 schrieb Kostas Kloudas

Re: No Alerts with FinkCEP

Hello Kostas, Thanks for the suggestions. I checked and I am getting my events in the partitionedInput stream when i am printing it but still nothing on the alert side. I checked flink UI for backpressure and all seems to be normal (I am having at max 1000 events per second on the kafka topic so

Re: Checkpointing SIGSEGV

Hi, what means „our state“ in this context? The total state in the backend or the state under one key? If you use, e.g. list state, I could see that the state for one key can grow above 2GB, but once we retrieve the state back from RocksDB as Java arrays (in your stacktrace, when making a

Re: invalid type code: 00

Hi, Could you provide some info on when is this error happening? From what I see you are using the heap or fs state backend and you are failing to read the state back when restoring from a failure. The failure can be unrelated to this, but it could be useful if you could check the task manager

Re: Duplicated data when using Externalized Checkpoints in a Flink Highly Available cluster

Hi Amara, Could you elaborate a bit more detail about your job? How are you producing the 4200 events into Kafka? Is that a separate process than the consuming job? Do note that sending data to a Kafka topic is currently only at-least-once delivery, so if you’re sending the data to the Kafka

Re: Checkpointing SIGSEGV

Hi Jason, This error is unexpected. I don't think its caused by insufficient memory. I'm including Stefan into the conversation, he's the RocksDB expert :) On Thu, May 25, 2017 at 4:15 PM, Jason Brelloch wrote: > Hey guys, > > We are running into a JVM crash on

Re: No Alerts with FinkCEP

One additional comment, from your code it seems you are using Flink 1.2. It would be worth upgrading to 1.3. The updated CEP library includes a lot of new features and bugfixes. Cheers, Kostas > On May 26, 2017, at 3:33 PM, Kostas Kloudas > wrote: > > Hi Biplob,

Re: No Alerts with FinkCEP

Hi Biplob, From a first scan of the code I cannot find sth fishy. You are working on ProcessingTime, given that you do not provide any time characteristic specification, right? In this case, if you print your partitionedInput stream, do you see elements flowing as expected? If elements are

Re: Flink parallel tasks, slots and vcores

Can you give us more information about what your Flink job is doing and the distribution of the Kinesis data/keys? The distribution of work depends a lot on that. Example: If you are using a kafka source with a single partition (I think they are called shards in Kenisis) in the datastream api

No Alerts with FinkCEP

Hi, I just started exploring Flink CEP a day back and I thought I can use it to make a simple event processor. For that I looked into the CEP examples by Till and some other articles. Now I have 2 questions which i would like to ask: *Part 1:* I came up with the following piece of code, but

Re: invalid type code: 00

Which Flink version you are using? 1.2 What is your job doing (e.g. operators that you are using)? ProcessFunction to determine if event is late change event time to current & then window Which operator throws this exception? i will have to dig it further Which state-backend are you using?

Re: Are timers in ProcessFunction fault tolerant?

Yes, that is correct. Kostas > On May 26, 2017, at 11:05 AM, Moiz S Jinia wrote: > > Thanks Kostas. So even though the timer state is managed separately from the > key state (from runtimeContext) I can safely assume both the states to be > fault tolerant and maintain

Re: Are timers in ProcessFunction fault tolerant?

Thanks Kostas. So even though the timer state is managed separately from the key state (from runtimeContext) I can safely assume both the states to be fault tolerant and maintain association with the key of the stream? On Fri, May 26, 2017 at 1:51 PM, Kostas Kloudas

Re: invalid type code: 00

Hi! Can you give us some information about your job? Which Flink version you are using? What is your job doing (e.g. operators that you are using)? Which operator throws this exception? Which state-backend are you using? This exception means that you cannot retrieve your state because of

Re: Are timers in ProcessFunction fault tolerant?

Hi Moiz, state.clear() refers to the state that you have registered in your job, using the getState() from the runtimeContext. Timers are managed by Flink’s timer service and they are cleaned up by Flink itself when the job terminates. Kostas > On May 26, 2017, at 6:41 AM, Moiz S Jinia