Re: Are timers in ProcessFunction fault tolerant?

A follow on question. Since the registered timers are part of the managed key state, do the timers get cancelled when i call state.clear()? Moiz On Thu, May 25, 2017 at 10:20 PM, Moiz S Jinia wrote: > Awesome. Thanks. > > On Thu, May 25, 2017 at 10:13 PM, Eron Wright > wrote: > >> Yes, registe

invalid type code: 00

Sprodically i am seeing this error. Any idea? java.lang.IllegalStateException: Could not initialize keyed state backend. at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:286) at org.apache.flink.streaming.api.operators.A

Flink parallel tasks, slots and vcores

Hi Till/ flink-devs, I am trying to understand why adding slots in the task manager is having no impact in performance for the test pipeline. Here is my flink-conf.yaml jobmanager.rpc.address: localhost jobmanager.rpc.port: 6123 jobmanager.heap.mb: 1024 taskmanager.memory.preallocate: false taskma

Re: Collapsible job plan visualization

You should be able to move the separator between the plan view and the bottom panel already. On 25.05.2017 19:45, Flavio Pompermaier wrote: Hi to all, In our experience the Flink plan diagram is a nice feature but it is useless almost all the time and it has an annoying interaction with the m

Collapsible job plan visualization

Hi to all, In our experience the Flink plan diagram is a nice feature but it is useless almost all the time and it has an annoying interaction with the mouse wheelI suggest to make it a collapsible div. IMHO that would be an easy thing that would definitively improve the user experience ...what

Re: Flink and swapping question

Hi to all, I think we found the root cause of all the problems. Looking ad dmesg there was a "crazy" total-vm size associated to the OOM error, a LOT much bigger than the TaskManager's available memory. In our case, the TM had a max heap of 14 GB while the dmsg error was reporting a required amount

Re: Are timers in ProcessFunction fault tolerant?

Awesome. Thanks. On Thu, May 25, 2017 at 10:13 PM, Eron Wright wrote: > Yes, registered timers are stored in managed keyed state and should be > fault-tolerant. > > -Eron > > On Thu, May 25, 2017 at 9:28 AM, Moiz S Jinia > wrote: > >> With a checkpointed RocksDB based state backend, can I expec

Re: Are timers in ProcessFunction fault tolerant?

Hi Moiz! Adding a bit of more detail here: Yes, the timer will be restored on whatever new instance is responsible for that key. There is one “gotcha” to look out for, though: the firing time of timers are absolute; what this means is that if the checkpoints timer’s firing processing timestamp

Re: Are timers in ProcessFunction fault tolerant?

Yes, registered timers are stored in managed keyed state and should be fault-tolerant. -Eron On Thu, May 25, 2017 at 9:28 AM, Moiz S Jinia wrote: > With a checkpointed RocksDB based state backend, can I expect the > registered processing timers to be fault tolerant? (along with the managed > ke

Are timers in ProcessFunction fault tolerant?

With a checkpointed RocksDB based state backend, can I expect the registered processing timers to be fault tolerant? (along with the managed keyed state). Example - A task manager instance owns the key k1 (from a keyed stream) that has registered a processing timer with a timestamp thats a day ahe

Re: Kafka 0.10 jaas multiple clients

Gordon's suggestion seems like a good way to provide per-job credentials based on application-specific properties. In contrast, Flink's built-in JAAS features are aimed at making the Flink cluster's Kerberos credentials available to jobs. I want to reiterate that all jobs (for a given Flink clus

Re: How can I handle backpressure with event time.

Try setting the assigner on the Kafka consumer, rather than on the DataStream: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/ kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission I believe this will produce a per-partition assigner and forward only the

How can I handle backpressure with event time.

For example, I want to merge two kafka topics (named topicA and topicB) by the specific key with a max timeout. I use event time and class BoundedOutOfOrdernessTimestampExtractor to generate water mark. When some partitions of topicA be delayed by backpressure, and the delays exceeds my max timeout

New "Powered by Flink" success case

Hello everybody, I am posting this here following the guidelines I've found in the "Powered by Flink" page. At KEEDIO (http://www.keedio.com) we use Apache Flink CEP API in a log aggregation solution for Red Hat OpenStack that enables us to disco

Re: Flink and swapping question

I can confirm that after giving less memory to the Flink TM the job was able to run successfully. After almost 2 weeks of pain, we summarize here our experience with Fink in virtualized environments (such as VMWare ESXi): 1. Disable the virtualization "feature" that transfer a VM from a (heavy