Resource under-utilization when using RocksDb state backend

In tests comparing RocksDb to fs state backend we observe much lower throughput, around 10x slower. While the lowered throughput is expected, what's perplexing is that machine load is also very low with RocksDb, typically falling to < 25% CPU and negligible IO wait (around 0.1%). Our test instance

Re: Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

Hello Fabian, I have created a directory on my host machine user directory ( /home/myuser/mydir ) and I am mapping it as a volume with Docker for the TaskManager and JobManager containers. Each container will thus have the following directory /home/flink/htmp host ---> container /home/myuser/mydi

stackoverflow question

Hello can someone take a look at my question aver at stackoverflow . or answer here.

Re: Query regarding state backend for Custom Map Function

Hi, unfortunately, I think it is a little unlikely that it will still make it into 1.2. Best, Stefan > Am 01.12.2016 um 20:29 schrieb Anirudh Mallem : > > Thanks a lot Stefan. I got what I was looking for. Is the MapState > functionality coming as a part of the 1.2 release? > > From: Stefan

Re: Flink 1.1.3 OOME Permgen

Thank you for reporting the issue Konstantin. I've filed a JIRA for the jackson issue: https://issues.apache.org/jira/browse/FLINK-5233. As I said in the JIRA, I propose to upgrade to Jackson 2.7.8, as this version contains the fix for the issue, but its not a major jackson upgrade. Any chance you

Re: separation of JVMs for different applications

Hi Manu, with Flip-6 we will be able to support stricter application isolation by starting for each job a dedicated JobManager which will execute its tasks on TM reserved solely for this job. But at the same time we will continue supporting the multi-tenant cluster mode where tasks belonging to mu

Re: Dealing with Multiple sinks in Flink

Hi Robert, I had resolved this issue earlier as I had not set the Kafka source parallelism to number of partitions, so I was getting the issue of window not getting triggered. Now I am facing the same issue, I tried to check the watermark value by using visualVM locally but I am not seeing that v

Re: separation of JVMs for different applications

Hi Manu, As far as I know, there are not plans to change the stand-alone deployment. FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, etc.) which allow to start Flink processes per job. Till (in CC) is more familiar with the FLIP-6 effort and might be able to add more detail

Re: Flink 1.1.3 OOME Permgen

Hi Konstantin, Regarding 2): I've opened FLINK-5227 to update the documentation [1]. Regarding the Row type: The Row type was introduced for flink-table and was later used by other modules. There is FLINK-5186 to move Row and all the related TypeInfo (+serializer and comparator) to flink-core [2]

Re: Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

Hi Miguel, the exception does indeed indicate that the process ran out of available disk space. The quoted paragraph of the blog post describes the situation when you receive the IOE. By default the systems default tmp dir is used. I don't know which folder that would be in a Docker setup. You ca