Re: off-heap size feature request

2016-03-16 Thread Fabian Hueske
Hi Ovidiu, the parameters to configure the amount of managed memory (taskmanager.memory.size, taskmanager.memory.fraction) are valid for on and off-heap memory. Have you tried these parameters and didn't they work as expected? Best, Fabian 2016-03-16 11:43 GMT+01:00 Ovidiu-Cristian MARCU <

off-heap size feature request

2016-03-16 Thread Ovidiu-Cristian MARCU
Hi, Is it possible to add a parameter off-heap.size for the task manager off-heap memory [1]? It is not possible to limit the off-heap memory size, at least I found nothing in the documentation. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#managed-memory

Re: realtion between operator and task

2016-03-16 Thread Till Rohrmann
Hi Radu, the mapping which StreamOperator is executed by which StreamTask happens first in the StreamGraph.addOperator method. However, there is a second step in the StreamingJobGraphGenerator.createChain where chainable operators are chained and then executed by a single StreamTask. The

Re: Error when accessing secure HDFS with standalone Flink

2016-03-16 Thread Stefano Baghino
Hi Max, thanks for the tips. What we did has been running kinit on each node with the same user that then went on running the start-cluster.sh script. Right now the LDAP groups are backed by the OS ones and the user that ran the launch script is part of the flink group, that is on every node of

Re: Error when accessing secure HDFS with standalone Flink

2016-03-16 Thread Maximilian Michels
Hi Stefano, You have probably seen https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#kerberos ? Currently, all nodes need to be authenticated with the Kerberos before Flink is started (not just the JobManager). Could it be that the start-cluster.sh script actually is

Re: JDBCInputFormat preparation with Flink 1.1-SNAPSHOT and Scala 2.11

2016-03-16 Thread Robert Metzger
Sorry for joining this discussion late. Maybe this is also interesting for you: http://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ On Wed, Mar 9, 2016 at 1:47 PM, Prez Cannady wrote: > Thanks. Need to dive in a bit better,

Re: Using a POJO class wrapping an ArrayList

2016-03-16 Thread Fabian Hueske
Hi Mengqi, I did not completely understand your use case. If you would like to use a composite key (a key with multiple fields) there are two alternatives: - use a tuple as key type. This only works if all records have the same number of key fields. Tuple serialization and comparisons are very

Re: Memory ran out PageRank

2016-03-16 Thread Fabian Hueske
Hi Ovidiu, putting the CompactingHashTable aside, all data structures and algorithms that use managed memory can spill to disk if data exceeds memory capacity. It was a conscious choice to not let the CompactingHashTable spill. Once the solution set hash table is spilled, (parts of) the hash

Re: Memory ran out PageRank

2016-03-16 Thread Ovidiu-Cristian MARCU
Hi, Regarding the solution set going out of memory, I would like an issue to be filled against it. Looking into code for CompactingHashTable I see The hash table is internally divided into two parts: The hash index, and the partition buffers that store the actual records. When records are