On 08/04/2014 10:57 PM, Michael Armbrust wrote:
> If mesos is allocating a container that is exactly the same as the max
> heap size then that is leaving no buffer space for non-heap JVM memory,
> which seems wrong to me.
>
This can be a cause. I am now wondering how mesos pick up the size and
set
If mesos is allocating a container that is exactly the same as the max heap
size then that is leaving no buffer space for non-heap JVM memory, which
seems wrong to me.
The problem here is that cacheTable is more aggressive about grabbing large
ByteBuffers during caching (which it later releases wh
On 08/03/2014 02:33 AM, Michael Armbrust wrote:
> I am not a mesos expert... but it sounds like there is some mismatch
> between the size that mesos is giving you and the maximum heap size of
> the executors (-Xmx).
>
It seems that mesos is giving the correct size to java process. It has
exact siz
I am not a mesos expert... but it sounds like there is some mismatch
between the size that mesos is giving you and the maximum heap size of the
executors (-Xmx).
On Fri, Aug 1, 2014 at 12:07 AM, Gurvinder Singh wrote:
> It is not getting out of memory exception. I am using Mesos as cluster
> ma
Thanks Michael for explaination. Actually I tried caching the RDD and
making table on it. But the performance for cacheTable was 3X better
than caching RDD. Now I know why it is better. But is it possible to
add the support for persistence level into cacheTable itself like RDD.
May be it is not rel
cacheTable uses a special columnar caching technique that is optimized for
SchemaRDDs. It something similar to MEMORY_ONLY_SER but not quite. You
can specify the persistence level on the SchemaRDD itself and register that
as a temporary table, however it is likely you will not get as good
perform
Hi,
I am wondering how can I specify the persistence level in cacheTable. As
it is takes only table name as parameter. It should be possible to
specify the persistence level.
- Gurvinder
