Re: off-heap certain operations

[Ovidiu-Cristian MARCU]

Well, it is quite important the off-heap setting and now I am curios about 
other parameters, I hope everything else is well documented or not missleading.

Best,
Ovidiu
> On 12 Feb 2016, at 19:18, Sean Owen  wrote:
> 
> I don't think much more is said since in fact it would affect parts of
> the implementations of lots of operations -- anything touching
> Tungsten. It wouldn't be meaningful to try to list everything.
> 
> The difference is allocating memory on the heap or with
> sun.misc.Unsafe. This is definitely something you'd need to be a
> developer to know whether to use, and if you're a developer and
> curious, you can just grep the code for this flag, and/or read into
> what Tungsten does.
> 
> Personally, I would leave this off.
> 
> On Fri, Feb 12, 2016 at 6:10 PM, Ovidiu-Cristian MARCU
>  wrote:
>> I found nothing about the certain operations. Still not clear, certain is
>> poor documentation. Can someone give an answer so I can consider using this
>> new release?
>> spark.memory.offHeap.enabled
>> 
>> If true, Spark will attempt to use off-heap memory for certain operations.
>> 
>> On 12 Feb 2016, at 13:21, Ted Yu  wrote:
>> 
>> SP
>> 
>> 


-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: off-heap certain operations

[Li Ming Tsai]

Hi Sean,

> Personally, I would leave this off.
Is this not production ready, thus we should disable it?

Thanks,
Liming



From: Sean Owen 
Sent: Saturday, February 13, 2016 2:18 AM
To: Ovidiu-Cristian MARCU
Cc: Ted Yu; Sea; user@spark.apache.org
Subject: Re: off-heap certain operations

I don't think much more is said since in fact it would affect parts of
the implementations of lots of operations -- anything touching
Tungsten. It wouldn't be meaningful to try to list everything.

The difference is allocating memory on the heap or with
sun.misc.Unsafe. This is definitely something you'd need to be a
developer to know whether to use, and if you're a developer and
curious, you can just grep the code for this flag, and/or read into
what Tungsten does.

Personally, I would leave this off.

On Fri, Feb 12, 2016 at 6:10 PM, Ovidiu-Cristian MARCU
 wrote:
> I found nothing about the certain operations. Still not clear, certain is
> poor documentation. Can someone give an answer so I can consider using this
> new release?
> spark.memory.offHeap.enabled
>
> If true, Spark will attempt to use off-heap memory for certain operations.
>
> On 12 Feb 2016, at 13:21, Ted Yu  wrote:
>
> SP
>
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: off-heap certain operations

[Sean Owen]

I don't think much more is said since in fact it would affect parts of
the implementations of lots of operations -- anything touching
Tungsten. It wouldn't be meaningful to try to list everything.

The difference is allocating memory on the heap or with
sun.misc.Unsafe. This is definitely something you'd need to be a
developer to know whether to use, and if you're a developer and
curious, you can just grep the code for this flag, and/or read into
what Tungsten does.

Personally, I would leave this off.

On Fri, Feb 12, 2016 at 6:10 PM, Ovidiu-Cristian MARCU
 wrote:
> I found nothing about the certain operations. Still not clear, certain is
> poor documentation. Can someone give an answer so I can consider using this
> new release?
> spark.memory.offHeap.enabled
>
> If true, Spark will attempt to use off-heap memory for certain operations.
>
> On 12 Feb 2016, at 13:21, Ted Yu  wrote:
>
> SP
>
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: off-heap certain operations

[Ovidiu-Cristian MARCU]

I found nothing about the certain operations. Still not clear, certain is poor 
documentation. Can someone give an answer so I can consider using this new 
release?
spark.memory.offHeap.enabled

If true, Spark will attempt to use off-heap memory for certain operations.

> On 12 Feb 2016, at 13:21, Ted Yu  wrote:
> 
> SP

Re: off-heap certain operations

[Ted Yu]

Ovidiu-Cristian:
Please see the following JIRA / PR :
[SPARK-12251] Document and improve off-heap memory configurations

Cheers

On Thu, Feb 11, 2016 at 11:06 PM, Sea <261810...@qq.com> wrote:

> spark.memory.offHeap.enabled (default is false) , it is wrong in spark
> docs. Spark1.6 do not recommend to use off-heap memory.
>
>
> -- 原始邮件 --
> *发件人:* "Ovidiu-Cristian MARCU";;
> *发送时间:* 2016年2月12日(星期五) 凌晨5:51
> *收件人:* "user";
> *主题:* off-heap certain operations
>
> Hi,
>
> Reading though the latest documentation for Memory management I can see
> that the parameter spark.memory.offHeap.enabled (true by default) is
> described with ‘If true, Spark will attempt to use off-heap memory for
> certain operations’ [1].
>
> Can you please describe the certain operations you are referring to?
>
> http://spark.apache.org/docs/latest/configuration.html#memory-management
>
> Thank!
>
> Best,
> Ovidiu
>

??????off-heap certain operations

[Sea]

spark.memory.offHeap.enabled (default is false) , it is wrong in spark docs. 
Spark1.6 do not recommend to use off-heap memory.




--  --
??: "Ovidiu-Cristian MARCU";;
: 2016??2??12??(??) 5:51
??: "user"; 

????: off-heap certain operations



Hi,

Reading though the latest documentation for Memory management I can see that 
the parameter spark.memory.offHeap.enabled (true by default) is described with 
??If true, Spark will attempt to use off-heap memory for certain operations?? 
[1].


Can you please describe the certain operations you are referring to?  


http://spark.apache.org/docs/latest/configuration.html#memory-management


Thank!


Best,
Ovidiu

off-heap certain operations

[Ovidiu-Cristian MARCU]

Hi,

Reading though the latest documentation for Memory management I can see that 
the parameter spark.memory.offHeap.enabled (true by default) is described with 
‘If true, Spark will attempt to use off-heap memory for certain operations’ [1].

Can you please describe the certain operations you are referring to?  

http://spark.apache.org/docs/latest/configuration.html#memory-management 


Thank!

Best,
Ovidiu