[julia-users] What does sizehint! do, exactly?

[Seth]

I know it's good to use sizehint! with an estimate of the sizes of 
(variable-length) containers such as vectors, but I have a couple of 
questions I'm hoping someone could answer:

1) what are the benefits of using sizehint!? (How does it work, and under 
what circumstances is it beneficial?)
2) what are the implications (positive/negative, if any) of overestimating 
the size of a container?

Thanks.

Re: [julia-users] What does sizehint! do, exactly?

[Stefan Karpinski]

If you expect that you're going to have to push a lot of values onto a
vector, you can avoid the cost of incremental reallocation by doing it once
up front.

On Wednesday, October 21, 2015, Jacob Quinn  wrote:

> The way I came to understand was to just take a peak at the [source code](
> https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670);
> I find it pretty legible. The basic idea is that the underlying "storage"
> of a Julia Array{T,N} can actually be (and often is) different than the
> size(A) in Julia. sizehint! modifies that underlying storage without
> changing the size(A) in Julia.
>
> -Jacob
>
> On Wed, Oct 21, 2015 at 12:46 PM, Seth  > wrote:
>
>> I know it's good to use sizehint! with an estimate of the sizes of
>> (variable-length) containers such as vectors, but I have a couple of
>> questions I'm hoping someone could answer:
>>
>> 1) what are the benefits of using sizehint!? (How does it work, and under
>> what circumstances is it beneficial?)
>> 2) what are the implications (positive/negative, if any) of
>> overestimating the size of a container?
>>
>> Thanks.
>>
>
>

Re: [julia-users] What does sizehint! do, exactly?

[Jacob Quinn]

The way I came to understand was to just take a peak at the [source code](
https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670);
I find it pretty legible. The basic idea is that the underlying "storage"
of a Julia Array{T,N} can actually be (and often is) different than the
size(A) in Julia. sizehint! modifies that underlying storage without
changing the size(A) in Julia.

-Jacob

On Wed, Oct 21, 2015 at 12:46 PM, Seth  wrote:

> I know it's good to use sizehint! with an estimate of the sizes of
> (variable-length) containers such as vectors, but I have a couple of
> questions I'm hoping someone could answer:
>
> 1) what are the benefits of using sizehint!? (How does it work, and under
> what circumstances is it beneficial?)
> 2) what are the implications (positive/negative, if any) of overestimating
> the size of a container?
>
> Thanks.
>

Re: [julia-users] What does sizehint! do, exactly?

[Seth]

Thanks, Jacob and Stefan. What happens if you overestimate? Is the 
allocated-but-not-used memory eventually freed, or is it tied up until the 
object gets removed?

On Wednesday, October 21, 2015 at 12:18:28 PM UTC-7, Stefan Karpinski wrote:
>
> If you expect that you're going to have to push a lot of values onto a 
> vector, you can avoid the cost of incremental reallocation by doing it once 
> up front.
>
> On Wednesday, October 21, 2015, Jacob Quinn  > wrote:
>
>> The way I came to understand was to just take a peak at the [source code](
>> https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670);
>>  
>> I find it pretty legible. The basic idea is that the underlying "storage" 
>> of a Julia Array{T,N} can actually be (and often is) different than the 
>> size(A) in Julia. sizehint! modifies that underlying storage without 
>> changing the size(A) in Julia.
>>
>> -Jacob
>>
>> On Wed, Oct 21, 2015 at 12:46 PM, Seth  wrote:
>>
>>> I know it's good to use sizehint! with an estimate of the sizes of 
>>> (variable-length) containers such as vectors, but I have a couple of 
>>> questions I'm hoping someone could answer:
>>>
>>> 1) what are the benefits of using sizehint!? (How does it work, and 
>>> under what circumstances is it beneficial?)
>>> 2) what are the implications (positive/negative, if any) of 
>>> overestimating the size of a container?
>>>
>>> Thanks.
>>>
>>
>>

Re: [julia-users] What does sizehint! do, exactly?

[Jacob Quinn]

I believe it stays allocated until the object is removed. There's an old
issue about providing a way to "shrink" the underlying storage:
https://github.com/JuliaLang/julia/issues/2879

-Jacob

On Wed, Oct 21, 2015 at 1:26 PM, Seth  wrote:

> Thanks, Jacob and Stefan. What happens if you overestimate? Is the
> allocated-but-not-used memory eventually freed, or is it tied up until the
> object gets removed?
>
> On Wednesday, October 21, 2015 at 12:18:28 PM UTC-7, Stefan Karpinski
> wrote:
>>
>> If you expect that you're going to have to push a lot of values onto a
>> vector, you can avoid the cost of incremental reallocation by doing it once
>> up front.
>>
>> On Wednesday, October 21, 2015, Jacob Quinn  wrote:
>>
>>> The way I came to understand was to just take a peak at the [source
>>> code](
>>> https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670);
>>> I find it pretty legible. The basic idea is that the underlying "storage"
>>> of a Julia Array{T,N} can actually be (and often is) different than the
>>> size(A) in Julia. sizehint! modifies that underlying storage without
>>> changing the size(A) in Julia.
>>>
>>> -Jacob
>>>
>>> On Wed, Oct 21, 2015 at 12:46 PM, Seth  wrote:
>>>
 I know it's good to use sizehint! with an estimate of the sizes of
 (variable-length) containers such as vectors, but I have a couple of
 questions I'm hoping someone could answer:

 1) what are the benefits of using sizehint!? (How does it work, and
 under what circumstances is it beneficial?)
 2) what are the implications (positive/negative, if any) of
 overestimating the size of a container?

 Thanks.

>>>
>>>

Re: [julia-users] What does sizehint! do, exactly?

[Seth]

Is there a reason Julia doesn't use jl_array_del_end as a means to allow 
users to shrink the allocation for an array (either within sizehint! or as 
a separate function)?

On Wednesday, October 21, 2015 at 12:32:04 PM UTC-7, Jacob Quinn wrote:
>
> I believe it stays allocated until the object is removed. There's an old 
> issue about providing a way to "shrink" the underlying storage: 
> https://github.com/JuliaLang/julia/issues/2879
>
> -Jacob
>
> On Wed, Oct 21, 2015 at 1:26 PM, Seth  > wrote:
>
>> Thanks, Jacob and Stefan. What happens if you overestimate? Is the 
>> allocated-but-not-used memory eventually freed, or is it tied up until the 
>> object gets removed?
>>
>> On Wednesday, October 21, 2015 at 12:18:28 PM UTC-7, Stefan Karpinski 
>> wrote:
>>>
>>> If you expect that you're going to have to push a lot of values onto a 
>>> vector, you can avoid the cost of incremental reallocation by doing it once 
>>> up front.
>>>
>>> On Wednesday, October 21, 2015, Jacob Quinn  wrote:
>>>
 The way I came to understand was to just take a peak at the [source 
 code](
 https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670
  
 );
  
 I find it pretty legible. The basic idea is that the underlying "storage" 
 of a Julia Array{T,N} can actually be (and often is) different than the 
 size(A) in Julia. sizehint! modifies that underlying storage without 
 changing the size(A) in Julia.

 -Jacob

 On Wed, Oct 21, 2015 at 12:46 PM, Seth  wrote:

> I know it's good to use sizehint! with an estimate of the sizes of 
> (variable-length) containers such as vectors, but I have a couple of 
> questions I'm hoping someone could answer:
>
> 1) what are the benefits of using sizehint!? (How does it work, and 
> under what circumstances is it beneficial?)
> 2) what are the implications (positive/negative, if any) of 
> overestimating the size of a container?
>
> Thanks.
>


>

Re: [julia-users] What does sizehint! do, exactly?

[Jacob Quinn]

I think that's why there's an open issue :)

On Wed, Oct 21, 2015 at 3:29 PM, Seth  wrote:

> Is there a reason Julia doesn't use jl_array_del_end as a means to allow
> users to shrink the allocation for an array (either within sizehint! or as
> a separate function)?
>
> On Wednesday, October 21, 2015 at 12:32:04 PM UTC-7, Jacob Quinn wrote:
>>
>> I believe it stays allocated until the object is removed. There's an old
>> issue about providing a way to "shrink" the underlying storage:
>> https://github.com/JuliaLang/julia/issues/2879
>>
>> -Jacob
>>
>> On Wed, Oct 21, 2015 at 1:26 PM, Seth  wrote:
>>
>>> Thanks, Jacob and Stefan. What happens if you overestimate? Is the
>>> allocated-but-not-used memory eventually freed, or is it tied up until the
>>> object gets removed?
>>>
>>> On Wednesday, October 21, 2015 at 12:18:28 PM UTC-7, Stefan Karpinski
>>> wrote:

 If you expect that you're going to have to push a lot of values onto a
 vector, you can avoid the cost of incremental reallocation by doing it once
 up front.

 On Wednesday, October 21, 2015, Jacob Quinn 
 wrote:

> The way I came to understand was to just take a peak at the [source
> code](
> https://github.com/JuliaLang/julia/blob/ae154d076a6ae75bfdb9a0a377a6a5f9b0e1096f/src/array.c#L670
> );
> I find it pretty legible. The basic idea is that the underlying "storage"
> of a Julia Array{T,N} can actually be (and often is) different than the
> size(A) in Julia. sizehint! modifies that underlying storage without
> changing the size(A) in Julia.
>
> -Jacob
>
> On Wed, Oct 21, 2015 at 12:46 PM, Seth  wrote:
>
>> I know it's good to use sizehint! with an estimate of the sizes of
>> (variable-length) containers such as vectors, but I have a couple of
>> questions I'm hoping someone could answer:
>>
>> 1) what are the benefits of using sizehint!? (How does it work, and
>> under what circumstances is it beneficial?)
>> 2) what are the implications (positive/negative, if any) of
>> overestimating the size of a container?
>>
>> Thanks.
>>
>
>
>>