[julia-users] Re: Is there a way to reclaim memory?
I almost think that you may need to come up with your own structure, instead of a Vector of Vectors, to avoid the problems you are having. The nice thing about Julia, you'd be able to make your own data structure <: AbstractArray, and be able to use it with many Julia functions. For example, you could have the data packed into a single large vector, and use another vector of Int32 to give the indices of the end of each "subvector". (to build it, you might need to use another structure to hold all of the edges, before sorting them to add to the final large vector) On Tuesday, August 2, 2016 at 7:41:26 PM UTC+2, Seth wrote: > > So, a 62.5 million edge LightGraphs.Graph should only take about a 1.25 > gigs of memory, all-in. However, because the edges are being added to > vectors within the datastructure, and each vector is doing its own memory > allocation, we wind up with the Julia process taking ~6.5 gigs. > > Given that we don't know the degree distribution of the graph before we > load it (and therefore can't use sizehint! effectively), is there any way > to reclaim the allocated-but-unused memory from all these vectors, with the > accepted large performance hit that would come should we decide to add > another edge? >
Re: [julia-users] Re: Is there a way to reclaim memory?
On Tuesday, August 2, 2016 at 11:07:29 AM UTC-7, Erik Schnetter wrote:
>
> Can you make a copy of the vector, and assign this copy to the original
> variable? As in:
>
> ```Julia
> type Node
> edges::Vector{X}
> end
>
> ...
> node.edges = copy(node.edges)
> ...
> ```
>
> -erik
>
>
>
> On Tue, Aug 2, 2016 at 1:56 PM, Seth
> > wrote:
>
>>
>>
>> On Tuesday, August 2, 2016 at 10:41:26 AM UTC-7, Seth wrote:
>>>
>>> So, a 62.5 million edge LightGraphs.Graph should only take about a 1.25
>>> gigs of memory, all-in. However, because the edges are being added to
>>> vectors within the datastructure, and each vector is doing its own memory
>>> allocation, we wind up with the Julia process taking ~6.5 gigs.
>>>
>>> Given that we don't know the degree distribution of the graph before we
>>> load it (and therefore can't use sizehint! effectively), is there any way
>>> to reclaim the allocated-but-unused memory from all these vectors, with the
>>> accepted large performance hit that would come should we decide to add
>>> another edge?
>>>
>>
>> Things that I've tried that *don't* work:
>>
>> - explicitly calling sizehint! after the vectors have been created and
>> populated
>> - calling resize! after the vectors have been created and populated
>>
>>
>
>
>
>
Thanks. I tried this:
for i = 1:nv(g)
g.fadjlist[i] = copy(g.fadjlist[i])
end
and that actually increased memory footprint from 6.25 to 7.96 GB. I even
tried an explicit gc() afterwards.
Re: [julia-users] Re: Is there a way to reclaim memory?
Can you make a copy of the vector, and assign this copy to the original
variable? As in:
```Julia
type Node
edges::Vector{X}
end
...
node.edges = copy(node.edges)
...
```
-erik
On Tue, Aug 2, 2016 at 1:56 PM, Seth wrote:
>
>
> On Tuesday, August 2, 2016 at 10:41:26 AM UTC-7, Seth wrote:
>>
>> So, a 62.5 million edge LightGraphs.Graph should only take about a 1.25
>> gigs of memory, all-in. However, because the edges are being added to
>> vectors within the datastructure, and each vector is doing its own memory
>> allocation, we wind up with the Julia process taking ~6.5 gigs.
>>
>> Given that we don't know the degree distribution of the graph before we
>> load it (and therefore can't use sizehint! effectively), is there any way
>> to reclaim the allocated-but-unused memory from all these vectors, with the
>> accepted large performance hit that would come should we decide to add
>> another edge?
>>
>
> Things that I've tried that *don't* work:
>
> - explicitly calling sizehint! after the vectors have been created and
> populated
> - calling resize! after the vectors have been created and populated
>
>
--
Erik Schnetter
http://www.perimeterinstitute.ca/personal/eschnetter/
[julia-users] Re: Is there a way to reclaim memory?
Copy it into a new array with the correct size? On Tuesday, August 2, 2016 at 7:56:35 PM UTC+2, Seth wrote: > > > > On Tuesday, August 2, 2016 at 10:41:26 AM UTC-7, Seth wrote: >> >> So, a 62.5 million edge LightGraphs.Graph should only take about a 1.25 >> gigs of memory, all-in. However, because the edges are being added to >> vectors within the datastructure, and each vector is doing its own memory >> allocation, we wind up with the Julia process taking ~6.5 gigs. >> >> Given that we don't know the degree distribution of the graph before we >> load it (and therefore can't use sizehint! effectively), is there any way >> to reclaim the allocated-but-unused memory from all these vectors, with the >> accepted large performance hit that would come should we decide to add >> another edge? >> > > Things that I've tried that *don't* work: > > - explicitly calling sizehint! after the vectors have been created and > populated > - calling resize! after the vectors have been created and populated > >
[julia-users] Re: Is there a way to reclaim memory?
On Tuesday, August 2, 2016 at 10:41:26 AM UTC-7, Seth wrote: > > So, a 62.5 million edge LightGraphs.Graph should only take about a 1.25 > gigs of memory, all-in. However, because the edges are being added to > vectors within the datastructure, and each vector is doing its own memory > allocation, we wind up with the Julia process taking ~6.5 gigs. > > Given that we don't know the degree distribution of the graph before we > load it (and therefore can't use sizehint! effectively), is there any way > to reclaim the allocated-but-unused memory from all these vectors, with the > accepted large performance hit that would come should we decide to add > another edge? > Things that I've tried that *don't* work: - explicitly calling sizehint! after the vectors have been created and populated - calling resize! after the vectors have been created and populated
