Re: [julia-users] High memory consumption with loops

[Venil Noronha]

Thanks Steve!

The problem was finally found to be with some matrix operations. We've used 
an alternate approach and it seems to be fixed now.

Venil

On Thursday, August 25, 2016 at 6:41:25 PM UTC-7, vav...@uwaterloo.ca wrote:
>
> I don't have any suggestions about the performance, but I do have a code 
> correctness suggestion.  It appears from your code snippet that you are 
> using dictionaries called "effectiveness" and "resource_fines", and that 
> the keys to these dictionaries are a mutable type.  This can lead to subtle 
> program bugs.  For example, the following code sequence:
>
> d = Dict{Array{Int,1},Float64}()
> a = [4,5,6]
> d[a] = 9.1
> a[2] = 4
>
> will leave internal indexing structure of d in a corrupted state.  It is 
> safer to use immutables as keys for Dict.  If you must use mutables, then 
> whenever you assign a dictionary entry it is safest to insert a 'deepcopy' 
> operation; e.g., the above snippet can be fixed by substituting the 
> following for d[a]=9.1:
>d[deepcopy(a)] = 9.1
>
> -- Steve Vavasis
>
>
> On Thursday, August 25, 2016 at 2:21:30 AM UTC-4, Venil Noronha wrote:
>>
>> I've also tried to @profile, however, julia libraries seem to be executed 
>> the most.
>>
>> Venil
>>
>> On Wednesday, August 24, 2016 at 11:16:53 PM UTC-7, Venil Noronha wrote:
>>>
>>> I've just been able to @time the loop so far to see allocations at 
>>> iteration level. I haven't yet tried @code_warntype; I'll probably do that 
>>> next and see if I can get somewhere.
>>>
>>> Thanks,
>>> Venil
>>>
>>> On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote:

 What have you tried so far? See 
 http://docs.julialang.org/en/latest/manual/ 

 performance-tips/#tools 
 , 
 esp. @code_warntype. 

 --Tim 

 On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: 
 > The following code seems to consume a lot of memory. Could anyone 
 spot what 
 > I'm doing wrong here? I'm also using the JuMP library. 
 > 
 > function initUtilityConstraint() 
 > c = categories[1] 
 > me = attack_methods[1] 
 > t = teams[1] 
 > tuple = Simulate.cmt(c, me, t) 
 > w = windows[1] 
 > r = resources[1] 
 > wrtuple = Simulate.wr(w, r) 
 > for ni in 1:size(list,1), c in categories,* f in flights* 
 > performloop(ni, c, f, tuple, wrtuple) 
 > end 
 > end 
 > 
 > function performloop(ni, c, f, tuple, wrtuple) 
 > fi = findfirst(flights, f) 
 > for w in windows, me in attack_methods 
 > tuple.c = c 
 > tuple.m = me 
 > total = 0.0 
 > for t in teams 
 > tuple.t = t 
 > strat = linearizeStrategyS(f, c, t, w, ni) 
 > total = total + effectiveness[tuple]*strat 
 > end 
 > 
 > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) 
 > 
 > for w2 in owindows, r2 in resources 
 > wrtuple.w = w2 
 > wrtuple.r = r2 
 > over = linearizeOverflow(w2, r2, ni) 
 > total = total - resource_fines[wrtuple]*over 
 > end 
 > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) 
 + 
 > flight_vals[fi] ))) 
 > @addConstraint(m, sc[c] <= total) 
 > end 
 > end 
 > 
 > Following are the stats for piece this code. Stats were recorded 
 using 
 > @time. 
 > 
 > For 1 item in the flights array, it takes about 620KB to execute the 
 > performloop method - peak memory consumption by the program is 
 8.84GBs. 
 > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. 
 > 3 flights - 3.45MB - 9.60GBs 
 > 4 flights - 4.35MB - 10.24GBs 
 > 5 flights - 10.8MB - 15.63GBs 
 > 
 > It'd be great if someone could help me with this asap! 
 > 
 > Thanks. 

Re: [julia-users] High memory consumption with loops

[vavasis]

I don't have any suggestions about the performance, but I do have a code 
correctness suggestion.  It appears from your code snippet that you are 
using dictionaries called "effectiveness" and "resource_fines", and that 
the keys to these dictionaries are a mutable type.  This can lead to subtle 
program bugs.  For example, the following code sequence:

d = Dict{Array{Int,1},Float64}()
a = [4,5,6]
d[a] = 9.1
a[2] = 4

will leave internal indexing structure of d in a corrupted state.  It is 
safer to use immutables as keys for Dict.  If you must use mutables, then 
whenever you assign a dictionary entry it is safest to insert a 'deepcopy' 
operation; e.g., the above snippet can be fixed by substituting the 
following for d[a]=9.1:
   d[deepcopy(a)] = 9.1

-- Steve Vavasis


On Thursday, August 25, 2016 at 2:21:30 AM UTC-4, Venil Noronha wrote:
>
> I've also tried to @profile, however, julia libraries seem to be executed 
> the most.
>
> Venil
>
> On Wednesday, August 24, 2016 at 11:16:53 PM UTC-7, Venil Noronha wrote:
>>
>> I've just been able to @time the loop so far to see allocations at 
>> iteration level. I haven't yet tried @code_warntype; I'll probably do that 
>> next and see if I can get somewhere.
>>
>> Thanks,
>> Venil
>>
>> On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote:
>>>
>>> What have you tried so far? See http://docs.julialang.org/en/latest/manual/ 
>>>
>>> performance-tips/#tools 
>>> , 
>>> esp. @code_warntype. 
>>>
>>> --Tim 
>>>
>>> On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: 
>>> > The following code seems to consume a lot of memory. Could anyone spot 
>>> what 
>>> > I'm doing wrong here? I'm also using the JuMP library. 
>>> > 
>>> > function initUtilityConstraint() 
>>> > c = categories[1] 
>>> > me = attack_methods[1] 
>>> > t = teams[1] 
>>> > tuple = Simulate.cmt(c, me, t) 
>>> > w = windows[1] 
>>> > r = resources[1] 
>>> > wrtuple = Simulate.wr(w, r) 
>>> > for ni in 1:size(list,1), c in categories,* f in flights* 
>>> > performloop(ni, c, f, tuple, wrtuple) 
>>> > end 
>>> > end 
>>> > 
>>> > function performloop(ni, c, f, tuple, wrtuple) 
>>> > fi = findfirst(flights, f) 
>>> > for w in windows, me in attack_methods 
>>> > tuple.c = c 
>>> > tuple.m = me 
>>> > total = 0.0 
>>> > for t in teams 
>>> > tuple.t = t 
>>> > strat = linearizeStrategyS(f, c, t, w, ni) 
>>> > total = total + effectiveness[tuple]*strat 
>>> > end 
>>> > 
>>> > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) 
>>> > 
>>> > for w2 in owindows, r2 in resources 
>>> > wrtuple.w = w2 
>>> > wrtuple.r = r2 
>>> > over = linearizeOverflow(w2, r2, ni) 
>>> > total = total - resource_fines[wrtuple]*over 
>>> > end 
>>> > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) 
>>> + 
>>> > flight_vals[fi] ))) 
>>> > @addConstraint(m, sc[c] <= total) 
>>> > end 
>>> > end 
>>> > 
>>> > Following are the stats for piece this code. Stats were recorded using 
>>> > @time. 
>>> > 
>>> > For 1 item in the flights array, it takes about 620KB to execute the 
>>> > performloop method - peak memory consumption by the program is 
>>> 8.84GBs. 
>>> > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. 
>>> > 3 flights - 3.45MB - 9.60GBs 
>>> > 4 flights - 4.35MB - 10.24GBs 
>>> > 5 flights - 10.8MB - 15.63GBs 
>>> > 
>>> > It'd be great if someone could help me with this asap! 
>>> > 
>>> > Thanks. 
>>>
>>>
>>>

Re: [julia-users] High memory consumption with loops

[Venil Noronha]

I've also tried to @profile, however, julia libraries seem to be executed 
the most.

Venil

On Wednesday, August 24, 2016 at 11:16:53 PM UTC-7, Venil Noronha wrote:
>
> I've just been able to @time the loop so far to see allocations at 
> iteration level. I haven't yet tried @code_warntype; I'll probably do that 
> next and see if I can get somewhere.
>
> Thanks,
> Venil
>
> On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote:
>>
>> What have you tried so far? See http://docs.julialang.org/en/latest/manual/ 
>>
>> performance-tips/#tools 
>> , 
>> esp. @code_warntype. 
>>
>> --Tim 
>>
>> On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: 
>> > The following code seems to consume a lot of memory. Could anyone spot 
>> what 
>> > I'm doing wrong here? I'm also using the JuMP library. 
>> > 
>> > function initUtilityConstraint() 
>> > c = categories[1] 
>> > me = attack_methods[1] 
>> > t = teams[1] 
>> > tuple = Simulate.cmt(c, me, t) 
>> > w = windows[1] 
>> > r = resources[1] 
>> > wrtuple = Simulate.wr(w, r) 
>> > for ni in 1:size(list,1), c in categories,* f in flights* 
>> > performloop(ni, c, f, tuple, wrtuple) 
>> > end 
>> > end 
>> > 
>> > function performloop(ni, c, f, tuple, wrtuple) 
>> > fi = findfirst(flights, f) 
>> > for w in windows, me in attack_methods 
>> > tuple.c = c 
>> > tuple.m = me 
>> > total = 0.0 
>> > for t in teams 
>> > tuple.t = t 
>> > strat = linearizeStrategyS(f, c, t, w, ni) 
>> > total = total + effectiveness[tuple]*strat 
>> > end 
>> > 
>> > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) 
>> > 
>> > for w2 in owindows, r2 in resources 
>> > wrtuple.w = w2 
>> > wrtuple.r = r2 
>> > over = linearizeOverflow(w2, r2, ni) 
>> > total = total - resource_fines[wrtuple]*over 
>> > end 
>> > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + 
>> > flight_vals[fi] ))) 
>> > @addConstraint(m, sc[c] <= total) 
>> > end 
>> > end 
>> > 
>> > Following are the stats for piece this code. Stats were recorded using 
>> > @time. 
>> > 
>> > For 1 item in the flights array, it takes about 620KB to execute the 
>> > performloop method - peak memory consumption by the program is 8.84GBs. 
>> > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. 
>> > 3 flights - 3.45MB - 9.60GBs 
>> > 4 flights - 4.35MB - 10.24GBs 
>> > 5 flights - 10.8MB - 15.63GBs 
>> > 
>> > It'd be great if someone could help me with this asap! 
>> > 
>> > Thanks. 
>>
>>
>>

Re: [julia-users] High memory consumption with loops

[Venil Noronha]

I've just been able to @time the loop so far to see allocations at 
iteration level. I haven't yet tried @code_warntype; I'll probably do that 
next and see if I can get somewhere.

Thanks,
Venil

On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote:
>
> What have you tried so far? See http://docs.julialang.org/en/latest/manual/ 
>
> performance-tips/#tools 
> , 
> esp. @code_warntype. 
>
> --Tim 
>
> On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: 
> > The following code seems to consume a lot of memory. Could anyone spot 
> what 
> > I'm doing wrong here? I'm also using the JuMP library. 
> > 
> > function initUtilityConstraint() 
> > c = categories[1] 
> > me = attack_methods[1] 
> > t = teams[1] 
> > tuple = Simulate.cmt(c, me, t) 
> > w = windows[1] 
> > r = resources[1] 
> > wrtuple = Simulate.wr(w, r) 
> > for ni in 1:size(list,1), c in categories,* f in flights* 
> > performloop(ni, c, f, tuple, wrtuple) 
> > end 
> > end 
> > 
> > function performloop(ni, c, f, tuple, wrtuple) 
> > fi = findfirst(flights, f) 
> > for w in windows, me in attack_methods 
> > tuple.c = c 
> > tuple.m = me 
> > total = 0.0 
> > for t in teams 
> > tuple.t = t 
> > strat = linearizeStrategyS(f, c, t, w, ni) 
> > total = total + effectiveness[tuple]*strat 
> > end 
> > 
> > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) 
> > 
> > for w2 in owindows, r2 in resources 
> > wrtuple.w = w2 
> > wrtuple.r = r2 
> > over = linearizeOverflow(w2, r2, ni) 
> > total = total - resource_fines[wrtuple]*over 
> > end 
> > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + 
> > flight_vals[fi] ))) 
> > @addConstraint(m, sc[c] <= total) 
> > end 
> > end 
> > 
> > Following are the stats for piece this code. Stats were recorded using 
> > @time. 
> > 
> > For 1 item in the flights array, it takes about 620KB to execute the 
> > performloop method - peak memory consumption by the program is 8.84GBs. 
> > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. 
> > 3 flights - 3.45MB - 9.60GBs 
> > 4 flights - 4.35MB - 10.24GBs 
> > 5 flights - 10.8MB - 15.63GBs 
> > 
> > It'd be great if someone could help me with this asap! 
> > 
> > Thanks. 
>
>
>

Re: [julia-users] High memory consumption with loops

[Tim Holy]

What have you tried so far? See http://docs.julialang.org/en/latest/manual/
performance-tips/#tools, esp. @code_warntype.

--Tim

On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote:
> The following code seems to consume a lot of memory. Could anyone spot what
> I'm doing wrong here? I'm also using the JuMP library.
> 
> function initUtilityConstraint()
> c = categories[1]
> me = attack_methods[1]
> t = teams[1]
> tuple = Simulate.cmt(c, me, t)
> w = windows[1]
> r = resources[1]
> wrtuple = Simulate.wr(w, r)
> for ni in 1:size(list,1), c in categories,* f in flights*
> performloop(ni, c, f, tuple, wrtuple)
> end
> end
> 
> function performloop(ni, c, f, tuple, wrtuple)
> fi = findfirst(flights, f)
> for w in windows, me in attack_methods
> tuple.c = c
> tuple.m = me
> total = 0.0
> for t in teams
> tuple.t = t
> strat = linearizeStrategyS(f, c, t, w, ni)
> total = total + effectiveness[tuple]*strat
> end
> 
> total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] )
> 
> for w2 in owindows, r2 in resources
> wrtuple.w = w2
> wrtuple.r = r2
> over = linearizeOverflow(w2, r2, ni)
> total = total - resource_fines[wrtuple]*over
> end
> # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) +
> flight_vals[fi] )))
> @addConstraint(m, sc[c] <= total)
> end
> end
> 
> Following are the stats for piece this code. Stats were recorded using
> @time.
> 
> For 1 item in the flights array, it takes about 620KB to execute the
> performloop method - peak memory consumption by the program is 8.84GBs.
> 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs.
> 3 flights - 3.45MB - 9.60GBs
> 4 flights - 4.35MB - 10.24GBs
> 5 flights - 10.8MB - 15.63GBs
> 
> It'd be great if someone could help me with this asap!
> 
> Thanks.

[julia-users] High memory consumption with loops

[Venil Noronha]

The following code seems to consume a lot of memory. Could anyone spot what 
I'm doing wrong here? I'm also using the JuMP library.

function initUtilityConstraint()
c = categories[1]
me = attack_methods[1]
t = teams[1]
tuple = Simulate.cmt(c, me, t)
w = windows[1]
r = resources[1]
wrtuple = Simulate.wr(w, r)
for ni in 1:size(list,1), c in categories,* f in flights*
performloop(ni, c, f, tuple, wrtuple)
end
end

function performloop(ni, c, f, tuple, wrtuple)
fi = findfirst(flights, f)
for w in windows, me in attack_methods
tuple.c = c
tuple.m = me
total = 0.0
for t in teams
tuple.t = t
strat = linearizeStrategyS(f, c, t, w, ni)
total = total + effectiveness[tuple]*strat
end

total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] )

for w2 in owindows, r2 in resources
wrtuple.w = w2
wrtuple.r = r2
over = linearizeOverflow(w2, r2, ni)
total = total - resource_fines[wrtuple]*over
end
# println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + 
flight_vals[fi] )))
@addConstraint(m, sc[c] <= total) 
end
end

Following are the stats for piece this code. Stats were recorded using 
@time.

For 1 item in the flights array, it takes about 620KB to execute the 
performloop method - peak memory consumption by the program is 8.84GBs.
2 flights - 1.02MB per iteration of performloop - peak 8.88GBs.
3 flights - 3.45MB - 9.60GBs
4 flights - 4.35MB - 10.24GBs
5 flights - 10.8MB - 15.63GBs

It'd be great if someone could help me with this asap!

Thanks.