Re: [julia-users] High memory consumption with loops
Thanks Steve! The problem was finally found to be with some matrix operations. We've used an alternate approach and it seems to be fixed now. Venil On Thursday, August 25, 2016 at 6:41:25 PM UTC-7, vav...@uwaterloo.ca wrote: > > I don't have any suggestions about the performance, but I do have a code > correctness suggestion. It appears from your code snippet that you are > using dictionaries called "effectiveness" and "resource_fines", and that > the keys to these dictionaries are a mutable type. This can lead to subtle > program bugs. For example, the following code sequence: > > d = Dict{Array{Int,1},Float64}() > a = [4,5,6] > d[a] = 9.1 > a[2] = 4 > > will leave internal indexing structure of d in a corrupted state. It is > safer to use immutables as keys for Dict. If you must use mutables, then > whenever you assign a dictionary entry it is safest to insert a 'deepcopy' > operation; e.g., the above snippet can be fixed by substituting the > following for d[a]=9.1: >d[deepcopy(a)] = 9.1 > > -- Steve Vavasis > > > On Thursday, August 25, 2016 at 2:21:30 AM UTC-4, Venil Noronha wrote: >> >> I've also tried to @profile, however, julia libraries seem to be executed >> the most. >> >> Venil >> >> On Wednesday, August 24, 2016 at 11:16:53 PM UTC-7, Venil Noronha wrote: >>> >>> I've just been able to @time the loop so far to see allocations at >>> iteration level. I haven't yet tried @code_warntype; I'll probably do that >>> next and see if I can get somewhere. >>> >>> Thanks, >>> Venil >>> >>> On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote: >>>> >>>> What have you tried so far? See >>>> http://docs.julialang.org/en/latest/manual/ >>>> >>>> performance-tips/#tools >>>> <http://docs.julialang.org/en/latest/manual/performance-tips/#tools>, >>>> esp. @code_warntype. >>>> >>>> --Tim >>>> >>>> On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: >>>> > The following code seems to consume a lot of memory. Could anyone >>>> spot what >>>> > I'm doing wrong here? I'm also using the JuMP library. >>>> > >>>> > function initUtilityConstraint() >>>> > c = categories[1] >>>> > me = attack_methods[1] >>>> > t = teams[1] >>>> > tuple = Simulate.cmt(c, me, t) >>>> > w = windows[1] >>>> > r = resources[1] >>>> > wrtuple = Simulate.wr(w, r) >>>> > for ni in 1:size(list,1), c in categories,* f in flights* >>>> > performloop(ni, c, f, tuple, wrtuple) >>>> > end >>>> > end >>>> > >>>> > function performloop(ni, c, f, tuple, wrtuple) >>>> > fi = findfirst(flights, f) >>>> > for w in windows, me in attack_methods >>>> > tuple.c = c >>>> > tuple.m = me >>>> > total = 0.0 >>>> > for t in teams >>>> > tuple.t = t >>>> > strat = linearizeStrategyS(f, c, t, w, ni) >>>> > total = total + effectiveness[tuple]*strat >>>> > end >>>> > >>>> > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) >>>> > >>>> > for w2 in owindows, r2 in resources >>>> > wrtuple.w = w2 >>>> > wrtuple.r = r2 >>>> > over = linearizeOverflow(w2, r2, ni) >>>> > total = total - resource_fines[wrtuple]*over >>>> > end >>>> > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) >>>> + >>>> > flight_vals[fi] ))) >>>> > @addConstraint(m, sc[c] <= total) >>>> > end >>>> > end >>>> > >>>> > Following are the stats for piece this code. Stats were recorded >>>> using >>>> > @time. >>>> > >>>> > For 1 item in the flights array, it takes about 620KB to execute the >>>> > performloop method - peak memory consumption by the program is >>>> 8.84GBs. >>>> > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. >>>> > 3 flights - 3.45MB - 9.60GBs >>>> > 4 flights - 4.35MB - 10.24GBs >>>> > 5 flights - 10.8MB - 15.63GBs >>>> > >>>> > It'd be great if someone could help me with this asap! >>>> > >>>> > Thanks. >>>> >>>> >>>>
Re: [julia-users] High memory consumption with loops
I've also tried to @profile, however, julia libraries seem to be executed the most. Venil On Wednesday, August 24, 2016 at 11:16:53 PM UTC-7, Venil Noronha wrote: > > I've just been able to @time the loop so far to see allocations at > iteration level. I haven't yet tried @code_warntype; I'll probably do that > next and see if I can get somewhere. > > Thanks, > Venil > > On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote: >> >> What have you tried so far? See http://docs.julialang.org/en/latest/manual/ >> >> performance-tips/#tools >> <http://docs.julialang.org/en/latest/manual/performance-tips/#tools>, >> esp. @code_warntype. >> >> --Tim >> >> On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: >> > The following code seems to consume a lot of memory. Could anyone spot >> what >> > I'm doing wrong here? I'm also using the JuMP library. >> > >> > function initUtilityConstraint() >> > c = categories[1] >> > me = attack_methods[1] >> > t = teams[1] >> > tuple = Simulate.cmt(c, me, t) >> > w = windows[1] >> > r = resources[1] >> > wrtuple = Simulate.wr(w, r) >> > for ni in 1:size(list,1), c in categories,* f in flights* >> > performloop(ni, c, f, tuple, wrtuple) >> > end >> > end >> > >> > function performloop(ni, c, f, tuple, wrtuple) >> > fi = findfirst(flights, f) >> > for w in windows, me in attack_methods >> > tuple.c = c >> > tuple.m = me >> > total = 0.0 >> > for t in teams >> > tuple.t = t >> > strat = linearizeStrategyS(f, c, t, w, ni) >> > total = total + effectiveness[tuple]*strat >> > end >> > >> > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) >> > >> > for w2 in owindows, r2 in resources >> > wrtuple.w = w2 >> > wrtuple.r = r2 >> > over = linearizeOverflow(w2, r2, ni) >> > total = total - resource_fines[wrtuple]*over >> > end >> > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + >> > flight_vals[fi] ))) >> > @addConstraint(m, sc[c] <= total) >> > end >> > end >> > >> > Following are the stats for piece this code. Stats were recorded using >> > @time. >> > >> > For 1 item in the flights array, it takes about 620KB to execute the >> > performloop method - peak memory consumption by the program is 8.84GBs. >> > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. >> > 3 flights - 3.45MB - 9.60GBs >> > 4 flights - 4.35MB - 10.24GBs >> > 5 flights - 10.8MB - 15.63GBs >> > >> > It'd be great if someone could help me with this asap! >> > >> > Thanks. >> >> >>
Re: [julia-users] High memory consumption with loops
I've just been able to @time the loop so far to see allocations at iteration level. I haven't yet tried @code_warntype; I'll probably do that next and see if I can get somewhere. Thanks, Venil On Wednesday, August 24, 2016 at 5:37:17 PM UTC-7, Tim Holy wrote: > > What have you tried so far? See http://docs.julialang.org/en/latest/manual/ > > performance-tips/#tools > <http://docs.julialang.org/en/latest/manual/performance-tips/#tools>, > esp. @code_warntype. > > --Tim > > On Wednesday, August 24, 2016 1:55:23 PM CDT Venil Noronha wrote: > > The following code seems to consume a lot of memory. Could anyone spot > what > > I'm doing wrong here? I'm also using the JuMP library. > > > > function initUtilityConstraint() > > c = categories[1] > > me = attack_methods[1] > > t = teams[1] > > tuple = Simulate.cmt(c, me, t) > > w = windows[1] > > r = resources[1] > > wrtuple = Simulate.wr(w, r) > > for ni in 1:size(list,1), c in categories,* f in flights* > > performloop(ni, c, f, tuple, wrtuple) > > end > > end > > > > function performloop(ni, c, f, tuple, wrtuple) > > fi = findfirst(flights, f) > > for w in windows, me in attack_methods > > tuple.c = c > > tuple.m = me > > total = 0.0 > > for t in teams > > tuple.t = t > > strat = linearizeStrategyS(f, c, t, w, ni) > > total = total + effectiveness[tuple]*strat > > end > > > > total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) > > > > for w2 in owindows, r2 in resources > > wrtuple.w = w2 > > wrtuple.r = r2 > > over = linearizeOverflow(w2, r2, ni) > > total = total - resource_fines[wrtuple]*over > > end > > # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + > > flight_vals[fi] ))) > > @addConstraint(m, sc[c] <= total) > > end > > end > > > > Following are the stats for piece this code. Stats were recorded using > > @time. > > > > For 1 item in the flights array, it takes about 620KB to execute the > > performloop method - peak memory consumption by the program is 8.84GBs. > > 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. > > 3 flights - 3.45MB - 9.60GBs > > 4 flights - 4.35MB - 10.24GBs > > 5 flights - 10.8MB - 15.63GBs > > > > It'd be great if someone could help me with this asap! > > > > Thanks. > > >
[julia-users] High memory consumption with loops
The following code seems to consume a lot of memory. Could anyone spot what I'm doing wrong here? I'm also using the JuMP library. function initUtilityConstraint() c = categories[1] me = attack_methods[1] t = teams[1] tuple = Simulate.cmt(c, me, t) w = windows[1] r = resources[1] wrtuple = Simulate.wr(w, r) for ni in 1:size(list,1), c in categories,* f in flights* performloop(ni, c, f, tuple, wrtuple) end end function performloop(ni, c, f, tuple, wrtuple) fi = findfirst(flights, f) for w in windows, me in attack_methods tuple.c = c tuple.m = me total = 0.0 for t in teams tuple.t = t strat = linearizeStrategyS(f, c, t, w, ni) total = total + effectiveness[tuple]*strat end total = ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ) for w2 in owindows, r2 in resources wrtuple.w = w2 wrtuple.r = r2 over = linearizeOverflow(w2, r2, ni) total = total - resource_fines[wrtuple]*over end # println(string( sc[c], "<=", ( total*(flight_vals[fi]*(-1)) + flight_vals[fi] ))) @addConstraint(m, sc[c] <= total) end end Following are the stats for piece this code. Stats were recorded using @time. For 1 item in the flights array, it takes about 620KB to execute the performloop method - peak memory consumption by the program is 8.84GBs. 2 flights - 1.02MB per iteration of performloop - peak 8.88GBs. 3 flights - 3.45MB - 9.60GBs 4 flights - 4.35MB - 10.24GBs 5 flights - 10.8MB - 15.63GBs It'd be great if someone could help me with this asap! Thanks.