I passed it as an argument from the REPL. I made a test script where I hardcode these things to make sure its not global, but the allocation remains. In this gist there is the code with allocations and @time result for the double for loop as well as the vectorised version: https://gist.github.com/timlod/6a2324db4d01dc07dd5a
On Sunday, 13 March 2016 13:36:10 UTC+1, tshort wrote: > > Where does your variable `dims` come from? As pointed out above, global > variables can hurt type inference. > On Mar 13, 2016 8:29 AM, "Tim Loderhose" <ti...@hotmail.de <javascript:>> > wrote: > >> I implemented the suggestions (see updated gist: >> https://gist.github.com/timlod/0f607e311d0464fd6c63). >> The allocation in the for loops disappeared and the time required halved >> to ~3s. The mmap access though still allocated a lot. >> And the vectorised access is still much faster and allocates less >> (although I want to get rid of the allocation altogether). >> Any other ideas? >> >> On Saturday, 12 March 2016 15:15:51 UTC+1, Dan wrote: >>> >>> Yep, `peCounters`, `paCounters` and `dims` are not type-stable. They are >>> one type by their default values and then assigned another. Perhaps rename >>> the default parameters, and copy them to `peCounters`, `paCounters` and >>> `dims` only if they are set to something other than `0`. >>> >>> Also, `mdhcounters` might not return a definite type (need to check that >>> function). >>> Fixing these should make the loop efficient. >>> >>> On Saturday, March 12, 2016 at 4:05:39 PM UTC+2, Tim Loderhose wrote: >>>> >>>> Here's the actual code: >>>> https://gist.github.com/timlod/0f607e311d0464fd6c63 >>>> <https://www.google.com/url?q=https%3A%2F%2Fgist.github.com%2Ftimlod%2F0f607e311d0464fd6c63&sa=D&sntz=1&usg=AFQjCNHB4_Itk64Xevbo5RACmmrF0lsgBA> >>>> >>>> I am running the code from the REPL, may that be a problem? (As I read >>>> in the REPL everything is global). In the file nothing is global. >>>> Also, the counters are UInt16s, but that shouldnt matter I guess. >>>> >>>> Thanks for the help so far! >>>> >>>> On Saturday, 12 March 2016 14:22:38 UTC+1, Dan wrote: >>>>> >>>>> It's better to have code which actually runs in the post. In any case, >>>>> the allocations at the `for` lines is suspicious - the for should >>>>> basically >>>>> only allocate a counter. Are there any global variables? Is `counter1` or >>>>> `counter2` or `dims` global? Globals are always a good source of >>>>> confusion >>>>> to the type-inference engine. >>>>> >>>>> On Saturday, March 12, 2016 at 2:28:51 PM UTC+2, Tim Loderhose wrote: >>>>>> >>>>>> The code is in a function. I changed the names a bit to make it more >>>>>> understandable. The actual function is longer and has different variable >>>>>> names. >>>>>> >>>>>> On Saturday, 12 March 2016 13:01:28 UTC+1, tshort wrote: >>>>>>> >>>>>>> Is that code in a function? (It should be.) Also, one of your >>>>>>> variable names changed to `counter1s`. Suspect a type instability. >>>>>>> On Mar 12, 2016 4:12 AM, "Tim Loderhose" <ti...@hotmail.de> wrote: >>>>>>> >>>>>>>> I tried around with that a bit, but then it gets much worse: From >>>>>>>> ~1s to ~6s, allocation as shown: >>>>>>>> >>>>>>>> 153710487 mat = Array{Complex64}(dims...) >>>>>>>> 4722450 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>>> (dims[2],length(counter1))) >>>>>>>> 9568 for i = 1:dims[2] >>>>>>>> 4000 for j = 1:length(counter1) >>>>>>>> 1690462534 mat[counter1s[j],i,counter2[j]] = file[i,j] >>>>>>>> - end >>>>>>>> >>>>>>>> I swapped the for loops around here, but that didn't matter. I can >>>>>>>> gain a little bit by indexing i into the first dimension of mat, but >>>>>>>> it >>>>>>>> still lags far behind. >>>>>>>> Any other ideas? >>>>>>>> >>>>>>>> On Saturday, 12 March 2016 03:15:33 UTC+1, Greg Plowman wrote: >>>>>>>>> >>>>>>>>> I think array slices (on right hand side of assignment) create new >>>>>>>>> arrays, hence the allocation. >>>>>>>>> Try writing an explicit loop instead, something like: >>>>>>>>> >>>>>>>>> for j = 1:length(counter1) >>>>>>>>> for i = 1:size(file,1) >>>>>>>>> mat[counter1[j],i,counter2[j]] = file[i,j] >>>>>>>>> end >>>>>>>>> end >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, March 12, 2016 at 12:25:00 PM UTC+11, Tim Loderhose >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have a question regarding some allocation in my code I would >>>>>>>>>> like to get rid of. >>>>>>>>>> I am memory mapping a file (which could be very large) which is >>>>>>>>>> part of a complex 3D matrix, and then put its contents into the >>>>>>>>>> preallocated matrix along the second dimension. I need the counters >>>>>>>>>> because >>>>>>>>>> the contents of file are only a subset of the full matrix. >>>>>>>>>> >>>>>>>>>> Here's a profiled snippet, where the file which is loaded has >>>>>>>>>> 120619520 bytes. >>>>>>>>>> >>>>>>>>>> 153705063 mat = Array{Complex64}(dims...) >>>>>>>>>> 4721282 file = Mmap.mmap(filename, Array{Complex64,2}, >>>>>>>>>> (dims[2],length(counter1))) >>>>>>>>>> 16 for i = 1:length(counter1) >>>>>>>>>> 148179531 mat[counter1[i],:,counter2[i]] = file[:,i] >>>>>>>>>> - end >>>>>>>>>> >>>>>>>>>> Why does the code allocate so much memory inside the for-loop >>>>>>>>>> (even more bytes than the contents of file)? >>>>>>>>>> It seems like this is a trivial matter, right now I just can't >>>>>>>>>> get my head around it, any help is appreciated :) >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>
