Hi @Amit, @Elliot and others, Thanks for assisting me in this issue, and let's hope it'd be fixed soon.
On Thursday, November 6, 2014 2:34:17 PM UTC+8, Elliot Saba wrote: > > Thanks for the sample code Seth, you're right. This is a different issue > entirely, and I can replicate your issue. Good find. > -E > > On Wed, Nov 5, 2014 at 9:15 PM, Amit Murthy <amit....@gmail.com > <javascript:>> wrote: > >> Issue created : https://github.com/JuliaLang/julia/issues/8912 >> >> >> >> >> On Thu, Nov 6, 2014 at 7:56 AM, Seth Yuan <seth...@gmail.com >> <javascript:>> wrote: >> >>> I don't think it's an OSX only issue, as I have duplicated the issue on >>> a Centos too. >>> >>> Perhaps you guys can run the demo code I gave, test it on your own >>> machines to better appreciate the problem. >>> >>> Here a modified code without module dependency. >>> >>> function test_allocation() >>> a = drand(10000, 10000) >>> b = map(x->x+one(x), a) >>> readline() >>> end >>> >>> test_allocation() >>> @everywhere gc() >>> @everywhere Base.flush_gc_msgs() >>> @everywhere gc() >>> readline() >>> >>> >>> On Thursday, November 6, 2014 10:05:21 AM UTC+8, Elliot Saba wrote: >>>> >>>> @Stefan, they do, but we've had a longstanding issue >>>> <https://github.com/JuliaLang/julia/issues/1339> regarding whether >>>> we're releasing it as aggressively as we might need to in order to really >>>> shrink the number of pages we've got allocated. >>>> >>>> In short; calling gc() may not actually reduce the "Memory" column on >>>> OSX, however once your process starts to page to disk, OSX seems to come >>>> to >>>> its wits and start unmapping pages. >>>> -E >>>> >>>> On Wed, Nov 5, 2014 at 5:58 PM, Seth Yuan <seth...@gmail.com> wrote: >>>> >>>>> I tried this as you recommended, sadly, it doesn't do better. :( >>>>> >>>>> On Wednesday, November 5, 2014 4:48:10 PM UTC+8, Amit Murthy wrote: >>>>>> >>>>>> Could you try: >>>>>> >>>>>> @everywhere gc() >>>>>> @everywhere Base.flush_gc_msgs() >>>>>> @everywhere gc() >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 5, 2014 at 2:07 PM, Seth Yuan <seth...@gmail.com> wrote: >>>>>> >>>>>>> Hi Amit, >>>>>>> >>>>>>> Thanks for the response, I tried Base.flush_gc_msgs(), the result is >>>>>>> shown below: >>>>>>> >>>>>>> >>>>>>> <https://lh5.googleusercontent.com/-bEVLUYu-DPk/VFnhFUh28CI/AAAAAAAABk0/WNl3ukSetUk/s1600/mem.png> >>>>>>> It did free some memory on some workers, but not quite. I think the >>>>>>> acceptable number for workers would be around 100MB (like the master >>>>>>> is). >>>>>>> >>>>>>> So what's wrong with DArrays? >>>>>>> >>>>>>> >>>>>>> On Wednesday, November 5, 2014 2:32:42 PM UTC+8, Amit Murthy wrote: >>>>>>>> >>>>>>>> Distributed objects are currently gc'ed asynchronously. >>>>>>>> >>>>>>>> You may force this using an internal function flush_gc_msgs like >>>>>>>> this: >>>>>>>> >>>>>>>> @everywhere Base.flush_gc_msgs() >>>>>>>> @everywhere gc() >>>>>>>> >>>>>>>> >>>>>>>> The above is an internal function and is not meant for general use. >>>>>>>> It will be great if you could open an issue on github requesting a >>>>>>>> feature >>>>>>>> to force gc of distributed objects. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 5, 2014 at 9:01 AM, Seth Yuan <seth...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I did a test on deallocations, and the behavior is not what I >>>>>>>>> expected. >>>>>>>>> >>>>>>>>> The code I ran is: >>>>>>>>> >>>>>>>>> @everywhere using Ops >>>>>>>>> >>>>>>>>> function test_allocation() >>>>>>>>> a = drand(10000, 10000) >>>>>>>>> b = 2a >>>>>>>>> readline() >>>>>>>>> end >>>>>>>>> >>>>>>>>> test_allocation() >>>>>>>>> @everywhere gc() >>>>>>>>> readline() >>>>>>>>> >>>>>>>>> Ops module defined the '*' operator (where a new DArray is >>>>>>>>> created). I can see that master's memory went down after the gc() >>>>>>>>> call, but >>>>>>>>> the workers' memory remained unchanged. >>>>>>>>> >>>>>>>>> >>>>>>>>> <https://lh5.googleusercontent.com/-qo7SKwkoya0/VFmZrIWL0EI/AAAAAAAABkk/FctwkEcQWnk/s1600/mem.png> >>>>>>>>> So the question is: "how to really free a DArray? Both master and >>>>>>>>> workers!" >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thursday, October 16, 2014 10:18:42 PM UTC+8, Amit Murthy wrote: >>>>>>>>>> >>>>>>>>>> meant to say "all references from all processes are removed" >>>>>>>>>> >>>>>>>>>> On Thu, Oct 16, 2014 at 7:47 PM, Amit Murthy <amit....@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Yes, it is gc'ed just like any other object, when all references >>>>>>>>>>> to it from any process is removed. >>>>>>>>>>> >>>>>>>>>>> On Thu, Oct 16, 2014 at 2:59 PM, Seth Yuan <seth...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Now, I know by reading the documentation how a DArray is >>>>>>>>>>>> created, but I'm not sure when a DArray is deallocated. >>>>>>>>>>>> >>>>>>>>>>>> Is it GC able when the master looses all references of it? Does >>>>>>>>>>>> somebody know the answer to this question? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> >
