Thanks for the sample code Seth, you're right. This is a different issue entirely, and I can replicate your issue. Good find. -E
On Wed, Nov 5, 2014 at 9:15 PM, Amit Murthy <amit.mur...@gmail.com> wrote: > Issue created : https://github.com/JuliaLang/julia/issues/8912 > > > > > On Thu, Nov 6, 2014 at 7:56 AM, Seth Yuan <sethy...@gmail.com> wrote: > >> I don't think it's an OSX only issue, as I have duplicated the issue on a >> Centos too. >> >> Perhaps you guys can run the demo code I gave, test it on your own >> machines to better appreciate the problem. >> >> Here a modified code without module dependency. >> >> function test_allocation() >> a = drand(10000, 10000) >> b = map(x->x+one(x), a) >> readline() >> end >> >> test_allocation() >> @everywhere gc() >> @everywhere Base.flush_gc_msgs() >> @everywhere gc() >> readline() >> >> >> On Thursday, November 6, 2014 10:05:21 AM UTC+8, Elliot Saba wrote: >>> >>> @Stefan, they do, but we've had a longstanding issue >>> <https://github.com/JuliaLang/julia/issues/1339> regarding whether >>> we're releasing it as aggressively as we might need to in order to really >>> shrink the number of pages we've got allocated. >>> >>> In short; calling gc() may not actually reduce the "Memory" column on >>> OSX, however once your process starts to page to disk, OSX seems to come to >>> its wits and start unmapping pages. >>> -E >>> >>> On Wed, Nov 5, 2014 at 5:58 PM, Seth Yuan <seth...@gmail.com> wrote: >>> >>>> I tried this as you recommended, sadly, it doesn't do better. :( >>>> >>>> On Wednesday, November 5, 2014 4:48:10 PM UTC+8, Amit Murthy wrote: >>>>> >>>>> Could you try: >>>>> >>>>> @everywhere gc() >>>>> @everywhere Base.flush_gc_msgs() >>>>> @everywhere gc() >>>>> >>>>> >>>>> >>>>> On Wed, Nov 5, 2014 at 2:07 PM, Seth Yuan <seth...@gmail.com> wrote: >>>>> >>>>>> Hi Amit, >>>>>> >>>>>> Thanks for the response, I tried Base.flush_gc_msgs(), the result is >>>>>> shown below: >>>>>> >>>>>> >>>>>> <https://lh5.googleusercontent.com/-bEVLUYu-DPk/VFnhFUh28CI/AAAAAAAABk0/WNl3ukSetUk/s1600/mem.png> >>>>>> It did free some memory on some workers, but not quite. I think the >>>>>> acceptable number for workers would be around 100MB (like the master is). >>>>>> >>>>>> So what's wrong with DArrays? >>>>>> >>>>>> >>>>>> On Wednesday, November 5, 2014 2:32:42 PM UTC+8, Amit Murthy wrote: >>>>>>> >>>>>>> Distributed objects are currently gc'ed asynchronously. >>>>>>> >>>>>>> You may force this using an internal function flush_gc_msgs like >>>>>>> this: >>>>>>> >>>>>>> @everywhere Base.flush_gc_msgs() >>>>>>> @everywhere gc() >>>>>>> >>>>>>> >>>>>>> The above is an internal function and is not meant for general use. >>>>>>> It will be great if you could open an issue on github requesting a >>>>>>> feature >>>>>>> to force gc of distributed objects. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 5, 2014 at 9:01 AM, Seth Yuan <seth...@gmail.com> wrote: >>>>>>> >>>>>>>> I did a test on deallocations, and the behavior is not what I >>>>>>>> expected. >>>>>>>> >>>>>>>> The code I ran is: >>>>>>>> >>>>>>>> @everywhere using Ops >>>>>>>> >>>>>>>> function test_allocation() >>>>>>>> a = drand(10000, 10000) >>>>>>>> b = 2a >>>>>>>> readline() >>>>>>>> end >>>>>>>> >>>>>>>> test_allocation() >>>>>>>> @everywhere gc() >>>>>>>> readline() >>>>>>>> >>>>>>>> Ops module defined the '*' operator (where a new DArray is >>>>>>>> created). I can see that master's memory went down after the gc() >>>>>>>> call, but >>>>>>>> the workers' memory remained unchanged. >>>>>>>> >>>>>>>> >>>>>>>> <https://lh5.googleusercontent.com/-qo7SKwkoya0/VFmZrIWL0EI/AAAAAAAABkk/FctwkEcQWnk/s1600/mem.png> >>>>>>>> So the question is: "how to really free a DArray? Both master and >>>>>>>> workers!" >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, October 16, 2014 10:18:42 PM UTC+8, Amit Murthy wrote: >>>>>>>>> >>>>>>>>> meant to say "all references from all processes are removed" >>>>>>>>> >>>>>>>>> On Thu, Oct 16, 2014 at 7:47 PM, Amit Murthy <amit....@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, it is gc'ed just like any other object, when all references >>>>>>>>>> to it from any process is removed. >>>>>>>>>> >>>>>>>>>> On Thu, Oct 16, 2014 at 2:59 PM, Seth Yuan <seth...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Now, I know by reading the documentation how a DArray is >>>>>>>>>>> created, but I'm not sure when a DArray is deallocated. >>>>>>>>>>> >>>>>>>>>>> Is it GC able when the master looses all references of it? Does >>>>>>>>>>> somebody know the answer to this question? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >
