We use tcmalloc in Ceph and at least on Debian it does free back to the system. However, it doesn't necessarily free all memory back to the system right away -- it retains free memory in per-thread pools so it can quickly allocate without going through a global lock, plus a bit more so it doesn't need to go to the system as often. This behavior has some pretty extensive customization options if you go through the documentation and I believe it checks the user environment variables so if you like this is something you can experiment with without any code changes. -Greg
On Apr 16, 4:00 am, Andy <[email protected]> wrote: > Hi, > > depending on the tcmalloc port, probably tcmalloc never free memory > back to os, because it was initially designed for server applications. > See TCMalloc_SystemRelease on the port your using. There are other > derivates of tcmalloc available which do free memory (e.g WebKit or > Chromium) or at least decommit allocates pages. Other allocators like > hoard (http://www.hoard.org/) use a different strategy of freeing > pages back to the os. > > -Andy > > On Apr 16, 6:51 am, Doug Judd <[email protected]> wrote: > > > > > > > > > Hi Pan, > > > [I'm CC'ing the list so others can benefit as well.] When free() is called, > > the memory is given back to the memory allocator (which may or may not give > > it back to the system right away). In the case of Hypertable, we link > > against tcmalloc <http://goog-perftools.sourceforge.net/doc/tcmalloc.html>. > > tcmalloc does not give the memory immediately back to the system. To > > verify this, I wrote a simple test program which I've attached. If you > > compile with tcmalloc, run it and wait for it enter poll(), you'll see that > > the RSS does not decrease: > > > $ g++ -o foo foo.cc -ltcmalloc > > $ ./foo 5 > > $ ps auxww | fgrep foo > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > doug 24076 54.4 32.0 5275268 5259260 pts/0 S+ 21:18 0:08 ./foo 5 > > > I tried this with glibc malloc and apparently it does give memory back to > > the system after it is freed: > > > $ g++ -o foo foo.cc > > $ ./foo 5 > > $ ps auxww | fgrep foo > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > doug 24292 64.6 0.0 11380 888 pts/0 S 21:37 0:11 ./foo 5 > > > So it it looks like this behavior is specific to tcmalloc. I tried to > > verify if the RSS would decrease if I ran another instance of foo while the > > other still existed, but there wasn't enough swap configured on the system, > > so the second time I ran it it dumped core with "bad alloc". However, I > > believe if the system did have enough swap space configured, it would swap > > out the first instance of foo to make room for the second instance thereby > > reducing the RSS of the first instance. > > > - DougOn Fri, Apr 15, 2011 at 7:10 PM, pan yue <[email protected]> > > wrote: > > > Does rangerserver alloc and free memory in it's own memory pool ? > > > I see PageArena.h: > > > 37 struct DefaultPageAllocator > > > { > > > > 38 void *allocate(size_t sz) { return std::malloc(sz); > > > } > > > 39 void deallocate(void *p) { std::free(p); > > > } > > > > 40 void freed(size_t sz) { /* mostly for effcient bulk stat reporting > > > */ > > > } > > > 41 }; > > > PageArena will free the memory to Linux system, don't manager the unused > > > memory by itself, according to this implementation, the RSS will drop to > > > a > > > low-level threshold when no insert or scan activities. > > > > 2011/4/16 Doug Judd <[email protected]> > > > >> This is not a problem. You've given the RangeServer 60% of memory which > > >> comes out to about 10GB. When the RangeServer frees memory it merely > > >> gives > > >> it back to the heap. If there is nothing going on on the system, the RSS > > >> won't change. If you run a different program that actively used a large > > >> amount of memory, only then would you see the RSS for the RangeServer > > >> shrink. > > > >> - Doug > > > >> On Fri, Apr 15, 2011 at 10:13 AM, pan yue <[email protected]> wrote: > > > >>> I use the second post: > > > >>> Hypertable.RangeServer.MemoryLimit.Percentage=60 > > > >>> The problem is RSS is always 8GB when no insert or scan happend, > > >>> According to log file: > > > >>> Memory Statistics (MB): VM=10259.90, RSS=8223.74, tracked=216.58, > > >>> computed=216.58 limit=8424.00 > > > >>> RangeServer only tracked 216MB. > > > >>> 2011/4/16 Doug Judd <[email protected]> > > > >>>> What is the config file that you're using? In your first post, you > > >>>> indicated that the config file contained the following lines: > > > >>>> Hypertable.RangeServer.MemoryLimit=4096000000 > > >>>> Hypertable.RangeServer.MemoryLimit.Percentage=25 > > > >>>> But in your second post, you showed a config file that contained: > > > >>>> Hypertable.RangeServer.MemoryLimit.Percentage=60 > > > >>>> If the RSS hits 8GB with a 25% memory limit, then that would indicate > > >>>> heap fragmentation. An RSS of 8GB with a 60% memory limit is normal > > >>>> and > > >>>> expected. > > > >>>> - Doug > > > >>>> P.S. Your client program looks ok to me. > > > >>>> On Fri, Apr 15, 2011 at 7:19 AM, pan yue <[email protected]>wrote: > > > >>>>> What I have seen is The RSS always 8GB for at least 5 hours after the > > >>>>> insert activity. > > >>>>> and it always 8GB for at lease 24 hours in another machine for having > > >>>>> the same problem which I tested yesterday. > > > >>>>> I don't know why, and How can I reduce heap fragmentation? > > >>>>> Did the HypertableClient I write have any problem? > > > >>>>> 2011/4/15 Doug Judd <[email protected]> > > > >>>>>> What you're seeing is heap fragmentation. During the heavy insert > > >>>>>> activity, the Memory usage grew up to the configured limit > > >>>>>> (Hypertable.RangeServer.MemoryLimit.Percentage=60). > > >>>>>> You can verify this by running the following command: > > > >>>>>> $ fgrep "Memory Usage" Hypertable.RangeServer.log > > >>>>>> [...] > > >>>>>> 1302844863 INFO Hypertable.RangeServer : > > >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) > > >>>>>> Memory Usage: 8519348352 bytes > > >>>>>> 1302844864 INFO Hypertable.RangeServer : > > >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) > > >>>>>> Memory Usage: 483976199 bytes > > >>>>>> 1302844864 INFO Hypertable.RangeServer : > > >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) > > >>>>>> Memory Usage: 534307847 bytes > > >>>>>> [...] > > > >>>>>> As you can see by the output, at Thu Apr 14 22:21:03 2011, the memory > > >>>>>> usage hit 8519348352 bytes. > > > >>>>>> Once the insert activity stopped, Hypertable started to aggressively > > >>>>>> compact in-memory CellCaches to on-disk CellStores in order to prune > > >>>>>> back > > >>>>>> the commit log (for faster recovery). Once a CellCache gets > > >>>>>> compacted, the > > >>>>>> memory gets freed and subtracted from the internal memory counter. > > >>>>>> In > > >>>>>> the 0.9.4.3 release, the minimum prune threshold for the commit log > > >>>>>> is 200M > > >>>>>> which is why the tracked memory number drops to ~200M. When memory > > >>>>>> gets > > >>>>>> freed, it is not immediately reflected in the RSS. If you were to > > >>>>>> run some > > >>>>>> other program after your Hypertable test, that used a lot of memory, > > >>>>>> I > > >>>>>> suspect you would see the Hypertable RSS shrink towards 200MB. > > > >>>>>> - Doug > > > >>>>>> On Thu, Apr 14, 2011 at 10:59 PM, pan yue > > >>>>>> <[email protected]>wrote: > > > >>>>>>> Thanks for Doug Judd 's reply. the following is what done when > > >>>>>>> RangerServer leak memory. > > > >>>>>>> 1. Deployment: I deployed hypertable-0.9.34 on one machine(16GB > > >>>>>>> memory, 8 cores). > > >>>>>>> ./start-dfsbroker.sh hadoop > > >>>>>>> ./start-hyperspace.sh > > >>>>>>> ./start-master.sh > > >>>>>>> ./start-rangerserver.sh > > > >>>>>>> 2. test insert data: one c++ hypertable client, single thread, here > > >>>>>>> is the test result: > > >>>>>>> ./pressClient 0 1 1000000 > output.txt > > > >>>>>>> TotalBytes: 65266475584B > > >>>>>>> TotalSeconds: 3285.826 > > >>>>>>> QPS: 18.943MB/S > > > >>>>>>> table schema: create table foo (name, age, sex); > > > >>>>>>> 3. hypertable.cfg > > >>>>>>> # HDFS Broker > > >>>>>>> HdfsBroker.Port=38030 > > >>>>>>> HdfsBroker.fs.default.name=hdfs://dev1:9000 > > >>>>>>> HdfsBroker.Workers=20 > > >>>>>>> # Hyperspace > > >>>>>>> Hyperspace.Replica.Host=localhost > > >>>>>>> Hyperspace.Replica.Port=38040 > > >>>>>>> Hyperspace.Replica.Dir=hyperspace > > >>>>>>> Hyperspace.Replica.Workers=20 > > > >>>>>>> # Hypertable.Master > > >>>>>>> Hypertable.Master.Host=localhost > > >>>>>>> Hypertable.Master.Port=38050 > > >>>>>>> Hypertable.Master.Workers=20 > > > >>>>>>> # Hypertable.RangeServer > > >>>>>>> Hypertable.RangeServer.Port=38060 > > >>>>>>> Hypertable.RangeServer.MemoryLimit.Percentage=60 > > >>>>>>> Hypertable.RangeServer.CommitLog.PruneThreshold.Min=50000000 > > >>>>>>> Hypertable.RangeServer.CommitLog.PruneThreshold.Max=100000000 > > >>>>>>> Hypertable.RangeServer.Timer.Interval=10000 > > >>>>>>> Hypertable.RangeServer.Maintenance.Interval=15000 > > > >>>>>>> Hyperspace.KeepAlive.Interval=30000 > > >>>>>>> Hyperspace.Lease.Interval=1000000 > > >>>>>>> Hyperspace.GracePeriod=200000 > > > >>>>>>> 4. memory status: (from Hypertable.RangerServer.log) > > > >>>>>>> STAT ***** MaintenanceScheduler::schedule() ***** > > >>>>>>> STAT revision_root 1302839066478990001 > > >>>>>>> STAT revision_metadata 1302839108088203001 > > >>>>>>> STAT revision_user 1302844927547261001 > > >>>>>>> STAT 0/0[..0/0: ](location) cumulative_size 137 <= prune_threshold > > >>>>>>> 50000000 > > >>>>>>> STAT 0/0[0/0: .. ](default) cumulative_size 28635 <= > > >>>>>>> prune_threshold 50000000 > > >>>>>>> STAT 0/0[0/0: ](location) cumulative_size 28635 <= prune_threshold > > >>>>>>> 50000000 > > >>>>>>> STATS user log prune threshold 50000000 > > >>>>>>> STAT 1/0[..row_key_of_302072](default) cumulative_size 40819432 <= > > >>>>>>> prune_threshold 50000000 > > >>>>>>> STAT 1/0[row_key_of_74178.. ](default) cumulative_size 40819432 <= > > >>>>>>> prune_threshold 50000000 > > >>>>>>> 1302845608 INFO Hypertable.RangeServer : > > >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) > > >>>>>>> Memory Usage: 216582972 bytes > > >>>>>>> 1302845618 INFO Hypertable.RangeServer : > > >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:84) > > >>>>>>> Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0) > > >>>>>>> 1302845618 INFO Hypertable.RangeServer :... > > read more » -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
