Hi, Recently we have been investigating the behavior of the Hypertable 0.9.3.8 built with different memory allocators.
The test that we used involved loading data from multiple data files into a set of HT tables. We were primarily interested in the effect different allocators have on memory utilization by various HT processes. Here are the results as observed in top: *glibc* VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30.3g 24g 6368 S 0 38.6 35:59.52 Hypertable.Rang 8886m 12m 5736 S 0 0.0 0:03.28 Hypertable.Mast 1525m 7376 5908 S 0 0.0 0:00.08 ThriftBroker 2725m 5348 4084 S 0 0.0 0:01.62 Hyperspace.Mast *tcmalloc-minimal* VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26.6g 25g 4492 S 0 40.1 33:05.92 Hypertable.Rang 1350m 13m 3744 S 0 0.0 0:02.47 Hypertable.Mast 385m 6224 3444 S 0 0.0 0:00.06 ThriftBroker 369m 5728 2964 S 0 0.0 0:01.19 Hyperspace.Mast *jemalloc* VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32.2g 23g 6428 S 0 38.1 34:55.09 Hypertable.Rang 1595m 27m 5892 S 0 0.0 0:03.59 Hypertable.Mast 455m 7988 5996 S 0 0.0 0:00.08 ThriftBroker 515m 9896 4748 S 0 0.0 0:01.44 Hyperspace.Mast With jemalloc the resulting table is as it appeared right after the insertion rate has dropped to 0. Immediately after the snapshot was taken the CPU usage of Hypertable.Range has spiked to 90% and the reserved memory began to drop until I saw the following: VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32.2g 1.5g 6428 S 0 2.3 35:55.57 Hypertable.Rang Furthermore, at about a halfway point through the run I observed the reserved memory drop from 20g to 14g and then increase again to 23g. Now this test mimicked out production environment - a single threaded process pegged to a single CPU. We are planning a more intricate test in the near future. I'd be happy to hear any insights that people may have regarding using the HT with different allocators. Ilya Sorkin -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/hypertable-dev. For more options, visit https://groups.google.com/d/optout.
