How interesting! Thank you for that.
On Sat, Apr 29, 2017 at 4:04 PM Bryan Henderson <bry...@giraffe-data.com>
wrote:
> A few months ago, I posted here asking why the Ceph program takes so much
> memory (virtual, real, and address space) for what seems to be a simple
> task.
> Nobody knew, but I have done extensive research and I have the answer now,
> and
> thought I would publish it here.
>
> All it takes to do a Ceph "status" command is to create a TCP connection to
> the monitor, do a small login handshake, send a JSON document that says
> "status command", and receive and print the text response. This could be
> done
> in 64K with maybe a few megabytes of additional address space for the
> shared
> C library.
>
> If you do it with a 'ceph status' command, though, in the Hammer release it
> has a 700M peak address space usage (though it varies a lot from one run to
> the next) and uses 60M of real memory.
>
> The reason for this is that the Ceph program uses facilities in the
> librados
> library that are meant for much more than just performing a command. These
> facilities are meant to be used by a full-blown server that is a Ceph
> client.
> The facilities deal with locating a monitor within a cluster and failing
> over
> when that monitor dies; they interpret the Ceph configuration file and
> adjust
> dynamically when that file changes; they do logging; and more. When you
> type
> 'ceph status', you are building a sophisticated command-issuing machine,
> having it issue one command, and then tearing it down.
>
> 'ceph creates about 20 threads. They are asynchronous enough that in some
> runs, multiple threads exist at the same time and in other ones, they exist
> serially. This is why peak memory usage varies from one run to the next.
> In
> its quiescent state, ready to perform a command, the program has 13 threads
> standing by for various purposes. Each of these has 8M of virtual memory
> reserved for its stack and most have 64M for a heap.
>
> Finally, there is a lock auditor facility ("lockdep") that watches for
> locks
> being acquired out of order, as evidence of a bug in the code. This
> facility
> is not optional; it is always there. To keep track of all the locking, it
> sets up a 2000x2000 array (2000 being an upper limit on the number of locks
> the program might contain). That's 32M of real memory. I read in a forum
> that this has been greatly reduced in later releases.
>
>
> I was able to reduce the usage to 130M address space and 9M of real memory,
> while still using most of the same librados code to do the work, by
> creating a
> stripped-down version of the librados 'MonClient' class, setting the
> maximum
> thread stack size to 1M with rlimits, and making the threads share a heap
> with
> the MALLOC_ARENA_MAX environment variable. I also disabled lockdep.
>
>
> I just thought this might be interesting to someone searching the archives
> for
> memory usage information.
>
>
> --
> Bryan Henderson San Jose, California
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
