Hi, so.. +1
We don't run compression as far as I know, so that wouldn't be it. We do actually run a mix of bluestore & filestore - due to the rest of the cluster predating a stable bluestore by some amount. The interesting part is - the behavior seems to be specific to our bluestore nodes. Below - yellow line, node with 10 x ~4TB SSDs, green line 8 x 800GB SSDs. Blue line - dump_mempools total bytes for all the OSDs running on the yellow line. The big dips - forced restarts after having suffered through after effects of letting linux deal with it by OOM->SIGKILL previously. A gross extrapolation - "right now" the "memory used" seems to be close enough to "sum of RSS of ceph-osd processes" running on the machines. -KJ On Thu, Mar 1, 2018 at 7:18 PM, Alex Gorbachev <a...@iss-integration.com> wrote: > On Thu, Mar 1, 2018 at 5:37 PM, Subhachandra Chandra > <schan...@grailbio.com> wrote: > > Even with bluestore we saw memory usage plateau at 3-4GB with 8TB drives > > filled to around 90%. One thing that does increase memory usage is the > > number of clients simultaneously sending write requests to a particular > > primary OSD if the write sizes are large. > > We have not seen a memory increase in Ubuntu 16.04, but I also > observed repeatedly the following phenomenon: > > When doing a VMotion in ESXi of a large 3TB file (this generates a log > of IO requests of small size) to a Ceph pool with compression set to > force, after some time the Ceph cluster shows a large number of > blocked requests and eventually timeouts become very large (to the > point where ESXi aborts the IO due to timeouts). After abort, the > blocked/slow requests messages disappear. There are no OSD errors. I > have OSD logs if anyone is interested. > > This does not occur when compression is unset. > > -- > Alex Gorbachev > Storcium > > > > > Subhachandra > > > > On Thu, Mar 1, 2018 at 6:18 AM, David Turner <drakonst...@gmail.com> > wrote: > >> > >> With default memory settings, the general rule is 1GB ram/1TB OSD. If > you > >> have a 4TB OSD, you should plan to have at least 4GB ram. This was the > >> recommendation for filestore OSDs, but it was a bit much memory for the > >> OSDs. From what I've seen, this rule is a little more appropriate with > >> bluestore now and should still be observed. > >> > >> Please note that memory usage in a HEALTH_OK cluster is not the same > >> amount of memory that the daemons will use during recovery. I have seen > >> deployments with 4x memory usage during recovery. > >> > >> On Thu, Mar 1, 2018 at 8:11 AM Stefan Kooman <ste...@bit.nl> wrote: > >>> > >>> Quoting Caspar Smit (caspars...@supernas.eu): > >>> > Stefan, > >>> > > >>> > How many OSD's and how much RAM are in each server? > >>> > >>> Currently 7 OSDs, 128 GB RAM. Max wil be 10 OSDs in these servers. 12 > >>> cores (at least one core per OSD). > >>> > >>> > bluestore_cache_size=6G will not mean each OSD is using max 6GB RAM > >>> > right? > >>> > >>> Apparently. Sure they will use more RAM than just cache to function > >>> correctly. I figured 3 GB per OSD would be enough ... > >>> > >>> > Our bluestore hdd OSD's with bluestore_cache_size at 1G use ~4GB of > >>> > total > >>> > RAM. The cache is a part of the memory usage by bluestore OSD's. > >>> > >>> A factor 4 is quite high, isn't it? Where is all this RAM used for > >>> besides cache? RocksDB? > >>> > >>> So how should I size the amount of RAM in a OSD server for 10 bluestore > >>> SSDs in a > >>> replicated setup? > >>> > >>> Thanks, > >>> > >>> Stefan > >>> > >>> -- > >>> | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 > >>> | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users@lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Kjetil Joergensen <kje...@medallia.com> SRE, Medallia Inc
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
