On Mon, 7 Jan 2019 at 21:04, Patrick Donnelly <pdonn...@redhat.com> wrote:
> Hello Mahmoud, > > On Fri, Dec 21, 2018 at 7:44 AM Mahmoud Ismail > <mahmoudahmedism...@gmail.com> wrote: > > I'm doing benchmarks for metadata operations on CephFS, HDFS, and HopsFS > on Google Cloud. In my current setup, i'm using 32 vCPU machines with 29 GB > memory, and i have 1 MDS, 1 MON and 3 OSDs. The MDS and the MON nodes are > co-located on one vm, while each of the OSDs is on a separate vm with 1 SSD > disk attached. I'm using the default configuration for MDS, and OSDs. > > > > I'm running 300 clients on 10 machines (16 vCPU), each client creates a > CephFileSystem using the CephFS hadoop plugin, and then writes empty files > for 30 seconds followed by reading the empty files for another 30 seconds. > The aggregated throughput is around 2000 file create opertions/sec and > 10000 file read operations/sec. However, the MDS is not fully utilizing the > 32 cores on the machine, is there any configuration that i should consider > to fully utilize the machine?. > > The MDS is not yet very parallel; it can only utilize about 2.5 cores > in the best circumstances. Make sure you allocate plenty of RAM for > the MDS. 16GB or 32GB would be a good choice. See (and disregard the > warning on that page): > http://docs.ceph.com/docs/mimic/cephfs/cache-size-limits/ > > You may also try using multiple active metadata servers to increase > throughput. See: http://docs.ceph.com/docs/mimic/cephfs/multimds/ How to often does the dynamic subtree paritioning kicks in? Can we control this interval? <http://docs.ceph.com/docs/mimic/cephfs/multimds/> > > Also, i noticed that running more than 20-30 clients (on different > threads) per machine degrade the aggregated throughput for read, is there a > limitation on CephFileSystem and libceph on the number of clients created > per machine? > > No. Can't give you any hints without more information about the test > setup. We also have not tested with the Hadoop plugin in years. There > may be limitations we're not presently aware of. On each machine, i’m running a simple java code that creates 30 CephFileSystems using the hadoop file system interface (hadoop plugin) and then on each thread, i’m doing write and then read operations on empty files in a loop. > > Another issue, Are the MDS operations single threaded as pointed here " > https://www.slideshare.net/XiaoxiChen3/cephfs-jewel-mds-performance-benchmark > "? > > Yes, this is still the case. > > > Regarding the MDS global lock, is it it a single lock per MDS or is it a > global distributed lock for all MDSs? > > per-MDS > > > -- > Patrick Donnelly >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
