Le 19/06/2018 à 09:02, Dan van der Ster a écrit :
The storage arrays are Nexsan E60 arrays having two active-active redundant
controllers, 60 3 TB disk drives. The disk drives are organized into six 8+2
Raid 6 LUNs of 24 TB each.
This is not the ideal Ceph hardware. Ceph is designed to use
Thanks for the advice, Dan.
I'll try to reconfigure the cluster and see if the performance changes.
Best,
Jialin
On Tue, Jun 19, 2018 at 12:02 AM Dan van der Ster
wrote:
> On Tue, Jun 19, 2018 at 1:04 AM Jialin Liu wrote:
> >
> > Hi Dan, Thanks for the follow-ups.
> >
> > I have just tried
Hi Dan, Thanks for the follow-ups.
I have just tried running multiple librados MPI applications from multiple
nodes, it does show increased bandwidth,
with ceph -w, I observed as high as 500MB/sec (previously only 160MB/sec ),
I think I can do finer tuning by
coordinating more concurrent
Hi,
Have you tried running rados bench in parallel from several client
machines? That would demonstrate the full BW capacity of the cluster.
e.g. make a test pool with e.g. 256 PGs (which will average 16 per OSD
on your cluster).
Then from several clients at once do `rados bench -p test 60
Hi, To make the the problem clearer, here is the configuration of the
cluster:
The 'problem' I have is the low bandwidth no matter how I increase the
concurrency.
I have tried using MPI to launch 322 processes, each calling librados to
create a handle and initialize the io context, and write one
Thank you Dan. I’ll try it.
Best,
Jialin
NERSC/LBNL
> On Jun 18, 2018, at 12:22 AM, Dan van der Ster wrote:
>
> Hi,
>
> One way you can see exactly what is happening when you write an object
> is with --debug_ms=1.
>
> For example, I write a 100MB object to a test pool: rados
> --debug_ms=1
Hi,
One way you can see exactly what is happening when you write an object
is with --debug_ms=1.
For example, I write a 100MB object to a test pool: rados
--debug_ms=1 -p test put 100M.dat 100M.dat
I pasted the output of this here: https://pastebin.com/Zg8rjaTV
In this case, it first gets the
Sorry about the misused term 'OSS: object storage server' (a term often
used in Lustre filesystem), what I meant is 4 hosts, each manages 12 OSDs.
Thanks for anyone who may answer any of my questions.
Best,
Jialin
NERSC/LBNL
On Sun, Jun 17, 2018 at 11:29 AM Jialin Liu wrote:
> Hello,
>
> I
Hello,
I have a couple questions regarding the IO on OSD via librados.
1. How to check which osd is receiving data?
2. Can the write operation return immediately to the application once the
write to the primary OSD is done? or does it return only when the data is
replicated twice? (size=3)
3.
