On Tue, Oct 3, 2017 at 3:15 AM Alexander Kushnirenko <kushnire...@gmail.com> wrote:
> Hello, > > I'm working on third party code (Bareos Storage daemon) which gives very > low write speeds for CEPH. The code was written to demonstrate that it is > possible, but the speed is about 3-9 MB/s which is too slow. I modified > the routine to use rados_aio_write instead of rados_write, and was able to > backup/restore data successfully with the speed about 30MB/s, which what I > would expect on 1GB/s network and rados bench results. I studied examples > in the documents and github, but still I'm afraid that by code is working > merely by accident. Could some one comment on the following questions: > > Q1. Storage daemon sends write requests of 64K size, so current code works > like this: > > rados_write(....., buffer, len=64K, offset=0) > rados_write(....., buffer, len=64K, offset=64K) > rados_write(....., buffer, len=64K, offset=128K) > ... and so on ... > > What is the correct way to use AIO (to use one completion or several?) > Version 1: > > rados_aio_create_completion(NULL, NULL, NULL, &comp); > rados_aio_write(....., comp, buffer, len=64K, offset=0) > rados_aio_write(....., comp, buffer, len=64K, offset=64K) > rados_aio_write(....., comp, buffer, len=64K, offset=128K) > rados_aio_wait_for_complete(comp); // wait for Async IO in memory > rados_aio_wait_for_safe(comp); // and on disk > rados_aio_release(comp); > > Version 2: > rados_aio_create_completion(NULL, NULL, NULL, &comp1); > rados_aio_create_completion(NULL, NULL, NULL, &comp2); > rados_aio_create_completion(NULL, NULL, NULL, &comp3); > rados_aio_write(....., comp1, buffer, len=64K, offset=0) > rados_aio_write(....., comp2, buffer, len=64K, offset=64K) > rados_aio_write(....., comp3, buffer, len=64K, offset=128K) > rados_aio_wait_for_complete(comp1); > rados_aio_wait_for_complete(comp2); > rados_aio_wait_for_complete(comp3); > rados_aio_write(....., comp1, buffer, len=64K, offset=192K) > rados_aio_write(....., comp2, buffer, len=64K, offset=256K) > rados_aio_write(....., comp3, buffer, len=64K, offset=320K) > ..... > Each operation needs its own completion. If you give them the same one, things will go very badly. > > Q2. Problem of maximum object size. When I use rados_write I get an > error when I exceed maximum object size (132MB in luminous). But when I > use rados_aio_write it happily goes beyond the limit of object, but > actually writes nothing, but does not make any error. Is there a way to > catch such situation? > > Each completion gets a response code included. You are apparently not actually looking at them; you should do so!
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
