On Mon, Jul 9, 2018 at 12:46 AM Zhou Choury <cho...@msn.cn> wrote:
> Hi all
> We mounted cephfs with ceph-fuse on two machines. We found that if a
> process writing a log on node A, while "tail -f" it on node B will quite
> slow, The mds server also complain like:
> > 2018-07-09 15:10:35.516602 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : 2 slow requests, 1 included below; oldest blocked for > 8.421551
> secs
> > 2018-07-09 15:10:35.516608 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 5.866578 seconds old, received at 2018-07-09
> 15:10:29.649997: client_request(client.3777088:24818 getattr pAsLsXsFs
> #1000008a41e 2018-07-09 15:10:03.842337) currently failed to rdlock, waiting
> > 2018-07-09 15:10:48.517367 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : 2 slow requests, 2 included below; oldest blocked for > 5.860196
> secs
> > 2018-07-09 15:10:48.517373 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 5.860196 seconds old, received at 2018-07-09
> 15:10:42.657139: client_request(client.3777088:24826 getattr pAsLsXsFs
> #1000008a41e 2018-07-09 15:10:16.843077) currently failed to rdlock, waiting
> > 2018-07-09 15:10:48.517375 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 5.622276 seconds old, received at 2018-07-09
> 15:10:42.895059: client_request(client.3775872:34689 lookup
> #10000011127/choury-test 2018-07-09 15:10:42.894941) currently failed to
> rdlock, waiting
> > 2018-07-09 15:10:51.517530 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : 2 slow requests, 1 included below; oldest blocked for > 8.622448
> secs
> > 2018-07-09 15:10:51.517536 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 5.399846 seconds old, received at 2018-07-09
> 15:10:46.117661: client_request(client.3775872:34690 lookup
> #10000011127/choury-test 2018-07-09 15:10:46.117586) currently failed to
> rdlock, waiting
> > 2018-07-09 15:10:53.517640 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : 2 slow requests, 1 included below; oldest blocked for > 10.622560
> secs
> > 2018-07-09 15:10:53.517646 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 10.622560 seconds old, received at 2018-07-09
> 15:10:42.895059: client_request(client.3775872:34689 lookup
> #10000011127/choury-test 2018-07-09 15:10:42.894941) currently failed to
> rdlock, waiting
> > 2018-07-09 15:10:56.517819 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : 1 slow requests, 1 included below; oldest blocked for > 10.400132
> secs
> > 2018-07-09 15:10:56.517826 7f32fa0c2700 0 log_channel(cluster) log
> [WRN] : slow request 10.400132 seconds old, received at 2018-07-09
> 15:10:46.117661: client_request(client.3775872:34690 lookup
> #10000011127/choury-test 2018-07-09 15:10:46.117586) currently failed to
> rdlock, waiting
>
> We reproduced this problem in the test cluster. there's only two
> processed(on two machines) access the cluster, the writer, and tail.
> The test writer code:
> > #include <stdio.h>
> > #include <string.h>
> > #include <stdlib.h>
> > #include <unistd.h>
> >
> > const char *s =
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
> > int main(int argc, char** argv){
> > FILE *f=fopen(argv[1],"ab+");
> > if(f==NULL){
> > printf("cant'to open destination file\n");
> > return 0;
> > }
> > int line = 0;
> > while(true){
> > char buff[1024]={0};
> > for(int i = 0; i< 200; i++){
> > buff[i] = s[rand()%62];
> > }
> > fprintf(f, "%d: %s\n", line++, buff);
> > fflush(f);
> > sleep(30);
> > }
> > fclose(f);
> > return 0;
> > }
> The version of ceph is 10.2.10.
> How can I reduce the latency and slow requests?
>
This is inherent to CephFS' strict consistency. When you issue a "tail" on
node B, you are forcing node A to flush out all of its data, then stop
writing, *then* the MDS returns the end of the file data to node B — then
node B drops its "capabilities" on the file and node A gets to start
writing again.
You can do things like this on some NFS systems because they don't actually
guarantee you'll see the results of the latest write to a file, but Ceph
doesn't work that way. There are some programmatic options for reducing
consistency, but I'm not sure any of them can be used to speed up a user
task like you have here.
-Greg
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
