Hi, I am keeping Raghvendra in loop and hope he can comment on the "Read being scheduled as slow fop".
Other than that, I would request you to provide following information to debug this issue. 1 - Profile information of the volume. You can find the steps here - https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/ 2 - Gluster volume <volname> status 3 - Gluster volume heal <volname> info 4 - Check the size of each bricks to see if it has been filled. 5 - gluster version of client and server. 6 - Host OS version of client and server. --- Ashish ----- Original Message ----- From: a...@telecompartners.ro To: gluster-users@gluster.org Sent: Thursday, October 10, 2019 8:08:24 PM Subject: [Gluster-users] Gluster slow reads Hi, I have a very small distributed dispersed volume with 4+2 and 2 servers with 3 bricks on each of them. The volume is mounted via fuse client on another Linux server. The volume worked well for a few months in this setup. However in the last few days I have a very slow read speed (gluster uploads to the gluster client via the mount point). By slow I mean 3-4 MB/sec on a gigabit link. I don't have small files stored on gluster, the smallest file is around 30-40 MB. The networking between the client and the bricks is fine (all of them are connected in the same switch, no errors, some iptraf tests directly between client and gluster servers are looking good). On the same client I have mounted another 2 gluster volumes from other servers, both of them are ok. My 'gluster volume info' details are the following : Volume Name: gluster4-vol Type: Disperse Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: gluster4:/export/sdb1/brick Brick2: gluster5:/export/sdb1/brick Brick3: gluster4:/export/sdc1/brick Brick4: gluster5:/export/sdc1/brick Brick5: gluster4:/export/sdd1/brick Brick6: gluster5:/export/sdd1/brick Options Reconfigured: network.ping-timeout: 60 performance.client-io-threads: on performance.io-thread-count: 32 cluster.readdir-optimize: on performance.cache-size: 1GB client.event-threads: 10 server.event-threads: 10 cluster.lookup-optimize: on server.allow-insecure: on storage.reserve: 0 transport.address-family: inet nfs.disable: on In the gluster client debug logs I noticed the following lines : [2019-10-10 13:49:58.571879] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN scheduled as fast fop [2019-10-10 13:49:58.571964] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT scheduled as fast fop [2019-10-10 13:49:58.572058] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH scheduled as normal fop [2019-10-10 13:49:58.572728] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN scheduled as fast fop [2019-10-10 13:49:58.576275] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT scheduled as fast fop [2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 times between [2019-10-10 13:49:58.357922] and [2019-10-10 13:50:07.837047] [2019-10-10 13:50:07.837068] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH scheduled as normal fop [2019-10-10 13:50:07.837165] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ scheduled as slow fop What does it mean exactly "READ scheduled as slow fop" ? Can I schedule READs as normal or fast fop like the other operations ? I'm using this gluster volume only for reading so I don't care about writes right now. Thanks. ________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users