On 12/16/2015 01:51 AM, Ankireddypalle Reddy wrote:
Thanks for the explanation. Valgrind profiling shows multiple memcpy's being
invoked for each write through libgfapi. Is there a way to avoid these
memcpy's?. Also is there a limit on the number of glfs_t* instances that can be
allocated at a given point of time. I've encountered cases where if more than 8
glfs_t* instances are being allocated then glfs_init fails.
Including maintainers of gfapi.
Pranith
Thanks and Regards,
Ram
-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Monday, December 14, 2015 11:13 PM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access
On 12/11/2015 08:58 PM, Ankireddypalle Reddy wrote:
Pranith,
Thanks for checking this. Though the time taken to run was 18
seconds if you look at the time consumed in user land as well as kernel land
for executing the command then it is evident that fuse took almost half the
time as libgfapi. Also from the collected profiles it is evident that the
average latency for the write command is less for fuse than for libgfapi. Are
there any recommendations for I/O through libgfapi for disperse volumes. Is
there any way to avoid the extra memcpy's that are being made when performing
I/O through libgfapi.
hi Ankireddy,
Oh this is not a problem. If we use fuse, the system call 'write'
from ./GlusterFuseTest will go through fuse-kernel, fuse kernel sends the write
operation to glusterfs mount process which is a user process. Time taken to
complete that call from then on is computed against the glusterfs mount process
until it responds to the fuse-kernel, not against the ./GlusterFuseTest
process. If we use gfapi, there is no system call over head, instead
./GlusterFuseTest process directly makes calls with the bricks through gfapi
library. So all the time that the process spends communicating with the bricks
and getting the response is counted against ./GlusterFuseTest. That is the
reason you see more 'user' time.
So again, There are quite a few workloads where gfapi has proven to give better response
times than fuse mounts because we avoid the context switch costs of ./GlusterFuseTest
-> fuse-kernel -> glusterfs-mount -> fuse-kernel (for response)->
./GlusterFuseTest (for response to 'write')
Hope that helps. Sorry for the delay in response, was in too many meetings
yesterday.
Pranith
Thanks and Regards,
Ram
-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Thursday, December 10, 2015 10:57 PM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access
On 12/10/2015 07:15 PM, Ankireddypalle Reddy wrote:
Hi,
Please let me know in case you need any more details. Even for only
write operations fuse seems to outperform libgfapi. Is it because of disperse
volumes?. Also I noticed a lot of data loss in case I use libgfapi asyn I/O for
disperse volumes.
Fuse and gfapi seem to take same amount of time to complete the run, i.e. 18
seconds. Could you let me know what you mean by fuse outperforming gfapi?
Pranith
Thanks and Regards,
Ram
-----Original Message-----
From: Ankireddypalle Reddy
Sent: Wednesday, December 09, 2015 5:01 PM
To: 'Pranith Kumar Karampuri'; Vijay Bellur;
gluster-users@gluster.org
Subject: RE: [Gluster-users] libgfapi access
Hi,
I upgraded my setup to gluster 3.7.3. I tested writes by performing
writes through fuse and through libgfapi. Attached are the profiles generated
from fuse and libgfapi. The test programs essentially writes 10000 blocks each
of 128K.
[root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 10000 Mount path:
/ws/glus Block size: 131072 Num of blocks: 10000 Will perform write test on
mount path : /ws/glus Succesfully created file /ws/glus/1449697583.glfs
Successfully filled file /ws/glus/1449697583.glfs Write test succeeded Write
test succeeded.
real 0m18.722s
user 0m3.913s
sys 0m1.126s
[root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2
24007 131072 10000 Host name: santest2
Volume: dispersevol
Port: 24007
Block size: 131072
Num of blocks: 10000
Will perform write test on volume: dispersevol Successfully filled file
1449697651.glfs Write test succeeded Write test succeeded.
real 0m18.630s
user 0m8.804s
sys 0m1.870s
Thanks and Regards,
Ram
-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Wednesday, December 09, 2015 1:39 AM
To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access
On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote:
Vijay,
We are trying to write data backed up by Commvault simpana to
glusterfs volume. The data being written is around 30 GB. Two kinds of write
requests happen.
1) 1MB requests
2) Small write requests of size 128 bytes. In case of libgfapi access
these are cached and a single 128KB write request is made where as in case of
FUSE the 128 byte write request is handled to FUSE directly.
glusterfs 3.6.5 built on Aug 24 2015 10:02:43
Volume Name: dispersevol
Type: Disperse
Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ssdtest:/mnt/ssdfs1/brick3
Brick2: sanserver2:/data/brick3
Brick3: santest2:/home/brick3
Options Reconfigured:
performance.cache-size: 512MB
performance.write-behind-window-size: 8MB
performance.io-thread-count: 32
performance.flush-behind: on
hi,
Things look okay. May be we can find something using profile info.
Could you post the results of the following operations:
1) gluster volume profile <volname> start
2) Run the fuse workload
3) gluster volume profile <volname> info > /path/to/file-1/to/send/us
4) Run the libgfapi workload
5)gluster volume profile <volname> info > /path/to/file-2/to/send/us
Send both these files to us to check what are the extra fops if any that are
sent over network which may be causing the delay.
I see that you are using disperse volume. If you are going to use disperse
volume for production usecases, I suggest you use 3.7.x preferably 3.7.3. We
fixed a bug in releases from 3.7.4 till 3.7.6 which will be released in 3.7.7.
Pranith
Thanks and Regards,
Ram
-----Original Message-----
From: Vijay Bellur [mailto:vbel...@redhat.com]
Sent: Monday, December 07, 2015 6:13 PM
To: Ankireddypalle Reddy; gluster-users@gluster.org
Subject: Re: [Gluster-users] libgfapi access
On 12/07/2015 10:29 AM, Ankireddypalle Reddy wrote:
Hi,
I am trying to use libgfapi interface to access
gluster volume. What I noticed is that reads/writes to the gluster
volume through libgfapi interface are slower than FUSE. I was
expecting the contrary. Are there any recommendations/settings
suggested to be used while using libgfapi interface.
Can you please provide more details about your tests? Providing information
like I/O block size, file size, throughput would be helpful.
Thanks,
Vijay
***************************Legal
Disclaimer***************************
"This communication may contain confidential and privileged material
for the sole use of the intended recipient. Any unauthorized review,
use or distribution by others is strictly prohibited. If you have
received the message by mistake, please advise the sender by reply email and delete
the message. Thank you."
********************************************************************
*
* _______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
***************************Legal
Disclaimer***************************
"This communication may contain confidential and privileged material
for the sole use of the intended recipient. Any unauthorized review,
use or distribution by others is strictly prohibited. If you have
received the message by mistake, please advise the sender by reply email and delete
the message. Thank you."
*********************************************************************
*
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material
for the sole use of the intended recipient. Any unauthorized review,
use or distribution by others is strictly prohibited. If you have
received the message by mistake, please advise the sender by reply email and delete
the message. Thank you."
**********************************************************************
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
