Hey Pranith, I am good, I hope you are doing good too.
Please find the comments inline.
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, June 21, 2016 5:58 AM
To: Sachin Pandit <span...@commvault.com>
Cc: gluster-devel@gluster.org
Subject: Re: [Gluster-devel] Reduce memcpy in glfs read and write
Hey!!
Hope you are doing good. I took a look at the bt. So when flush comes
write-behind has to flush all the writes down. I see the following frame hung
in iob_unref:
Thread 7 (Thread 0x7fa601a30700 (LWP 16218)):
#0 0x00007fa60cc55225 in pthread_spin_lock () from /lib64/libpthread.so.0
<<---- Does it always hang there?
---------------------------------
>>It does always hang here.
---------------------------------
#1 0x00007fa60e1f373e in iobref_unref (iobref=0x19dc7e0) at iobuf.c:907
#2 0x00007fa60e246fb2 in args_wipe (args=0x19e70ec) at default-args.c:1593
#3 0x00007fa60e1ea534 in call_stub_wipe_args (stub=0x19e709c) at
call-stub.c:2466
#4 0x00007fa60e1ea5de in call_stub_destroy (stub=0x19e709c) at call-stub.c:2482
Is this on top of master branch? It seems like we missed an unlock of the
spin-lock or the iobref has junk value which gives the feeling that it is in
locked state (May be double free?). Do you have any extra patches you have in
your repo which make changes in iobuf?
----------------------------------
>>I have implemented a method to reduce memcpy in libgfapi (My patch is on top
>>of master branch), by making use of buffer from iobuf pool and passing the
>>buffer to application. However, I have not made any changes in iobuf core
>>feature. I don’t think double free is happening anywhere in the code (I did
>>check this using logs)
Method that I have implemented:
1) Application asks for a buffer of specific size, and the buffer is
allocated from the iobuf pool.
2) Buffer is passed on to application, and the application writes the data
into that buffer.
3) Buffer with data in it is passed from application to libgfapi and the
underlying translators (no memcpy in glfs_write)
I have couple of questions, and observations:
Observations:
------------------
1) For every write if I get a fresh buffer then I don’t see any problem.
All the writes are going through.
2) If I try to make use of buffer for consecutive writes, then I am seeing
the hang in flush.
Question1: Is it fine if I reuse the buffer for consecutive writes??
Question2: Is it always ensured that the data is written to the file when I get
a response from syncop_writev.
Thanks,
Sachin Pandit.
----------------------------------
On Tue, Jun 21, 2016 at 4:07 AM, Sachin Pandit
<span...@commvault.com<mailto:span...@commvault.com>> wrote:
Hi all,
I bid adieu to you all with the hope of crossing path again, and the time has
come rather quickly. It feels great to work on GlusterFS again.
Currently we are trying to write data backed up by Commvault Simpana to
glusterfs volume (Disperse volume). To improve the performance, I have
implemented the proposal put forward my Rafi K C [1]. I have some questions
regarding libgfapi and iobuf pool.
To reduce an extra level of copy in glfs read and write, I have implemented few
APIs to request a buffer (similar to the one represented in [1]) from iobuf
pool which can be used by the application to write data to. With this
implementation, when I try to reuse the buffer for consecutive writes, I could
see a hang in syncop_flush of glfs_close (BT of the hang can be found in [2]).
I wanted to know if reusing the buffer is recommended. If not, do we need to
request buffer for each writes?
Setup : Distributed-Disperse ( 4 * (2+1)). Bricks scattered over 3 nodes.
[1] http://www.gluster.org/pipermail/gluster-devel/2015-February/043966.html
[2] Attached file - bt.txt
Thanks & Regards,
Sachin Pandit.
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org<mailto:Gluster-devel@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-devel
--
Pranith
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel