[Gluster-devel] Need help with bitrot
Hi, I am a student interested in GlusterFS. Trying to understand the design of GlusterFS. Came across the Bitrot design document in Google. There is a mention of expiry policy used to sign the files. I did not clearly understand what the expiry policy is. Can somebody please help? -Ajil ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Need help with bitrot
Thanks FNU Raghavendra. Does the signing happen only when the file data changes or even when extended attribute changes? I am also trying to understand the Gluster internal data structures. Are there any materials for the same? Similarly for the translators, the way they are stacked on client & server side, how control flows between them. Can somebody please help? - Ajil On Thu, Feb 25, 2016 at 7:27 AM, FNU Raghavendra Manjunath < rab...@redhat.com> wrote: > Hi Ajil, > > Expiry policy tells the signer (Bit-rot Daemon) to wait for a specific > period of time before signing a object. > > Whenever a object is modified, a notification is sent to the signer by > brick process (bit-rot-stub xlator sitting in the I/O path) upon getting a > release (i.e. when all the fds of that object are closed). The expiry > policy tells the signer to wait for some time (by default its 120 seconds) > before signing that object. It is done because, suppose the signer starts > signing (i.e. read the object + calculate the checksum + store the > checksum) a object the object gets modified again, then a new notification > has to be sent and again signer has to sign the object by calculating the > checksum. Whereas if the signer waits for some time and receives a new > notification on the same object when its waiting, then it can avoid signing > for the first notification. > > Venky, do you want to add anything more? > > Regards, > Raghavendra > > > > On Wed, Feb 24, 2016 at 12:28 AM, Ajil Abraham > wrote: > >> Hi, >> >> I am a student interested in GlusterFS. Trying to understand the design >> of GlusterFS. Came across the Bitrot design document in Google. There is a >> mention of expiry policy used to sign the files. I did not clearly >> understand what the expiry policy is. Can somebody please help? >> >> -Ajil >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel >> > > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Need help with bitrot
Thank you sir. I will read these documents. - Ajil On Thu, Feb 25, 2016 at 9:05 PM, FNU Raghavendra Manjunath < rab...@redhat.com> wrote: > As of now, signing happens only upon data modification. Metadata changes > and xattr changes does not trigger signing. > > For more information about gluster and its internals, you can look here " > https://gluster.readthedocs.org/en/latest/";. > > Regards, > Raghavendra > > On Thu, Feb 25, 2016 at 10:28 AM, Ajil Abraham > wrote: > >> Thanks FNU Raghavendra. Does the signing happen only when the file data >> changes or even when extended attribute changes? >> >> I am also trying to understand the Gluster internal data structures. Are >> there any materials for the same? Similarly for the translators, the way >> they are stacked on client & server side, how control flows between them. >> Can somebody please help? >> >> - Ajil >> >> >> On Thu, Feb 25, 2016 at 7:27 AM, FNU Raghavendra Manjunath < >> rab...@redhat.com> wrote: >> >>> Hi Ajil, >>> >>> Expiry policy tells the signer (Bit-rot Daemon) to wait for a specific >>> period of time before signing a object. >>> >>> Whenever a object is modified, a notification is sent to the signer by >>> brick process (bit-rot-stub xlator sitting in the I/O path) upon getting a >>> release (i.e. when all the fds of that object are closed). The expiry >>> policy tells the signer to wait for some time (by default its 120 seconds) >>> before signing that object. It is done because, suppose the signer starts >>> signing (i.e. read the object + calculate the checksum + store the >>> checksum) a object the object gets modified again, then a new notification >>> has to be sent and again signer has to sign the object by calculating the >>> checksum. Whereas if the signer waits for some time and receives a new >>> notification on the same object when its waiting, then it can avoid signing >>> for the first notification. >>> >>> Venky, do you want to add anything more? >>> >>> Regards, >>> Raghavendra >>> >>> >>> >>> On Wed, Feb 24, 2016 at 12:28 AM, Ajil Abraham >> > wrote: >>> >>>> Hi, >>>> >>>> I am a student interested in GlusterFS. Trying to understand the >>>> design of GlusterFS. Came across the Bitrot design document in Google. >>>> There is a mention of expiry policy used to sign the files. I did not >>>> clearly understand what the expiry policy is. Can somebody please help? >>>> >>>> -Ajil >>>> >>>> ___ >>>> Gluster-devel mailing list >>>> Gluster-devel@gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-devel >>>> >>> >>> >> > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Need help with bitrot
Thanks Joseph for the precise set of information. I will follow these. - Ajil On Fri, Feb 26, 2016 at 7:05 AM, Joseph Fernandes wrote: > Hope this helps > > Courtesy : Raghvendra Talur (rta...@redhat.com) > > 1. Clone glusterfs repo to your laptop and get acquainted with dev > workflow. > https://gluster.readthedocs.org/en/latest/Developer-guide/Developers-Index/ > > 2. If you find using your laptop as the test machine for Gluster as too > scary, here is a vagrant based mechanism to get VMs setup on your laptop > easily for Gluster testing. > http://comments.gmane.org/gmane.comp.file-systems.gluster.devel/13494 > > 3.Find my Gluster Introduction blog post here in the preview link: > > > > https://6227134958232800133_bafac39c28bee4f256bbbef7510c9bb9b44fca05.blogspot.com/b/post-preview?token=s6_4MVIBAAA.zY--3ij00CkDwnitBOwnFBowEvCsKZ0o4ToQ0KYk9Po4pKujPj9ugmn-fm-XUFdLQxU50FmnCxBBr_IkSzuSlA.l_XFe1UvIEAiqkFAZZPdqQ&postId=4168074834715190149&type=POST > > 4. Follow all the lessons here in Translator 101 series for build on your > understanding of Gluster. > > http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-class-1-setting-the-stage/ > > http://hekafs.org/index.php/2011/11/translator-101-lesson-2-init-fini-and-private-context > > http://hekafs.org/index.php/2011/11/translator-101-lesson-3-this-time-for-real > > http://hekafs.org/index.php/2011/11/translator-101-lesson-4-debugging-a-translator > > > 5. Try to fix or understand any of the bugs in this list > > https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&classification=Community&f1=keywords&list_id=4424622&o1=substring&product=GlusterFS&query_format=advanced&v1=easyfix > > > Regards, > Joe > > > - Original Message - > From: "Ajil Abraham" > To: "FNU Raghavendra Manjunath" > Cc: "Gluster Devel" > Sent: Thursday, February 25, 2016 8:58:35 PM > Subject: Re: [Gluster-devel] Need help with bitrot > > Thanks FNU Raghavendra. Does the signing happen only when the file data > changes or even when extended attribute changes? > > I am also trying to understand the Gluster internal data structures. Are > there any materials for the same? Similarly for the translators, the way > they are stacked on client & server side, how control flows between them. > Can somebody please help? > > - Ajil > > On Thu, Feb 25, 2016 at 7:27 AM, FNU Raghavendra Manjunath < > rab...@redhat.com > wrote: > > > > Hi Ajil, > > Expiry policy tells the signer (Bit-rot Daemon) to wait for a specific > period of time before signing a object. > > Whenever a object is modified, a notification is sent to the signer by > brick process (bit-rot-stub xlator sitting in the I/O path) upon getting a > release (i.e. when all the fds of that object are closed). The expiry > policy tells the signer to wait for some time (by default its 120 seconds) > before signing that object. It is done because, suppose the signer starts > signing (i.e. read the object + calculate the checksum + store the > checksum) a object the object gets modified again, then a new notification > has to be sent and again signer has to sign the object by calculating the > checksum. Whereas if the signer waits for some time and receives a new > notification on the same object when its waiting, then it can avoid signing > for the first notification. > > Venky, do you want to add anything more? > > Regards, > Raghavendra > > > > On Wed, Feb 24, 2016 at 12:28 AM, Ajil Abraham < ajil95.abra...@gmail.com > > wrote: > > > > Hi, > > I am a student interested in GlusterFS. Trying to understand the design of > GlusterFS. Came across the Bitrot design document in Google. There is a > mention of expiry policy used to sign the files. I did not clearly > understand what the expiry policy is. Can somebody please help? > > -Ajil > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > > > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Understanding ePoll, an interesting tutorial
I was trying to understand what epoll is from Gluster mailing lists. Unfortunately, could not find anything useful for a newbie like me. :). Found this tutorial extremely helpful for likes of me. https://banu.com/blog/2/how-to-use-epoll-a-complete-example-in-c/ - Ajil ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] glusterd crashing
For my project, I am trying to do some automation using glusterd. It is very frustrating to see it crashing frequently. Looks like input validation is the culprit. I also see lot of buffer overflow and memory leak issues. Making a note of these and will try to fix them. Surprised to see such basic issues still existing in Gluster. -Ajil ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] glusterd crashing
Hi Atin, The inputs I use are as per the requirements of a project I am working on for one of the large finance institutions in Dubai. I will try to handle the input validation within my code. I uncovered some of the issues while doing a thorough testing of my code. I tried with 3.7.6 and also my own build from master branch. I will check with my superiors before sending you backtrace and other details. So far, I have seen memory leak in 100s of KBs. -Ajil On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee wrote: > Hi Ajil, > > Its good to see that you are doing a thorough testing gluster. From your > mail it looks like your automation focuses on mostly negative tests. I need > few additional details to get to know whether they are known: > > 1. Version of gluster > 2. Backtrace of the crash along with reproducer > 3. Amount of memory leak in terms of bytes/KB/MB?? Have you already > identified them? > > -Atin > Sent from one plus one > On 03-Mar-2016 10:01 pm, "Ajil Abraham" wrote: > >> For my project, I am trying to do some automation using glusterd. It is >> very frustrating to see it crashing frequently. Looks like input validation >> is the culprit. I also see lot of buffer overflow and memory leak issues. >> Making a note of these and will try to fix them. Surprised to see such >> basic issues still existing in Gluster. >> >> -Ajil >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel >> > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] glusterd crashing
Thanks for all the support. After handling the input validation in my code, Glusterd no longer crashes. I am still waiting for clearance from my superior to pass on all the details. Expecting him to revert by this Sunday. - Ajil On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes wrote: > Well that may not be completely correct ! > > Its "gluster volume status all", unlike volume maintenance operation > which are rare. > > Status can be issued multiple times in a day or might be put in a > script/cron-job to check the health of the > cluster. > But anyways the fix is ready as the bug says. > > Crash is what we need to worry about. > > ~Joe > > - Original Message - > > From: "Atin Mukherjee" > > To: "Joseph Fernandes" , "Atin Mukherjee" < > atin.mukherje...@gmail.com> > > Cc: "Gluster Devel" , "Ajil Abraham" < > ajil95.abra...@gmail.com> > > Sent: Friday, March 4, 2016 9:37:43 AM > > Subject: Re: [Gluster-devel] glusterd crashing > > > > > > > > On 03/04/2016 07:10 AM, Joseph Fernandes wrote: > > > Might be this bug can give some context on the mem-leak (fix recently > > > merged on master but not on 3.7.x) > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517 > > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1] > > the hike is seen when a command is run in a loop which is typically not > > a use case in any production setup. > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15 > > > > > > ~Joe > > > > > > > > > - Original Message - > > >> From: "Atin Mukherjee" > > >> To: "Joseph Fernandes" > > >> Cc: "Gluster Devel" , "Ajil Abraham" > > >> > > >> Sent: Friday, March 4, 2016 7:01:54 AM > > >> Subject: Re: [Gluster-devel] glusterd crashing > > >> > > >> -Atin > > >> Sent from one plus one > > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" > wrote: > > >>> > > >>> Hi Ajil, > > >>> > > >>> Well few things, > > >>> > > >>> 1. Whenever you see a crash its better to send across the > Backtrace(BT) > > >> using gdb and attach the log files (or share it via some cloud drive) > > >>> > > >>> 2. About the memory leak, What kind of tools are you using for > profiling > > >> memory, valgrind ? if so please attach the valgrind reports. > > >>>$> glusterd --xlator-option *.run-with-valgrind=yes > > >>> > > >>> 3. Well I am not sure if glusterd uses any of the mempools as we do > in > > >> client and brick processes, Atin can shed some light on this. > > >>>Well In that case you can used the statedump mechanism check for > > >> mem-leaks check the glusterfs/doc/debugging/statedump.md > > >> GlusterD does use mempool and it has infra for capturing statedump as > > >> well. > > >> I am aware of few bytes of memory leaks in few paths which is really > not a > > >> huge concern but it shouldn't crash. > > >>> > > >>> Hope this helps > > >>> > > >>> ~Joe > > >>> > > >>> > > >>> - Original Message - > > >>>> From: "Ajil Abraham" > > >>>> To: "Atin Mukherjee" > > >>>> Cc: "Gluster Devel" > > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM > > >>>> Subject: Re: [Gluster-devel] glusterd crashing > > >>>> > > >>>> Hi Atin, > > >>>> > > >>>> The inputs I use are as per the requirements of a project I am > working > > >> on for > > >>>> one of the large finance institutions in Dubai. I will try to > handle the > > >>>> input validation within my code. I uncovered some of the issues > while > > >> doing > > >>>> a thorough testing of my code. > > >>>> > > >>>> I tried with 3.7.6 and also my own build from master branch. I will > > >> check > > >>>> with my superiors before sending you backtrace and other details. So > > >> far, I > > >>>> have seen memory leak in 100s of KBs. > > >>>> > &g
Re: [Gluster-devel] glusterd crashing
Sure Atin. I am itching to contribute code. But worried due to lack of experience in sending patches. Can somebody please send me across how to do this? Consider me a total newbie and please be as descriptive as possible :). -Ajil On Sat, Mar 5, 2016 at 12:46 PM, Atin Mukherjee wrote: > -Atin > Sent from one plus one > On 05-Mar-2016 11:46 am, "Ajil Abraham" wrote: > > > > Thanks for all the support. After handling the input validation in my > code, Glusterd no longer crashes. I am still waiting for clearance from my > superior to pass on all the details. Expecting him to revert by this Sunday. > Great to know that and we appreciate your contribution, if you happen to > find any issues feel free to send patches :) > > > > - Ajil > > > > On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes > wrote: > >> > >> Well that may not be completely correct ! > >> > >> Its "gluster volume status all", unlike volume maintenance operation > which are rare. > >> > >> Status can be issued multiple times in a day or might be put in a > script/cron-job to check the health of the > >> cluster. > >> But anyways the fix is ready as the bug says. > >> > >> Crash is what we need to worry about. > >> > >> ~Joe > >> > >> - Original Message - > >> > From: "Atin Mukherjee" > >> > To: "Joseph Fernandes" , "Atin Mukherjee" < > atin.mukherje...@gmail.com> > >> > Cc: "Gluster Devel" , "Ajil Abraham" < > ajil95.abra...@gmail.com> > >> > Sent: Friday, March 4, 2016 9:37:43 AM > >> > Subject: Re: [Gluster-devel] glusterd crashing > >> > > >> > > >> > > >> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote: > >> > > Might be this bug can give some context on the mem-leak (fix > recently > >> > > merged on master but not on 3.7.x) > >> > > > >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517 > >> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1] > >> > the hike is seen when a command is run in a loop which is typically > not > >> > a use case in any production setup. > >> > > >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15 > >> > > > >> > > ~Joe > >> > > > >> > > > >> > > - Original Message - > >> > >> From: "Atin Mukherjee" > >> > >> To: "Joseph Fernandes" > >> > >> Cc: "Gluster Devel" , "Ajil Abraham" > >> > >> > >> > >> Sent: Friday, March 4, 2016 7:01:54 AM > >> > >> Subject: Re: [Gluster-devel] glusterd crashing > >> > >> > >> > >> -Atin > >> > >> Sent from one plus one > >> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" > wrote: > >> > >>> > >> > >>> Hi Ajil, > >> > >>> > >> > >>> Well few things, > >> > >>> > >> > >>> 1. Whenever you see a crash its better to send across the > Backtrace(BT) > >> > >> using gdb and attach the log files (or share it via some cloud > drive) > >> > >>> > >> > >>> 2. About the memory leak, What kind of tools are you using for > profiling > >> > >> memory, valgrind ? if so please attach the valgrind reports. > >> > >>>$> glusterd --xlator-option *.run-with-valgrind=yes > >> > >>> > >> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we > do in > >> > >> client and brick processes, Atin can shed some light on this. > >> > >>>Well In that case you can used the statedump mechanism check > for > >> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md > >> > >> GlusterD does use mempool and it has infra for capturing statedump > as > >> > >> well. > >> > >> I am aware of few bytes of memory leaks in few paths which is > really not a > >> > >> huge concern but it shouldn't crash. > >> > >>> > >> > >>> Hope this helps > >> > >>> > >> > >>> ~Joe > >> > >>>
Re: [Gluster-devel] glusterd crashing
Thank you Joseph. I fell sick and stayed out of action. Hope to continue end of this week. Planning to subscribe to gluster-devel. Any criteria to be met for approving my request? -Ajil On Sat, Mar 5, 2016 at 10:19 PM, Joseph Fernandes wrote: > > http://www.gluster.org/community/documentation/index.php/Archives/Development_Work_Flow > > > http://www.gluster.org/community/documentation/index.php/Simplified_dev_workflow > > Leaving the fun of exploration to you :) > > ~Joe > > - Original Message - > > From: "Ajil Abraham" > > To: "Atin Mukherjee" > > Cc: "Joseph Fernandes" , "Gluster Devel" < > gluster-devel@gluster.org> > > Sent: Saturday, March 5, 2016 10:06:37 PM > > Subject: Re: [Gluster-devel] glusterd crashing > > > > Sure Atin. I am itching to contribute code. But worried due to lack of > > experience in sending patches. Can somebody please send me across how to > do > > this? Consider me a total newbie and please be as descriptive as possible > > :). > > > > -Ajil > > > > On Sat, Mar 5, 2016 at 12:46 PM, Atin Mukherjee < > atin.mukherje...@gmail.com> > > wrote: > > > > > -Atin > > > Sent from one plus one > > > On 05-Mar-2016 11:46 am, "Ajil Abraham" > wrote: > > > > > > > > Thanks for all the support. After handling the input validation in > my > > > code, Glusterd no longer crashes. I am still waiting for clearance > from my > > > superior to pass on all the details. Expecting him to revert by this > > > Sunday. > > > Great to know that and we appreciate your contribution, if you happen > to > > > find any issues feel free to send patches :) > > > > > > > > - Ajil > > > > > > > > On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes < > josfe...@redhat.com> > > > wrote: > > > >> > > > >> Well that may not be completely correct ! > > > >> > > > >> Its "gluster volume status all", unlike volume maintenance > operation > > > which are rare. > > > >> > > > >> Status can be issued multiple times in a day or might be put in a > > > script/cron-job to check the health of the > > > >> cluster. > > > >> But anyways the fix is ready as the bug says. > > > >> > > > >> Crash is what we need to worry about. > > > >> > > > >> ~Joe > > > >> > > > >> - Original Message - > > > >> > From: "Atin Mukherjee" > > > >> > To: "Joseph Fernandes" , "Atin Mukherjee" < > > > atin.mukherje...@gmail.com> > > > >> > Cc: "Gluster Devel" , "Ajil Abraham" < > > > ajil95.abra...@gmail.com> > > > >> > Sent: Friday, March 4, 2016 9:37:43 AM > > > >> > Subject: Re: [Gluster-devel] glusterd crashing > > > >> > > > > >> > > > > >> > > > > >> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote: > > > >> > > Might be this bug can give some context on the mem-leak (fix > > > recently > > > >> > > merged on master but not on 3.7.x) > > > >> > > > > > >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517 > > > >> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer > to [1] > > > >> > the hike is seen when a command is run in a loop which is > typically > > > not > > > >> > a use case in any production setup. > > > >> > > > > >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15 > > > >> > > > > > >> > > ~Joe > > > >> > > > > > >> > > > > > >> > > - Original Message - > > > >> > >> From: "Atin Mukherjee" > > > >> > >> To: "Joseph Fernandes" > > > >> > >> Cc: "Gluster Devel" , "Ajil > Abraham" > > > >> > >> > > > >> > >> Sent: Friday, March 4, 2016 7:01:54 AM > > > >> > >> Subject: Re: [Gluster-devel] glusterd crashing > > > >> > >> > > > >> > >> -Atin > > > >> > >> Sent from on
[Gluster-devel] Gluster-devel subscription
Hi, I subscribed to gluster-devel 2 days ago. How do I know that my request is approved? I so far have not got any email. Can somebody please help? -Ajil ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
